search

sefinek / Sefinek-Blocklist-Collection

🌍 A comprehensive repository of blocklists for various DNS servers, featuring over 100 links and more than 6 million domains. Additionally, you can use our personalized Blocklist Generator to tailor content filtering according to your specific needs, giving you full control over what gets blocked on your network.
https://blocklist.sefinek.net
Other
554 stars 26 forks source link

duplicated domain #65

Closed alsyundawy closed 1 week ago

alsyundawy commented 1 month ago

duplicate list on

https://blocklist.sefinek.net/generated/v1/rpz/social/tiktok.txt

then can check all list you have when remove duplicated

sefinek commented 1 month ago

I don't quite understand what you mean 🤔

alsyundawy commented 1 month ago

do you check duplicated record on tiktok.txt?

alsyundawy commented 1 month ago 
#!/bin/bash

# URL of the file
URL="https://blocklist.sefinek.net/generated/v1/rpz/social/tiktok.txt"

# Download the file and remove duplicates without sorting
curl -s "$URL" | awk '!seen[$0]++' > tiktok_cleaned.txt

echo "Duplicates removed and cleaned file saved as tiktok_cleaned.txt"
sefinek commented 3 weeks ago

image There are no duplicates here

sefinek commented 3 weeks ago

Indeed, the RPZ had duplicate lines, thanks for pointing that out. I fixed it, although I really didn't have the desire and time

https://github.com/sefinek24/Sefinek-Blocklist-Collection/commit/882ce9999323c62da75e2e5054fe981a8d0157d0