search

sefinek24 / Sefinek-Blocklist-Collection

🌍 A comprehensive repository of blocklists for various DNS servers, featuring over 100 links and more than 6 million domains. Additionally, you can use our personalized Blocklist Generator to tailor content filtering according to your specific needs, giving you full control over what gets blocked on your network.
https://blocklist.sefinek.net
Other
546 stars 25 forks source link

duplicated domain #65

Open alsyundawy opened 1 month ago

alsyundawy commented 1 month ago

duplicate list on

https://blocklist.sefinek.net/generated/v1/rpz/social/tiktok.txt

then can check all list you have when remove duplicated

sefinek24 commented 1 month ago

I don't quite understand what you mean 🤔

alsyundawy commented 1 month ago

do you check duplicated record on tiktok.txt?

alsyundawy commented 1 month ago 
#!/bin/bash

# URL of the file
URL="https://blocklist.sefinek.net/generated/v1/rpz/social/tiktok.txt"

# Download the file and remove duplicates without sorting
curl -s "$URL" | awk '!seen[$0]++' > tiktok_cleaned.txt

echo "Duplicates removed and cleaned file saved as tiktok_cleaned.txt"
sefinek24 commented 1 week ago

image There are no duplicates here

sefinek24 commented 1 week ago

Indeed, the RPZ had duplicate lines, thanks for pointing that out. I fixed it, although I really didn't have the desire and time

https://github.com/sefinek24/Sefinek-Blocklist-Collection/commit/882ce9999323c62da75e2e5054fe981a8d0157d0