search

sefinek / Sefinek-Blocklist-Collection

🌍 A comprehensive repository of blocklists for various DNS servers, featuring over 100 links and more than 6 million domains. Additionally, you can use our personalized Blocklist Generator to tailor content filtering according to your specific needs, giving you full control over what gets blocked on your network.
https://blocklist.sefinek.net
Other
560 stars 26 forks source link

duplicated domain #65

Closed alsyundawy closed 1 month ago

alsyundawy commented 2 months ago

duplicate list on

https://blocklist.sefinek.net/generated/v1/rpz/social/tiktok.txt

then can check all list you have when remove duplicated

sefinek commented 2 months ago

I don't quite understand what you mean 🤔

alsyundawy commented 2 months ago

do you check duplicated record on tiktok.txt?

alsyundawy commented 2 months ago 
#!/bin/bash

# URL of the file
URL="https://blocklist.sefinek.net/generated/v1/rpz/social/tiktok.txt"

# Download the file and remove duplicates without sorting
curl -s "$URL" | awk '!seen[$0]++' > tiktok_cleaned.txt

echo "Duplicates removed and cleaned file saved as tiktok_cleaned.txt"
sefinek commented 1 month ago

image There are no duplicates here

sefinek commented 1 month ago

Indeed, the RPZ had duplicate lines, thanks for pointing that out. I fixed it, although I really didn't have the desire and time

https://github.com/sefinek24/Sefinek-Blocklist-Collection/commit/882ce9999323c62da75e2e5054fe981a8d0157d0