alsyundawy commented 4 months ago

Can you rename all Duplicates files names?

sefinek commented 4 months ago

Hmm, I don't quite understand what the issue is

alsyundawy commented 4 months ago

here you go

https://blocklist.sefinek.net/generated/v1/unbound/abuse/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/ads/adaway/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/ads/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/fakenews/StevenBlack/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/fakenews/marktron/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/fraud/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/gambling/StevenBlack/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/gambling/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/unbound/porn/4skinSkywalker/hosts.fork.conf

https://blocklist.sefinek.net/generated/v1/rpz/abuse/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/ads/adaway/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/ads/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/fakenews/StevenBlack/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/fakenews/marktron/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/fraud/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/gambling/StevenBlack/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/gambling/blocklistproject/hosts.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/porn/4skinSkywalker/hosts.fork.conf

the name must like

https://blocklist.sefinek.net/generated/v1/rpz/porn/4skinSkywalker/4skinSkywalker-porn.fork.conf
https://blocklist.sefinek.net/generated/v1/rpz/gambling/blocklistproject/blocklistproject-gambling.fork.conf

sefinek commented 4 months ago

I named these files hosts.fork.txt because they are the only files in a specific folder

alsyundawy commented 4 months ago

I found the same name again

https://blocklist.sefinek.net/generated/v1/0.0.0.0/abuse/blocklistproject/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/fraud/blocklistproject/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/porn/4skinSkywalker/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/fakenews/StevenBlack/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/fakenews/marktron/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/gambling/StevenBlack/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/gambling/blocklistproject/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/127.0.0.1/ads/adaway/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/127.0.0.1/ads/blocklistproject/hosts.fork.txt

https://blocklist.sefinek.net/generated/v1/0.0.0.0/abuse/blocklistproject/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/fraud/blocklistproject/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/porn/4skinSkywalker/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/fakenews/StevenBlack/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/fakenews/marktron/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/gambling/StevenBlack/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/0.0.0.0/gambling/blocklistproject/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/noip/ads/adaway/hosts.fork.txt https://blocklist.sefinek.net/generated/v1/noip/ads/blocklistproject/hosts.fork.txt

then I focused on naming this file because it didn't occur to you that if the user were to download all the files there would definitely be a conflict or overwrite.

sefinek commented 4 months ago

Ah, now I understand what you mean. Usually, these lists are downloaded by an automated script written by the user, not by the user themselves. Therefore, in my opinion, renaming these files differently is unnecessary, as the end user can freely manipulate the file names when downloading.

In theory, it could look like this:

const axios = require('axios');
const fs = require('fs/promises');
const path = require('path');

// List of files to download with their respective URLs and paths
const filesToDownload = [
    {
        url: 'https://blocklist.sefinek.net/generated/v1/0.0.0.0/gambling/StevenBlack/hosts.fork.txt',
        path: 'lists/StevenBlack.hosts.txt'
    },
    {
        url: 'https://blocklist.sefinek.net/generated/v1/0.0.0.0/gambling/blocklistproject/hosts.fork.txt',
        path: 'lists/blocklistproject.hosts.txt'
    }
];

// Function to download a file given its URL and save it to the specified path
const downloadFile = async file => {
    try {
        // Fetch the file data from the URL
        const res = await axios.get(file.url, { responseType: 'arraybuffer' });

        // Ensure the directory exists or create it recursively
        await fs.mkdir(path.dirname(file.path), { recursive: true });

        // Write the fetched data to the specified file path
        await fs.writeFile(file.path, res.data);
        console.log(`Downloaded file ${file.url} to ${file.path}`);
    } catch (err) {
        // Log any errors that occur during the download process
        console.error(`Error downloading file ${file.url}.`, err);
    }
};

// Main function to download all files concurrently
(async () => {
    await Promise.all(filesToDownload.map(downloadFile));
})();

alsyundawy commented 4 months ago

this my bash script dns rpz updater

BLACKLIST_URLS=(
  "https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn-only/hosts"
  "https://raw.githubusercontent.com/Bon-Appetit/porn-domains/master/block.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-onlydomains.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt"
)
ADDITIONAL_URLS=(
  "https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-rpz.conf"
  "https://malware-filter.gitlab.io/malware-filter/vn-badsite-filter-rpz.conf"
  "https://malware-filter.gitlab.io/pup-filter/pup-filter-rpz.conf"
  "https://malware-filter.gitlab.io/malware-filter/phishing-filter-rpz.conf"
)
WHITELIST_URL="https://raw.githubusercontent.com/alsyundawy/hds-pihole-blacklist/main/dnssehat/whitelist.txt"

BLACKLIST_OUTPUT="/etc/bind/zones/blacklist.rpz"
WHITELIST_OUTPUT="/etc/bind/zones/whitelist.rpz"
SAFESEARCH_OUTPUT="/etc/bind/zones/google_safesearch.rpz"
ADDITIONAL_OUTPUT_DIR="/etc/bind/zones/"

sefinek commented 4 months ago

In theory, it could look like this in bash:

#!/bin/bash

# List of URLs and their corresponding file paths for blacklists
BLACKLIST_URLS=(
  "https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn-only/hosts, lists/StevenBlack.hosts.txt"
  "https://raw.githubusercontent.com/Bon-Appetit/porn-domains/master/block.txt, lists/Bon-Appetit.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/doh-onlydomains.txt, lists/hagezi-doh-onlydomains.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/fake-onlydomains.txt, lists/hagezi-fake-onlydomains.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/tif-onlydomains.txt, lists/hagezi-tif-onlydomains.txt"
  "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/wildcard/gambling-onlydomains.txt, lists/hagezi-gambling-onlydomains.txt"
)

# List of URLs and their corresponding file paths for additional URLs
ADDITIONAL_URLS=(
  "https://malware-filter.gitlab.io/malware-filter/urlhaus-filter-rpz.conf, lists/urlhaus-filter-rpz.conf"
  "https://malware-filter.gitlab.io/malware-filter/vn-badsite-filter-rpz.conf, lists/vn-badsite-filter-rpz.conf"
  "https://malware-filter.gitlab.io/pup-filter/pup-filter-rpz.conf, lists/pup-filter-rpz.conf"
  "https://malware-filter.gitlab.io/malware-filter/phishing-filter-rpz.conf, lists/phishing-filter-rpz.conf"
)

# URL and file path for the whitelist
WHITELIST_URL="https://raw.githubusercontent.com/alsyundawy/hds-pihole-blacklist/main/dnssehat/whitelist.txt, lists/whitelist.txt"

# Function to create necessary directories
# Takes a list of URL-path pairs, extracts paths, and creates directories
create_directories() {
  local entries=("$@")
  for entry in "${entries[@]}"; do
    IFS=', ' read -r _ path <<< "$entry"
    dir=$(dirname "$path")
    mkdir -p "$dir"
  done
}

# Function to download files
# Takes a list of URL-path pairs, downloads each file to the specified path
download_files() {
  local entries=("$@")
  for entry in "${entries[@]}"; do
    IFS=', ' read -r url path <<< "$entry"
    echo "Downloading $url to $(realpath $path)..."
    curl -s -o "$path" "$url"
  done
}

# Create directories for all lists
create_directories "${BLACKLIST_URLS[@]}"
create_directories "${ADDITIONAL_URLS[@]}"
create_directories "$WHITELIST_URL"

# Download files for all lists
download_files "${BLACKLIST_URLS[@]}"
download_files "${ADDITIONAL_URLS[@]}"
download_files "$WHITELIST_URL"

alsyundawy commented 4 months ago

Function to process the blacklist using aria2c

process_blacklist() { local output_file=$BLACKLIST_OUTPUT echo "$(generate_rpz_header)" > "$output_file" for url in "${BLACKLIST_URLS[@]}"; do aria2c --check-certificate=false -x 16 -s 16 --retry-wait=5 --retry=3 --allow-overwrite=true --conditional-get=true -o temp_blacklist.txt "$url"

Ensure proper RPZ format: domains should be redirected to a specific domain

cat temp_blacklist.txt | sed '/^#/d' | sed '/^;/d' |sed '/^\s*$/d' | sort | uniq | awk '{if ($1 ~ /^0\.0\.0\.0/) print $2 " IN CNAME ."; else print $1 " IN CNAME ."; "*." $1 " IN CNAME ."}' >> "$output_file"
rm -rf temp_blacklist.txt

done }

sefinek / Sefinek-Blocklist-Collection

Duplicates files names #35

Function to process the blacklist using aria2c

Ensure proper RPZ format: domains should be redirected to a specific domain