Enrich your newly created domain names feed

[maaaaz]

Hello @atenreiro,

FYI, I assembled this project in order to provide everyday newly created domain names (NRD) that might not already be inside the "WhoisDS" source: https://github.com/maaaaz/dnsdumps

It might help to complete your own daily feed.

Cheers!

[atenreiro]

Great! Will take a look at this 😊

Thanks a lot buddy

[atenreiro]

I'm currently testing the DNS dumps and fetching this list daily.

At what time (and timezone) do you update this file?

[maaaaz]

• At day D, the remote crawling script starts at 20:00 Paris time (18h GMT) . Some zone walks can talk up to 5 hours.

  • Look at "Last modified" times here: https://dailynrdscan.alwaysdata.net/logs/?C=M;O=D

• So at day D+1, the merging Github Action starts around 02:20 Paris time (00:20 GMT) and takes few seconds:

  • https://github.com/maaaaz/dnsdumps/blob/main/.github/workflows/daily_update.yml
  • https://github.com/maaaaz/dnsdumps/actions/runs/5418504616/jobs/9850671681

[atenreiro]

I'm parsing the today_new.gz and removing the entries that are not domain names (e.g: mx1.domain.com)

[maaaaz]

Indeed you're right, I am not doing this removal as I have worklows to find potential malicious websites, so I need FQDN and not only domains.

[atenreiro]

Ok, I'm filtering using a RegEx for everything that is not in the domain.tld format.

[atenreiro]

Hey @maaaaz

Do you need a whitelisted SSH key to access this rsync folder?

# rsynced data is in /tmp/zones
rsync --stats -az -e "ssh -o LogLevel=quiet -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i /tmp/sshkey" dailynrdscan@ssh-dailynrdscan.alwaysdata.net:/home/dailynrdscan/www/zones 

[maaaaz]

Hello @atenreiro,

Why do you want to perform a rsync ?

I do a rsync to bring these data to the GitHub repository, then you have the very same information as on https://github.com/maaaaz/dnsdumps/tree/main

Cheers!

[atenreiro]

Hey @maaaaz

I was trying to replicate your script and avoid getting the feeds directly from GitHub, to avoid an additional point of failure.

[maaaaz]

Okay, nice idea but to be frank the SPOF would rather be that tiny free web remote server dailynrdscan.alwaysdata.net

Would you prefer to have a daily zip at the root of https://dailynrdscan.alwaysdata.net/zones/ containing the "zones" folder, for you to be able to download the content ?

Anyway I am planning to publish the (ugly) bash script I am using to perform the daily scans: you could also reuse it on your side.

Cheers!

[atenreiro]

Ok, I will continue downloading from the GitHub but for now.

Do you manage the hosting server https://dailynrdscan.alwaysdata.net ?

[maaaaz]

No, it is the free plan of https://www.alwaysdata.com/en/ where as a user, you just have an SSH access to a shared VPS, without root permissions.