Domains-names-txt not found

[wargasmtg]

Downloading the fresh domain list seems to be broken when using -p day for the daily download. Executing without -p day still works.

[atenreiro]

Hello @wargasmtg

For an unknown reason at this point in time, the daily feed for 4th July is not available. the latest available daily data is for the 3rd of July. This should be fixed during the course of today or tomorrow when the data for the 5th July will be made available.

As a quick workaround, I will add the "domain-names.txt" with the data from the 3rd of July.

Thanks a lot for reporting this. I will enhance the code so when this happens again it doesn't break the usage.

[wargasmtg]

Thanks! Seems today 7/7 the daily is not working. Does the full pull contain the data of the previous zonefile update and includes the daily? I assume you pull the data from https://czds.icann.org/home ?

[atenreiro]

Thanks,

Fixed the same way.

We rely on private feeds to get this information (for free). I would like to improve the feeds by adding more sources that contain more information that can be used for threat intelligence (e.g: register email and owner) however, it's a bit challenging at this point of time as most of these feeds are not free and this is a zero budget project.

[wargasmtg]

Thanks for the fix. So you can obtain zone file changes from ICANN as mentioned earlier; https://czds.icann.org/home this is the source for many companies. Registries are contractually required to publish the zone files. And ICANN publishes this for free. So you might be able to new domain names even more quickly?

Another source also free is; https://otx.alienvault.com/user/ZENDataGE/pulses I use that now and then because I need to renew my ICANN CZDS every 3 months.

Hope this helps!

[atenreiro]

@wargasmtg

Thanks for the inputs, I will be evaluating the sources, if they support API, it shouldn't be a problem.

Just to let you know, that the openSquat was built in a way that you can use other external files/sources by using the "-d myfile.txt" flag to do the analysis, as long you keep the same structure (one domain per line). Maybe I can also add support for custom URL.

If there are other questions/enhancements that you would raise, feel free to add a "new issue" anytime.

[wargasmtg]

yes the -d myfile works perfect :)

[wargasmtg]

There is an API but CZDS only does the new generic TLD's (not legacy like .com) and it pulls ALL the domain names (I forgot) not just the new ones. API https://github.com/icann/czds-api-client-java

[atenreiro]

Great stuff!

Already working on the CZDS API. Apart from OTX, do you have any other free sources for .COM/.NET/.ORG?

[wargasmtg]

I will check how to get access to the TLDs like .com .biz .org etc. This info should be for free also.

[wargasmtg]

Seems .com, .net access will be available before July 25, 2020, it should have been available 7 days ago, but there is a delay.

[wargasmtg]

Keep in mind if you pull stuff from restricted TLDs, like .Bank or .BMW will contain brands. .BANK is only for banks and to get a registration you need to be a bank. So it is likely that abnamro.bank or Barclays.bank will score very high but is still legit.

[atenreiro]

Out of curiosity, what is your current use case for openSquat? Threat intelligence?

[wargasmtg]

Yes TI and a large focus on BEC fraud. We know what criminals register for such crimes. And Opensquat is just awesome to get the raw batch. It requires a lot more vetting beyond that. But it is awesome to pull that info really quick and disrupt ongoing BEC fraud. I guess thanks to you we disrupted a lot of fraud and saved a lot of people money! Sure they will never credit us, but we know ;)

[atenreiro]

I have spent a lot of hours thinking about how to combat online frauds and scams using OSINT and of course, coding. Knowing that openSquat is making an impact is very encouraging and fulfilling.

[atenreiro]

Credits to @mateuszz0000 as well!

[wargasmtg]

Well, we spend a lot of time investigating stuff. And your tool is super handy. And yes we do not get paid and we do not get the glory. We are a trusted source for several parties and stuff gets even blocked at a browser level. While we are somewhat new to OSINT. We do know a lot about DNS. So we are all doing something right :)

Theo

[atenreiro]

If you need additional features, just create a new request and I'll have a lot of it. Very nice that the tool is handy :-)

[atenreiro]

@wargasmtg

Speaking about DNS, I am working with Quad9 (quad9.net) to integrate openSquat with their service to validate if a domain has been flagged as malicious or not. Would this be useful for you? Do you have any other suggestion?

[wargasmtg]

Quad 9 is pretty good, but you also have VT on the roadmap. It would be great to use VirusTotal and be able to use our API Key.

[atenreiro]

Yeah, probably there are overlaps between the two services. I like the fact that the Quad9 can be integrated without API, for power users that already have a VT API, this last service is probably more useful.

[wargasmtg]

Quad 9 results would be handy as you can easily zoom in for further investigation

[atenreiro]

VT does API throttle, the free version is 4 per minute. The Quad9 is able to handle significantly more requests.

[wargasmtg]

Agreed, the throttle is annoying, and for large amounts of suspected IOC's it can take hours to process. You also might want to look into https://pulsedive.com/ and talk to them on their slack. Very flexible and great people. They are always on the lookout for new opportunities.

[atenreiro]

Great, will take a look!

Will close this issue and create a new one with the DNS subject.