Modernize Virustotal API scraping

[theteleforce]

Address #194 by modernizing the Virustotal scraper.

Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.

This solves the same problem as #285, but does not require an API key and uses the existing parser logic.

This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

[famdude]

It worked. Tnx!

[ko2sec]

works fine, thanks for the fix.

[OMGd0g]

I'm still learning to Linux here and I'm in douobt which file to change... When doing a search i find it at 2 locations /usr/lib/python3/dist-packages/sublist3r.py /usr/lib/python3/dist-packages/theHarvester/discovery/sublist3r.py Should I edit both for this fix to work?

Thanks :)

[ko2sec]

I'm still learning to Linux here and I'm in douobt which file to change... When doing a search i find it at 2 locations

/usr/lib/python3/dist-packages/sublist3r.py

/usr/lib/python3/dist-packages/theHarvester/discovery/sublist3r.py

Should I edit both for this fix to work?

Thanks :)

theHarvester is another tool, you need to edit /usr/lib/python3/dist-packages/sublist3r.py

[wvdongen]

The fix works, merge it!

[Defirence]

Can we get this merged please? cc: @aboul3la

[xmegax]

it worked , thanks

[firingdove]

Address #194 by modernizing the Virustotal scraper.

Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.

This solves the same problem as #285, but does not require an API key and uses the existing parser logic.

This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

I am new to linux and using sublist3r. please it possible to provide a step by step instruction on how I can solve the problem. Because I tried using the API key and is not working. thanks

[firingdove]

it worked , thanks

Please How did you do it?

[xmegax]

it worked , thanks

Please How did you do it?

use this repo. https://github.com/AetherBreeze/Sublist3r.git

[firingdove]

it worked , thanks

Please How did you do it?

use this repo. https://github.com/AetherBreeze/Sublist3r.git

thanks Man. it is working.

[Defirence]

Can we merge this change or is this project now considered being in "development hell"? If it works, WHY can't we commit this? Hello?

EDIT: Looking at @aboul3la 's front page it seems they're either a: working full-time b: dead (hopefully not) or c: this project has been abandoned.

I'm going with the latter being c: Project maintenance has been abandoned. This is a great tool, sad to see even PR's are not being merged. If anyone is looking for a working version that respects the new changes use @AetherBreeze 's forked repo here:

https://github.com/AetherBreeze/Sublist3r

[panchal2001]

Can we merge this change or is this project now considered being in "development hell"? If it works, WHY can't we commit this? Hello?

EDIT: Looking at @aboul3la 's front page it seems they're either a: working full-time b: dead (hopefully not) or c: this project has been abandoned.

I'm going with the latter being c: Project maintenance has been abandoned. This is a great tool, sad to see even PR's are not being merged. If anyone is looking for a working version that respects the new changes use @AetherBreeze 's forked repo here:

https://github.com/AetherBreeze/Sublist3r

Error: Virustotal probably now is blocking our requests [!] Error: Google probably now is blocking our requests [~] Finished now the Google Enumeration ...

[aleeshaford]

maybe it is possibility to decompile the inofficial VirusTotal Mobile app to have again a rate limit free scraping method without api key.

i haven't gotten that far yet, but it seems like the particular "partner" is calling urls to scan

[anurag-sachan]

works fine for me, thanks brother!

[Morgoth04]

Address #194 by modernizing the Virustotal scraper. Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds. This solves the same problem as #285, but does not require an API key and uses the existing parser logic. This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

I am new to linux and using sublist3r. please it possible to provide a step by step instruction on how I can solve the problem. Because I tried using the API key and is not working. thanks

Almost 4 years late, sorry: 1) The url line in the class corresponding to Virustotal (class Virustotal(enumratorBaseThreaded):) of the sublist3er.py program must be corrected (base_url = 'https://www.virustotal.com/api/v3/domains/{domain}/subdomains'). 2)2) Place the apikey that was obtained once registered as a virustotal user in the variable sel.apikey (self.apikey = os.getenv('VT_APIKEY', 'hereenterthealphanumeric code')) 3)Save the file 4) Overwrite the sublist3r.py file in the Kali Linux path where the one you have was placed 5) Try with the command sudo ./sublist3r.py -d domainchosen.com