search

elceef / dnstwist

Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation
https://dnstwist.it
Apache License 2.0
4.73k stars 753 forks source link

Feature: Allow to increase `WEBDRIVER_TIMEOUT`? #212

Open olifre opened 5 months ago

olifre commented 5 months ago

Currently, the WEBDRIVER_TIMEOUT seems hardcoded in dnstwist.

This causes flakiness with one of the pages I test, which sometimes has very complex javascript-rich content (sadly), and dnstwist errors out with:

fetching content from: http://uni-bonn.de/ > https://www.uni-bonn.de/en [110.4 KB]
rendering web page: http://uni-bonn.de/
Message: timeout: Timed out receiving message from renderer: 4.154
  (Session info: headless chrome=116.0.5845.180)

Would it be possible to add an option to increase this timeout when required?

(I hope this is the correct timeout I am looking at)

olifre commented 5 months ago

Note: In case you want to reproduce, this was executed using the latest official docker container (phash flavour), on GitHub actions. The same test worked fine a day before, so it seems to me that this is related to site complexity and server performance on the site end, which could match hitting a timeout.

elceef commented 5 months ago

Although I wasn't able to reproduce this error with that particular domain name, I agree this might be "fixed" with setting a longer timeout in the web driver. I will figure out a way to make it configurable - probably with environment variables.

olifre commented 5 months ago

Thanks! In fact, I also can not reproduce with that domain "now", it seems to be time-dependent — I likely hit a period of high load when running my tests. So having this configurable (environment variables is fine of course) for "flaky servers" seems like a good "solution".