search

smicallef / spiderfoot

SpiderFoot automates OSINT for threat intelligence and mapping your attack surface.
http://www.spiderfoot.net
MIT License
12.93k stars 2.25k forks source link

Feature request: adjustable timeout or module retries #239

Closed mavensecurity closed 5 years ago

mavensecurity commented 5 years ago

Sometimes modules simply timeout like this:

modules.sfp_phishtank | INFO | Fetching: http://data.phishtank.com/data/online-valid.csv [timeout: 5]

In this case it resulted in a false negative; running the scan again solved the problem (the module ran to completion, and a finding was generated since a site within the seed domain was in Phish Tank). Since the only signs of trouble are in the Log it may not be obvious to the user this scan is missing data.

Re-running an entire scan (or creating a new scan with just the timed out modules) is inefficient.
A user-defined / adjustable timeout would be very helpful in preventing this on slower/busy systems. Or maybe it would be better to have modules that timeout retry a set number of times, and use a Log Type of WARN if they ultimately timeout.

smicallef commented 5 years ago

The phishtank module uses the global timeout config, which you can set in the Settings option "Number of seconds before giving up on a HTTP request."