Better Timeout Policies

[s-rah]

Some of the new improvements e.g. spider/ and bitcoin changes have dramatically increased the timing expectations for certain sites. For example scanning for onion peers in bitcoin takes a rather long time and a user configuring that and a small timeout should likely be warned that it is a bad idea.

On top of that, we need to put some thought into why timeouts exist and how they can be helpful. Some thoughts:

• Some sites are really really slow but for large scans we want to wait for them.
• If the first protocol scan succeeds we probably want to ignore timeouts and keep checking since we know at least something is there.
• As said above, some protocol scans are really slow and can potentially contradict and confuse user specified timeouts.
• Currently timeouts are applied at two levels - on a web page basis and on a scan basis as a whole - this makes actually predicting how long a scan will take is pretty difficult.

[laanwj]

As said above, some protocol scans are really slow and can potentially contradict and confuse user specified timeouts.

Yes, things are becoming slow by default, I'm partially to blame for that :) The reason the various bitcoin scans are so slow (when they connect) is that a node can take significant time to reply to a getaddr message, as addr notifications are queued up and handled periodically. Even the currently hardcoded 30 seconds deadline is sometimes not enough to get the result. It would make sense to make that configurable, or depend on some global setting.

[laanwj]

If the first protocol scan succeeds we probably want to ignore timeouts

Another potential idea would be to distinguish the error codes returned by the proxy. Tried out a bit:

• SOCKS5_NOT_ALLOWED (proxy error 2) is returned when hidden service authentication is used, and an incorrect or no auth cookie was provided.
• SOCKS5_HOST_UNREACHABLE (proxy error 4) is returned when the host could not be resolved. I don't think this can happen for .onions, but can when contacting other hosts.
• SOCKS5_CONNECTION_REFUSED (proxy error 5) is returned when the onion was found, but the port was closed.
• SOCKS5_TTL_EXPIRED (proxy error 6) is returned when the .onion was completely unreachable

Oct 05 15:25:32.000 [notice] Tried for 120 seconds to get a connection to [scrubbed]:8333. Giving up. (waiting for rendezvous desc)

In which case a timeout would likely trigger sooner. But at least "connection refused" could be used as a flag to know for sure that the port was closed and it was not some transient Tor issue.

For golang's socks implementation these are mapped to errors here.