Ratelimit countermeasure for STARTTLS in mailtest

[gthess]

For the STARTTLS test some mailservers use strict ratelimits on the number and the rate of connections used when testing.

The current tests are run in series to avoid the limit on concurrent connections. However stricter limits on the rate of connections were observed at roughly 2 allowed connections per minute. The only way to circumvent that in the future is to try the different STARTTLS tests with a pool of IP addresses.

[baknu]

See also: https://github.com/internetstandards/Internet.nl/issues/658

[baknu]

It might also help to slow down the rate when a domain is retested within a given timeframe (e.g. 5 minutes).

[mxsasha]

It might also help to slow down the rate when a domain is retested within a given timeframe (e.g. 5 minutes).

I did see we already do some caching for repeat scans for the same mail server, which makes sense as many domains use the same small set of servers and we don't need to retest for every domain.

[baknu]

I did see we already do some caching for repeat scans for the same mail server, which makes sense as many domains use the same small set of servers and we don't need to retest for every domain.

Yes, that's right (see also #171). I am sure we use MX result caching on batch. However, I am not sure if we also use that on the single scan version. Furthermore, MX result caching can still be improved (see #670).

[mxsasha]

However, I am not sure if we also use that on the single scan version. Furthermore, MX result caching can still be improved (see #670).

I checked, we do.

[mxsasha]

Moving this off 1.8 for now, but to summarise, there are three fundamental approaches:

• Caching on IP basis to reduce repeat connections for frequently used hosts (attempt was made in #889 but would break DANE, also check if SNI is a factor).
• Testing from an IP pool, but it would ideally be a set of unrelated subnets. This is infrastructurally complex.
• Slowing down the test on a retry if it seems to be rate limited. This will affect load balancing and job/queue management.

[bwbroersma]

Related:

• 339

After new TLS implementation, recheck if there are more/less rate limit problems.