Mitigation options for updating being so slow

[scolby33]

Ask Censys for a higher rate limit

Probably the easiest option and it seems likely to be granted, but we can be more clever!

Use some heuristics to determine update frequency on a per-responder basis.

Run the update script/task in a continuous loop. If all of a responder's last n results have been good, don't query it again for, say, an hour. If there are multiple statuses in the last n tests, query it every 5 minutes or so. If all of the m>>n results are bad and the responder isn't current, only query it once a day or so as it may be permanently down.

[cthoyt]

so you need to keep track of this sort of information in another column in the table?

[scolby33]

No, just do query(whatever).join(as necessary).groupby(as necessary).limit(5)

[scolby33]

Censys has upgraded my rate limit. In the future, we will move to BigQuery to lessen the impact on their infrastructure.