Closed scolby33 closed 6 years ago
so you need to keep track of this sort of information in another column in the table?
No, just do query(whatever).join(as necessary).groupby(as necessary).limit(5)
Censys has upgraded my rate limit. In the future, we will move to BigQuery to lessen the impact on their infrastructure.
Ask Censys for a higher rate limit
Probably the easiest option and it seems likely to be granted, but we can be more clever!
Use some heuristics to determine update frequency on a per-responder basis.
Run the update script/task in a continuous loop. If all of a responder's last
n
results have been good, don't query it again for, say, an hour. If there are multiple statuses in the lastn
tests, query it every 5 minutes or so. If all of them>>n
results are bad and the responder isn't current, only query it once a day or so as it may be permanently down.