cisagov / admiral

Distributed certificate transparency log harvester
Creative Commons Zero v1.0 Universal
14 stars 3 forks source link

Stabilize Admiral #21

Closed king-alexander closed 2 years ago

king-alexander commented 2 years ago

πŸ—£ Description

The goal was to keep Admiral running in a steady state. That end precipitated two primary changes: (1) setting a default rate limit for Celery tasks; and (2) adding timeouts to certificate requests.

πŸ’­ Motivation and context

The changes made in this PR resolve #16. Admiral no longer crashes on domains with a large number of certificates, but still struggles to process them. Admiral can be further stabilized by reducing the amount of time these queries take. I opened #20 to track work on this improvement.

πŸ§ͺ Testing

With these changes in place, I could process ahrq.gov -- the domain Admiral previously crashed on. And I have continued to run Admiral since without a crash. So far, I have processed 138 domains.

βœ… Pre-approval checklist