search

cisagov / admiral

Distributed certificate transparency log harvester
Creative Commons Zero v1.0 Universal
14 stars 3 forks source link

Expedite queries for large result sets #20

Closed king-alexander closed 1 year ago

king-alexander commented 2 years ago

💡 Summary

Admiral struggles with domains that have large result sets (~1000+ certificates). Processing typically takes multiple hours, and will occasionally hang. We should refactor to improve performance.

Motivation and context

This improvement will make Admiral more robust, which in turn will make it easier for agencies to comply with ED 19-01.

Implementation notes

The crt.sh interface was not designed to handle large result sets. Yet we want to process these troublesome domains in roughly the same amount of time it takes to process the others. One path forward might be implementing CeRTSearcH as a task. But we should conduct more research on potential solutions first.

king-alexander commented 1 year ago

Closing as a duplicate of #33. The information herein is out of date since we switched to the Certificate Search API.