mozilla / crlite

WebPKI-level Certificate Revocation via Multi-Level Bloom Filter Cascade
http://www.cs.umd.edu/~dml/papers/crlite_oakland17.pdf
Mozilla Public License 2.0
67 stars 8 forks source link

Reduce memory requirements of the aggregate-known task #280

Closed jschanck closed 1 year ago

jschanck commented 1 year ago

The redis backend stores serials sharded by issuer and expiry date, but aggregate-known concatenates all serials from a single issuer into an in-memory buffer before writing them out to disk. Since enrolling Let's Encrypt, this has caused some OOM failures in our production instance. Let's stream the serials to disk as we read shards.