AGWA
commented
2 years ago There's not much certspotter can do about the fact that CT logs are huge, though we do advocate for policies (e.g. sharding by expiration year, retirement of excessively large logs) that reduce the burden on log monitors.
To reduce the amount of data you have to download the first time you run certspotter, use the -start_at_end option, which tells certspotter to start monitoring at the end of the log rather than the beginning, though you could miss certificates this way.
Hi, this isn't an issue, but because I didn't immediately find something I naively started running this until I questioned me again how long it'll take while it had fetched about one TiB of data.
I found some papers. These indicate that we can expect an estimated size of roughly 50 TiB as of now (16 TiB until 2018 + 44 GiB/day[1]). This means it'll take 6 days with constant download of 100 MiB/s (which also requires sufficient CPU power). If you have more precise information or think I've written garbage let me know.
[1] https://www.inforsec.org/wp/wp-content/uploads/2020/04/CCS2019-Li.pdf