search

SSLMate / certspotter

Certificate Transparency Log Monitor
https://sslmate.com/certspotter
Mozilla Public License 2.0
983 stars 84 forks source link

Error and Lack of Results #71

Closed jrotunno closed 1 year ago

jrotunno commented 1 year ago

I've been running Certspotter for about two weeks and it has found only a handful of certs for the watched domain that actually has close to 100 of them. I'm not sure if this is related, but I see that over 800 of the following messages have been generated:

2023/10/04 10:05:33 error downloading entries from https://sabre2025h2.ct.sectigo.com/: GET https://sabre2025h2.ct.sectigo.com/ct/v1/get-entries?start=20217&end=21216: 400 Bad Request (Bad Request need tree size: 20218 to get leaves but only got: 20217

The URLs are all related to Sectigo/Comodo:

https://mammoth2024h1.ct.sectigo.com/ https://mammoth2024h2.ct.sectigo.com/ https://mammoth2025h1.ct.sectigo.com/ https://mammoth2025h2.ct.sectigo.com/ https://sabre2024h1.ct.sectigo.com/ https://sabre2024h2.ct.sectigo.com/ https://sabre2025h1.ct.sectigo.com/ https://sabre2025h2.ct.sectigo.com/ https://sabre.ct.comodo.com/

I also see about 100 of these messages, but it doesn't sound like it should be a concern:

certspotter has been unable to download entries from https://ct.googleapis.com/logs/argon2023/ in a timely manner. Consequentially, certspotter may be slow to notify you about certificates in this log.

The URLs in these messages are:

https://ct.cloudflare.com/logs/nimbus2023/ https://ct.cloudflare.com/logs/nimbus2024/ https://ct.googleapis.com/logs/argon2023/ https://ct.googleapis.com/logs/eu1/xenon2024/ https://ct.googleapis.com/logs/us1/argon2024/ https://ct.googleapis.com/logs/xenon2023/ https://oak.ct.letsencrypt.org/2023/ https://oak.ct.letsencrypt.org/2024h1/ https://sabre.ct.comodo.com/ https://yeti2024.ct.digicert.com/log/

AGWA commented 1 year ago

Hi @jrotunno, Certificate Transparency logs contain a huge number of certificates (hundreds of millions to billions) so it's going to take a very long time to download all of them to find the 100 certificates that you are looking for. Each "unable to download in a timely manner" warning shows the backlog, which is the number of entries in the log which have not yet been consumed. You can look at how this number has been changing to get an estimate of how much longer you have to go before you're caught up with the end of the log.

As long as the backlog is decreasing you can ignore the "error downloading entries" messages as certspotter automatically retries failed requests.

You may want to consider using SSLMate's Certificate Transparency Search API if you need to quickly enumerate all of the certificates for a domain.