search

SSLMate / certspotter

Certificate Transparency Log Monitor
https://sslmate.com/certspotter
Mozilla Public License 2.0
955 stars 83 forks source link

Add option to scan all logs from all time (not just new logs) #5

Closed konklone closed 7 years ago

konklone commented 8 years ago

I can't get the certspotter client to do an -all_time scan. It ends immediately, only checking for newer certs:

$ certspotter -state_dir /opt/certspotter -all_time -verbose

certspotter: https://ct.googleapis.com/pilot: 2016/07/30 01:53:54 Existing log; scanning 0 new entries since previous scan (previous size 24395383, previous root hash = 141b4982f720f248cd8190f46a914643933041e045e27a98700470a2947c7411)
certspotter: https://ct.googleapis.com/pilot: 2016/07/30 01:53:54 final log size = 24395383, final root hash = 141b4982f720f248cd8190f46a914643933041e045e27a98700470a2947c7411
certspotter: https://ct.googleapis.com/aviator: 2016/07/30 01:53:54 Existing log; scanning 0 new entries since previous scan (previous size 23120116, previous root hash = d1f305aba4234e39e26d00657c234f785d3edf370833042c1ab56454cdea7de3)
certspotter: https://ct.googleapis.com/aviator: 2016/07/30 01:53:54 final log size = 23120116, final root hash = d1f305aba4234e39e26d00657c234f785d3edf370833042c1ab56454cdea7de3
certspotter: https://ct1.digicert-ct.com/log: 2016/07/30 01:53:54 Existing log; scanning 0 new entries since previous scan (previous size 442176, previous root hash = fb00226b2e2ad19ed68b84f3d64399ee25dee4228700274117c7024d1f66ecf3)
certspotter: https://ct1.digicert-ct.com/log: 2016/07/30 01:53:54 final log size = 442176, final root hash = fb00226b2e2ad19ed68b84f3d64399ee25dee4228700274117c7024d1f66ecf3
certspotter: https://ct.googleapis.com/rocketeer: 2016/07/30 01:53:55 Existing log; scanning 0 new entries since previous scan (previous size 21952056, previous root hash = 51cf0dd50b08de1125104cbeff458b5f89e95fafcaee1c502cb467e9a0af6de8)
certspotter: https://ct.googleapis.com/rocketeer: 2016/07/30 01:53:55 final log size = 21952056, final root hash = 51cf0dd50b08de1125104cbeff458b5f89e95fafcaee1c502cb467e9a0af6de8
certspotter: https://ct.ws.symantec.com: 2016/07/30 01:53:55 Existing log; scanning 0 new entries since previous scan (previous size 1034913, previous root hash = ebb8f8d4faec771b572fb2607a39db43fbfa44b28155650f1ecf0945ff115505)
certspotter: https://ct.ws.symantec.com: 2016/07/30 01:53:55 final log size = 1034913, final root hash = ebb8f8d4faec771b572fb2607a39db43fbfa44b28155650f1ecf0945ff115505
certspotter: https://ctlog.api.venafi.com: 2016/07/30 01:53:55 Existing log; scanning 0 new entries since previous scan (previous size 53969, previous root hash = 68659773bbcf01f5df14122bfac7d5a80c63ea61ec64de5a603d3e88306a9c2e)
certspotter: https://ctlog.api.venafi.com: 2016/07/30 01:53:55 final log size = 53969, final root hash = 68659773bbcf01f5df14122bfac7d5a80c63ea61ec64de5a603d3e88306a9c2e
certspotter: https://vega.ws.symantec.com: 2016/07/30 01:53:55 Existing log; scanning 0 new entries since previous scan (previous size 3440, previous root hash = ff13a267dd08a64a311b8b813362ead1f23ca3237310fec562f26044abdaad02)
certspotter: https://vega.ws.symantec.com: 2016/07/30 01:53:55 final log size = 3440, final root hash = ff13a267dd08a64a311b8b813362ead1f23ca3237310fec562f26044abdaad02
certspotter: https://ctserver.cnnic.cn: 2016/07/30 01:53:56 Existing log; scanning 0 new entries since previous scan (previous size 1882, previous root hash = 9cdaf17577f64130c99c317817ccae27b118c16a93ca577a4d296be5d20fc0d6)
certspotter: https://ctserver.cnnic.cn: 2016/07/30 01:53:56 final log size = 1882, final root hash = 9cdaf17577f64130c99c317817ccae27b118c16a93ca577a4d296be5d20fc0d6
AGWA commented 8 years ago

-all_time is working as intended; it's just badly named and documented. -all_time only affects scans of new logs. I'll add an option to do what you expected, but I need to figure out what these two options should be called (suggestions welcome).

In the meantime, you can force a rescan of all logs by removing all the files in your ~/.certspotter/sths directory (or wherever your state_dir is) and then running certspotter with -all_time.

lvh commented 7 years ago

In the interest of documentation for people who find this issue, what is the current behavior? It appears that -all_time trusts existing logs to go back until the beginning?

lvh commented 7 years ago

(Also, what does it do without that flag?)

gerhard-tinned commented 7 years ago

I came across the same question when I tried certspotter the first time.

What i figured out is the following (please correct me if I'm wrong).

When certspotter is started without the -all_time, it reads the latest CT entries from the logs. When it is done it stores the last entry retrieved in the directory structure under ~/.certspotter/. When started the first time, it seems ad certspotter will just store the last position.

As it seems, when certspotter is started the very first time, there is no information for the last log position (for that domain and CT log). In this state, the -all_time is causing certspotter to check the complete CT log. If a last position is already there, the-all_time seems to have no effect.

When the file containing the last position is deleted, the -all_time behaves as expected.

Again, this is my observation.

AGWA commented 7 years ago

In Cert Spotter 0.3, -all_time will work as expected and always scan the entirety of logs. For details, see the commit message for 31f2316aa2cdb56fc355f181f55da3e938fd9f55.