search

google / certificate-transparency

Auditing for TLS certificates.
https://certificate.transparency.dev
Apache License 2.0
869 stars 283 forks source link

scanner.go: out of memory #1387

Closed gliwka closed 6 years ago

gliwka commented 7 years ago

Hi,

i've tried to use the scanner.go implementation to fetch a large number of certificates stored in a ct log and save them into a csv file (for code see here). After some time i receive the following error message:

2017/05/02 16:11:36.017863 scanner.go:420: Processed: 66147 certs (to index 66147). Throughput: 1836.16 ETA: 14 hours 51 minutes 57 seconds fatal error: runtime: out of memory

The go process takes up 15GB and then crashes due to no more memory being available on my system.

I've never written much in golang, but suspect the context.TODO() to be at fault here: https://github.com/google/certificate-transparency/blob/master/go/scanner/scanner.go#L281

Best, Matthias

taknira commented 7 years ago

Hi Matthias,

We've just started looking into this for you, but could use a little bit more information.

When you are running the scanner, what flag values are you passing to it?

Thanks! Kat

taknira commented 7 years ago

I've just realised you did post a link to the code you're running, which contains the answer to my question! Looking at it now...

gliwka commented 7 years ago

@pphaneuf @taknira Is there something new on this?