CT client never gives up, leading to excessive tries when a log is down

[jsha]

In Publisher, we use github.com/google/certificate-transparency/go/client, which takes care of retries and backoffs for us. Unfortunately, it will loop forever on errors. Combined with the OCSP Updater, which will periodically call to Publisher for any certs it doesn't have an SCT for, this leads to a rapidly increasing number of outstanding requests when a log is down.

A couple possible fixes:

• Add deadlining or max tries to the upstream client.
• Make Publisher aware of which certs it is currently trying to publish, and ignore RPCs requesting publishing of a cert it is already working on.

[jmhodges]

Forked the ct client to https://github.com/letsencrypt/certificate-transparency

[jmhodges]

I think we should probably also get rid of the log statements in it. Maybe replace them with stats.

[rolandshoemaker]

Because of how certificate-transparency is structured, switching to a fork is actually kind of a pain. I think our best bet here is to wait for google/certificate-transparency#1098 to land and continue using the upstream versus switching to our fork (which would make submitting patches upstream really tricky).