Add feature flags to control OCSP Responder read behavior

[jcjones]

Currently, an OCSP Responder configured with a Redis backend uses the MultiSource Source to blend the MariaDB and Redis results. As part of that blend, it compares byte-for-byte that the responses are the same.

When we roll this out, the each datacenter will have different OCSP Responses than the other, as the OCSP Updaters will be updating their own Redis instance.

Byte comparison is a good starting point, though it limits us to only deploying to the primary datacenter at present. A next phase, after proving the byte-for-byte operation, should be a feature flag to instead compare OCSP Update times.

The MultiSource would need to compare the DB row that is returned to both the OCSP Metadata to determine relative freshness, and the OCSP response data, to determine correctness. The general algorithm might look like:

# At this point, two Redis calls (Metadata and Data)  and the MariaDB row have both returned.
If redisMetadata.ocspResponseGenerationTime is newer than databaseRow.ocspResponseGenerationTime:
   if redisData.ocspResponse either parses or otherwise appears valid:
       return redisData.ocspResponse
return databaseRow.ocspResponse

Also, appropriate metrics should be updated at each leg of the way.

[jsha]

it compares byte-for-byte that the responses are the same.

To clarify, this actually does a comparison of the statuses - so the ocsp-responder won't serve a good response from Redis when the DB says revoked, but it can serve different bytes.

[aarongable]

I believe all necessary feature flags have been added:

• Stages 1, 2, and 3 added by @jsha
• Stage 4 (disabling ocsp-updater) does not require a feature flag
• Stage 5 no longer exists (it got rolled into Stage 3, above)
• Stage 6 implemented by me
• Stage 7 is just cleanup, no feature flags, and is tracked in Issue https://github.com/letsencrypt/boulder/issues/6285