exposing locally generated roots for integration tests

[icing]

After updating to current boulder, my ocsp stapling tests fail. Reason being that the openssl client fails with Verification error: unable to get local issuer certificate.

I am giving the call -CAfile $BOULDER/test/test-root.pemand that worked until the boulder update. (I had not updated for some time, so the change might have been done months ago in your timeline.)

I assume the pem has moved somewhere else?

[alexzorin]

I think #4832 changed things around so that the root and intermediates are generated at runtime using the new ceremony tool.

PKI.md got updated as well:

The private keys are stored in SoftHSM, and the public keys and certificates are written out to /tmp

Doesn't look like they're exported out of the container by default when using docker-compose.

[jsha]

@alexzorin has got it exactly right, as usual. :-) We now generate a fresh hierarchy with each run. Sorry for the breakage! We don't currently export the certs, but I am open to ideas on how to do it for integration tests that incorporate Boulder.

[icing]

@jsha not sure I have a bright idea here. I added to my test suite code that copies docker:/tmp/*.pem into a local ca.pem and use that. The older path was reverse engineer also, so I cannot really complain. ;)

I guess, offering a ca.pem resource from some web server would work for an "official" test api endpoint. Maybe even for the live servers it wouldn't hurt?

[icing]

Coming back to this, I need an "official" point to get the root certificates for my test suite. I am prepared to do a docker cat from somewhere.

After hacking the location between different master revisions, I am currently unable to see any recent .pem in the boulder file system. Maybe its only in memory/database now?

[jsha]

The generated roots should still be in /tmp/*.pem. The call path is test/startservers.py:setupHierarchy -> test/cert-ceremonies/generate.go, which in turn loads config files from test/cert-ceremonies/*.yaml. Those yamls configure the output paths. For instance, https://github.com/letsencrypt/boulder/blob/main/test/cert-ceremonies/intermediate-ceremony-rsa.yaml#L9 outputs to /tmp/root-cert-rsa.pem. Note that there are some small shenanigans in generate.go that rewrite some output paths, but not for the root, and everything still gets put in /tmp/.

[icing]

Thanks jacob. For some reasons, the files were not there. I removed all docker things and updated everything and now it's running fine again.

Should have done that right away. Sorry for the noise.

[jsha]

No problem. Always nice hearing from you. :-)

[acharis-do]

@aarongable is there any appetite for serving the root CA cert from the docker container at runtime at a stable path? (or is this issue closed because this is done already?)

[jsha]

Yes, I think there's some appetite for that. From the discussion about, @icing was able to get it out by docker cating (presumably docker exec boulder cat /path/to/file) the files from /tmp/root-cert-rsa.pem (and now /tmp/root-cert-ecdsa.pem). Does that work for you? We could just commit to those paths being medium-term stable, since they haven't changed in a while and we don't expect them to change soon.

[acharis-do]

i think they're in /hierarchy now, rather than /tmp if i'm reading things correctly. i was hoping one of the services started by startservers would expose them at an endpoint.

[aarongable]

The integration tests now have aia-test-srv exposed on port 4502, which serves all of the dynamically generated integration test CA certs on per-name URLS. The CA certs are also now generated inside the "//test/certs/webpki" directory of the boulder repo, if that's easier than getting them from the aia-test-srv.