search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.21k stars 24.84k forks source link

Improve support for customizing TLS environment in test clusters #111532

Open DaveCTurner opened 3 months ago

DaveCTurner commented 3 months ago

Today tests which use TLS work in terms of key pairs[^1], certificates, key stores and trust stores, which are committed to Git as source files and largely treated as opaque blobs of data. Yet these things are not really opaque source files, they could be generated at test runtime. This would let us be more specific in the test source code about which properties of these artifacts matter and which can be varied randomly, which would let us improve our coverage of the various configurations that users create in practice. I claim (without evidence) it would also help the dev team understand our interactions with TLS better.

The most obvious example of this is the hard-coded expiry dates in all the certificates we use in tests which will at some point cause those tests to stop working. If a test needs a valid certificate, we should be generating it to avoid the time-bombs we've left for ourselves all over the codebase.

The JDK supports many formats for certificates, key stores and trust stores, but we are only really testing one combination thereof. If we generated these artifacts at test runtime then we could choose between any of the various supported formats (which depend on JDK version and other environmental factors).

I recently introduced org.elasticsearch.test.TestTrustStore (#111228) as a simple mechanism for constructing a trust store at runtime in order to avoid hard-coding such a trust store in the tests. My approach was rather naive, and while it worked ok for the tests that run during the intake build I could not (easily) make it compatible with FIPS clusters since these clusters already specify a different hard-coded trust store, using a different format. This mechanism also does not work with clusters running on JDK17 (see #111569 for suboptimal workaround). These seem to me like problems we should be able to solve in the test infrastructure.

[^1]: Key pairs are expensive to generate, contain very little metadata, and do not expire, so I think it'd be ok to hard-code them in the source rather than generate them all at test runtime. But it'd be nice if we loaded them from the source and wrote them out in different formats still rather than just exposing the PEM file(s) verbatim to the test cluster.

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-security (Team:Security)

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-delivery (Team:Delivery)