Provider ABI testing

[t-j-h]

Background:

We need to expanding our testing process to cover the ABI guarantee we made for 3.0 for providers. Some issues have been noted in #19171 Partial fixes are in #19201

All provider modules for all prior releases are meant to be able to work seamlessly with later releases. It was a fundamental part of the provider design for 3.0. We aren't testing this well enough currently.

Proposal:

Prior to any future release, we will confirm the provider ABI works for all prior released versions of provider modules (currently fips and legacy).

[t-j-h]

Vote [+1]

[mattcaswell]

Vote: [+1]

[iamamoose]

Vote: 0

[kroeckx]

Why isn't this something for the OTC to decide?

[t-j-h]

Why isn't this something for the OTC to decide?

Because there was confusion in terms of understanding the core objective here of OpenSSL 3.0 and this objective was specified by the OMC. If the OTC itself had decided in its weekly meeting that testing of this manner is something that gets added into the technical release criteria then an OMC vote might have not been necessary - but that was not decided - and the sense of urgency and the scope did not match the criticality of the core objective. This is something that some felt was already being tested and that nothing new needed to be done or that the proposed PR went far enough or that this was just a FIPS specific thing.

That was (and is) incorrect in terms of covering a core objective of the release. This testing does come at a cost. Hence the vote to make it unambiguous that this must not be missed in testing and that we need to put in place a setup that ensures we meet this objective for all future releases (and that implies we need to sort out any current issues).

It was also discussed in the OMC meeting that we haven't been as clear as we could in terms of a procedure that our users should follow when building provider modules from earlier releases to use with a later release - what set of steps should be followed - as different users are doing different things and may be getting surprising outcomes depending on the order of the steps.

[paulidale]

Vote: -1

The wording here is far too loose. We have a release policy and the vote's text should match the terminology used there. As it stands, providers built as part of alpha and beta releases would need to be included in this testing which is not the intention.

I'm also concerned about the ever sheer quantity of testing mandated & how it will grow over time.

[t-j-h]

@paulidale I think you have missed the context - in that we have defined the terms in https://www.openssl.org/policies/general/versioning-policy.html such that alpha a beta are pre-releases not releases.

The context was clear in our OMC meeting discussion and I think the wording remains clear and consistent and the OTC will turn the decision into the more detailed working in the release requirements document. At the OMC level this wording is correct and consistent with our definitions.

[paulidale]

I haven't missed the context. I completely understand where this is intended to go and I support that direction.

There are just too many problems with the vote as it stands. It is still far too lose and doesn't cover anything like enough IMO:

• Every future major version must maintain strict provider compatibility with all earlier versions, regardless of how old. This is despite major versions allowing ABI breaks. This is unworkable IMO.
• It doesn't cater for broken releases which are unadoptable. This would normally be an OTC call but not with this vote in place -- the OTC would need to recommend the OMC to have a vote to violate this vote.
• It gives the OTC absolutely zero discretion as to what is permitted or not: it's everything unconditionally.
• It presents only half of the mandates for compatibility which the 3.x design was meant to encompass. FIPS providers were designed to move forward and backwards across OpenSSL versions -- this only cover forwards. It must cover both directions.

[t-j-h]

At some stage we will tackle the other aspects - but right now the vote covers the specific problem we have. Things will get adjusted over time - we will figure out a balance. Votes are not locked in stone - in that we can vote to adjust. Waiting for a perfect vote is not the right approach in this and many other contexts in my opinion.

This fixes the immediate issue and prevents the immediate issue from coming back for the foreseeable future. I do expect the details to get worked out to a finer level over time. What I don't want to see is arguments about whether or not we should actually be testing even to this level - and that was the reason for calling this vote - so it is clear that at a minimum we must do this now.

Right now we haven't decided for how long a 3.0.0 provider needs to work for future releases and that isn't something we can or will decide now in terms of a shorter time period or end of life details for this. That is something we will only learn over time.

If a release occurs that breaks this then it should be a discussion between OTC and OMC as to the path forward with OTC recommending the technical path and OMC considering the broader context than the technical path. That is what is meant to happen.

Everything you are raising here is all reasonable and acceptable outcomes of the vote for the current context and making a vote down the track that deals with these issues is what is expected. We can (and have) voted the exact opposite of previous votes when we get more information / experience and the objectives shift.

[kroeckx]

Voting -1

[levitte]

Vote: 0

[t-j-h]

Vote failed.

[paulidale]

Process concern: this vote was never properly recorded. The caller is responsible for doing this on close.