Improve facilities for wire protocol versioning

[paulhowardarm]

Summary

We need to tweak the way wire protocol versioning is handled. At the moment, it is effectively hard-coded to 1.0. This is fine since there aren't any other versions yet, but we want to make the system flexible enough so that we don't have to shy away from making new versions.

Detail

At the moment we already have a good abstraction layer between the raw wire layout of request/response headers, and the semantically-modelled headers that are used in most parts of the service. This is good, because the semantically-modelled header objects (RequestHeader and ResponseHeader in the interface crate) are effectively version-neutral, and can be changed incrementally to bring in new fields. So we already have a good way of preventing wire protocol changes from polluting the entire system.

The main thing missing is a decision point where we consume the invariant header fields. The wire protocol versioning requires the first four header fields to be invariant across all versions: the magic number, the header size, the major version number and the minor version number. Common code can be written to consume these four fields, and it will hold for all time. Having consumed these four fields, we then know the version number of the incoming request, and we can have a branch point. This branch point can select variant code or raw modelling structures to consume the version-specific remainder of the wire header.

Since there can be multiple distinct raw wire header structs, we might need some kind of uniform interface that they all implement to convert their fields into or out-of the semantically-modelled structures. We currently do this using standard Rust conversion primitives. I'm not sure whether that scales to multiple versions.

Of course, we also need to make sure that any response is always written to the same protocol version as the matching request. We may be doing that already. But if not, it might be something that has to be tracked in the FrontEndHandler.

Definition of Done

There should exist some tests that can exercise the system using more than one wire protocol version, and validate that the requests can successfully be processed, and also that the responses are correctly encoded. If we defer this work until there is a concrete use case for a new protocol version, then we can use that new version in the testing. However, if we want to do this work in advance of any new version being defined, we will have to invent a test-only "dummy version" (maybe something less than 1.0) to prove the mechanism, while 1.0 continues to be the only version used in real deployments.

[paulhowardarm]

Raising this issue in the master repo because I'm not sure how many parts would necessarily be touched. The work might be limited to the parsec-interface-rs repo, or it may cut across both.

[ionut-arm]

@paulhowardarm - do you think this should be done prior to the service being used in production somewhere? I think since we can guarantee that the wire exchanges will remain the same, compatibility would be preserved between implementations with and without this support for versioning.

[paulhowardarm]

@ionut-arm No, I don't think this is necessary for production use. It's not necessary until we have a proven need to introduce a new version. You're quite right - just doing this work does not impact wire compatibility.