API versioning spec

[touilleMan]

What is the API ?

The API refers to everything exchanged between Parsec clients or between Parsec client and Parsec server.

The different layers are:

• Protocol API: Contains the commands available to communication between the client and the server (e.g. user_create, vlob_update).
• Transport API: Contains all the mechanism to be able to send command (i.e. you need to setup a transport before using the protocol).
• Data API: Contains the data with a persistent existence (i.e. stored in the server and consulted by the clients and/or the server in the future).

api layer	stored	example
data	yes	FileManifest, UserCertificate
protocol	no	vlob_update, user_create
transport	no	Websocket + handshake, HTTP POST + authorization headers

What about local data ?

Data with only a local persistency (i.e. LocalManifest) are not currently part of the Data API.

The rational is to consider they are much easier to migrate to a new format: when a Parsec client starts it can choose to reuse the local data (regular case), apply a conversion to the local data (upgrade path 1) or to simply remove/ignore the incompatible data (upgrade path 2).

Upgrade path 1 is typically what is done when a change in the local data format is released while upgrade path 2 is taken when starting a new client on much older data (where the upgrade has been dropped) or an older client on newer data (i.e. rollback to a previous Parsec version).

However this is far form perfect:

• Ignoring older data works fine with cache, but not so much for not yet synced data.
• Ignoring older data just doesn't work for DeviceFileSchema (hence why we still support LegacyDeviceFileSchema)
• Local data store API data schemas, hence Data API evolve capacity is tied to local data's.

So in a nutshell things work fine as long as Data API keeps as strict backward compatibility (i.e. only adding optional fields that can be omitted without changing the meaning of the rest of the data).

This is what has been observed so far, hence why the Data API has no version/revision information (unlike for the Transport/Protocol API), hence why we considered the local data as not part of the API.

Current status

As we said, Data API has no version/revision information, so full forward (client in version n can read data generated by client in version n-1) and backward (client in version n generate data that can be can read by client in version n-1) compatibility is expected.

Regarding Transport/Protocol API, parsec/api/version.py defines version/revision supported:

https://github.com/Scille/parsec-cloud/blob/7ba22e3ade40a4c28996a3f0e84f3467d2f5cfec/parsec/api/version.py#L11-L14

This is used during API handshake:

• when the client connect to the server in Websocket, the server return a handshake challenge message
• handshake challenge message contains the list of API versions supported by the server (typically [(1, 3), (2, 7), (3, 0)])
• the client compare this list to the API versions it itself support and select the highest compatible API version (with previous example, if client supports [(1, 3), (2, 9), (4, 0)] it will select 2.9)
• the client return an answer message to the server which contains the API version used by the client (so 2.9 in our example) and additional content that depend of the API version used.

Currently there is three major versions of the Transport/Protocol API:

• version 1 that is mostly deprecate (only used to boostrap organization since at least Parsec v2.0, not used anymore since Parsec v2.9)
• version 2 which replaced the enrollment commands
• version 3 that introduced an improve the handshake system

Breaking/non-breaking changes

So each time a change in the API breaks existing client we provide it a new major API version. On top of that, each time a modification on the API that extend without breakage, the revision of the API is bumped.

It should be noted that client and server don't have a symmetrical role when dealing with compatibility: if client and server have different API revisions we consider it's the client job to handle the differences.

If client is older than server (e.g. client in APIv2.3 vs server in APIv2.10):

• The server always serve an APIv2.10 and don't care about client API revision
• The client send 100% valid command request to the server (given client's APIv2.3 is a subset of server's APIv2.10)
• The client however may receive unexpected response status from the server, it is expected to handle them in a generic way.
• The client may also receive a command response with an expected status, but with additional fields. In such case those fields are simply discarded during deserialization.

If client is newer than server (e.g. client in APIv2.3 vs server in APIv2.0):

• The server always serve an APIv2.0 and don't care about client API revision (not like it can do anything else ^^)
• The client has to take care sending only commands present in APIv2.0, or be prepare to deal with unknown_command status returned by the server. Similarly it can send command with newer fields but they will just be discarded by the server when deserializing the command request.
• The client receive 100% valid command response from the server (given server's APIv2.0 is a subset of client's APIv2.0)

Non breaking changes

• adding a new command
• adding to an existing command request a new optional field that can be omitted without changing the command overall meaning
• adding a new status to a command response
• adding to an existing command response a new optional[^1] field that can be omitted without changing the command overall meaning

[^1]: In theory a new required field can also be added to the command response. But the field appear pretty much optional from the client point of view: it is always present on newer servers and never present on older ones.

Breaking changes

• removing a command
• modifying a field type in a command request/response
• removing a field from a command request/response
• making a mandatory field optional in a command request[^2]/response

[^2]: In theory this is a non-breaking changes, though it is pretty much useless given the client has to handle compatibility with older server (hence seems just simpler to always send the field). On top it seems rather odd that turning a field optional would not modify the overall meaning of the command (the field should basically not be used in the first place).

Example

Considering the following fictional API changes:

api version	change
1.0	add user_get
1.1	add user_create, user_get can return a new not_found status
1.2	modify user_get to add a page parameter
2.0	remove user_get

We would have:

	server v1.0	server v1.1	server v1.2	server v2.0
client v1.0	✅	✔️	✔️	❌
client v1.1	❔	✅	✔️	✔️
client v1.2	❔	❔	✅	✔️
client v2.0	❌	❌	❌	✅

With:

• ✅ Perfect version match, nothing to do
• ✔️ API revision provided by the backend is more recent, almost nothing to do: the client don't use the new feature it doesn't know about, and unknown error types returned by the server are considered valid by the client and handled in a generic way.
• ❔ API revision provided by the backend older recent, the client must be careful not to use too recent commands or parameters (in our example user_get's page parameter and/or user_create)
• ❌ Incompatible API

[FirelightFlagboy]

the Data API has no version/revision information [...] we considered the local data as not part of the API.

Even if the data is not part of the Api, I recommend to add a version field to that data.