RFC: Separate versioning of data types

[kjetilkjeka]

Summary

Instead of talking about DSDL versions and stabilization of DSDL as a whole, this RFC suggests the alternative of versioning data types individually. This will ease the task of stabilization, allowing it to happen sooner, giving a more flexible way to describe compatibility between DSDL, and enabling a way to accept bit compatible data types without freezing data type signatures [#4]. Some robustness is also gained by having ways to check if single frame data types have conflicting definitions.

Motivation

There is an obvious need to stabilize DSDL. Every time a message is changed every software depending on it breaks. There is also a need to not prematurely stabilize. Being stuck with badly functioning data types will be the consequence, and it won't be fixable until next major version.

With the current system (waiting for dsdl 1.0) no stabilization is realistically within reach. For every battle proven data type, there is a data type that is not yet ready as well. By keeping on adding data types when it is convenient a dsdl stabilization is further pushed in the future as well.

By accepting separate versioning of data types, these problems should be much easier to handle. The proven parts of dsdl can be stabilized while allowing rapid development of the non-stabilized parts. This also allows software that only relies on stable messages to be stabilized, this is a huge advantage for uavcan users.

The data type signatures are really great! They let you know when you're using the same data types, and even enforce this programmatically for multi frames. The problem is that when you do have a miss match, it's not trivial to find out which unit is erroneous. Separate versioning of data types give us tools to talk about the version of the messages. For instance, if a unit (outside our control) states that it sends 0.2 frames and we know that we currently accept 0.3 frames, we will be much closer to fix the problem. It will also make it possible to (non-automatically) detect different single frame transfers, this would not otherwise be noticed as no CRC is used.

There are a lot of things that can go wrong when stabilizing DSDL. By stabilizing one data type at a time we make eventual "screw ups" less possible and more handleable. By combining this with using gained knowledge from previous stabilizations to stabilize new data types, stabilization should have a much lower chance of causing headaches.

Explanation

The approach to versioning is based upon the proven idea of semantic versioning but adapted to make sense for dsdl. The version information is encoded directly in the name of the file where the definition is contained such as <ID>.<Name>.<Major version>.<Minor version>.uavcan. For example 341.NodeStatus.1.0.uavcan

• A change that respects bit compatibility, and doesn't break code compatibility is allowed to be done by incrementing minor version. This holds even if the data type signature is changed.
• All other changes must increment the major version.
• The exception to this is if the more important version numbers are zeros. E.g. 0.2 is allowed to break systems dependent on 0.1.
• Version 0.0 is special since no data types tagged with this version number should be considered the same or compatible in any way (everything goes).

Examples

Let's use the following frame as an example

#version 1.0
uint2 state
void2 reserved1
void4 reserved2

If we were to add a comment we would have to increment the minor version (this doesn't change data type signature nor bit compatibility)

#version 1.1
uint2 state # i like comments
void2 reserved1
void4 reserved2

In the same way, adding constants would only require an increment of the minor version

#version 1.2
uint2 STATE_IDLE = 0
uint2 STATE_TRANSMITTING = 1
uint2 STATE_RECEIVING = 2
uint2 STATE_ERROR = 3
uint2 state # i like comments

void2 reserved1
void4 reserved2

But removing/changing a constant would require a major version (since it breaks backward compatibility)

#version 2.0
uint2 STATE_OK = 0
uint2 STATE_STARTING = 1
uint2 STATE_ERROR = 3
uint2 state # i like comments

void2 reserved1
void4 reserved2

Changing the name of fields or constants would not retain code compatibility and will also require a major version increment.

#version 3.0
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # i like comments

void2 reserved1
void4 reserved2

Removing/editing a comment will both keep data type signature and code compatibility, thus only requiring minor version increment

#version 3.1
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

void2 reserved1
void4 reserved2

Using new fields in a bit compatible way or changing the name of void fields only require a minor version update.

#version 3.2
uint2 STATUS_OK = 0
uint2 STATUS_STARTING = 1
uint2 STATUS_ERROR = 3
uint2 status # This is the status

uint3 STATE_IDLE = 0
uint3 STATE_TRANSMITTING = 1
uint3 STATE_RECEIVING = 2
uint3 STATE_ERROR = 7
void3 state

void3 reserved

Implementation

This feature should be regarded as a low impact feature. The feature itself will not change the generated code. Meaning that by treating files without version as 0.0 (and forbidding explicit version 0.0) this change will be fully backward compatible. This means that for example 341.NotStatus.uavcan "really" means 341.NodeStatus.0.0.uavcan, while explicitly writing 341.NodeStatus.0.0.uavcan will result in an error.

Open questions

How do we handle versioning of data types that uses other data types?

In some way or another we will have to specify the exact version the data type is dependent on. One way to do this would be as following

@use Command version 0.2
Command[<=15] commands

Another alternative would be

Command.0.2[<=15] commands

The following rules must be adopted

• You must not have a stable data type depending on an unstable data type
• When updating to a dependency with higher major/minor version, you must increment the major/minor version
• The exception beeing that in version 0.0 you're not required explicitely list the dependency. This is required to ensure backward compatibility..

Further work

• A way to accept bit compatible frames (#36)
• Creating CI tests to see if the rules of versioning are upheld.

[pavel-kirienko]

What about calling it "data type id lower bits (DSDL data type version)" and first say that this field is for the message type arbitration for the "representation layer" and then concertize how it's used in DSDL?

I don't fully grasp yet what is the best way to reflect that in the specification. Adding an extra bit of indirection doesn't seem to contribute to clarity. The specification is flexible, we can change its wording and formulations later while keeping its implementations fully compatible. Meaning that if there is a need to decouple the lower bits from versioning, it can be done in v1.1 rather than in v2.0.

[thirtytwobits]

Okay. As long as we paint it blue I'm in.

I'll back away with this one nit-pick:

Should we require reserved bits to be recessive? I know the standard prefers dominant reserved bits but for bits in the identifier, if we always have the reserved (i.e. older) version dominant then these bits will always be higher priority then any new uses. It seems like we would want the newer versions to be higher priority. Again, a real nit-pick since these aren't priority bits in the specification and we're looking into the future here which is always dangerous. Either way I'm good.

[pavel-kirienko]

I proposed zero because I expect that in the future we may want to re-use them for upper bits of the message data type ID. In that case, nodes that leverage an older version of the protocol will be implicitly compatible with the newer protocol by setting the higher bits always to zero. The fact that zero=dominant is a side effect here that has no practical purpose.

[pavel-kirienko]

:tada: