ECS logging library versioning now and going forward

[felixbarny]

We've had multiple strems of discussions about how to properly version ECS libraries. There are three main options:

1. Version loggers independently, starting with 1.0.0 and using semver Loggers just depend on a small subset of ECS fields that are available since ECS 1.2. We don't expect many breaking changes on these fields, even in new major versions of ECS.
2. Version them in alignment with ECS The .NET loggers does that as it generates types based on the ECS schema definition. For ECS, there's a desire to align with the stack release versions as of 8.0.
3. Version loggers in alignment with the stack release Makes it easier for users to grasp which versions work well with each other, especially when it comes to breaking changes.

Currently, loggers are using a mix of 1. and 2.

We actually have a breaking change lined up for 8.0 regarding event.dataset vs data_stream.dataset (https://github.com/elastic/ecs-logging/pull/38) where it would help if loggers would be aligned with the stack release.

Let's discuss which versioning scheme we want to follow for the library's GA and whether to change things up when 8.0 comes along.

I personally think that we should aim to align with the stack versions as of 8.0.

I'm not so sure if we should jump into that right away for the GA of the ECS loggers. My main concern would be the management overhead associated with it. I anticipate that most loggers are fairly stable with infrequent code changes after GA. When we align with stack releases, we'd still need to release minor and bugfix versions in sync with the stack release even if there are no changes in a logger. With automation in place, we can hopefully bring that overhead down to a minimum.

Open questions when aligning with the stack release

• Can we automatically create stack release branches to minimize the management overhead?
• How would the integration into unified release work?
• We shouldn't require users to update their Elastic stack and their log library versions at the same time. How should we define the support range of a ECS logger with the stack version? I guess at the least, a ECS logger with version 8.x should be compatible with all minors within 8.x and 8.last (or even 8.x?) should be compatible with 9.x. But should 8.x loggers also be compatible with 7.last or even 7.x versions of the stack?

[Mpdreamz]

I personally think that we should aim to align with the stack versions as of 8.0.

Stack release all the things!, I think this is a paramount piece in order to clearly communicate support, EOL's and compatibility. We're kind of dependent on ECS jumping on the stack release train too, which I hope it does.

8.0 seems like a realistic time frame.

Can we automatically create stack release branches to minimize the management overhead?

There is tooling int /infra that can help here (not linking in public repos DM me)

The clients are going through a similar transformation at the moment

How would the integration into unified release work?

There are various ways to integrate, the clients are going through this effort with @Conky5 as our main point of contact.

The way we plan to do it now is to have the branching, building bc's and creating github/tags automated.

For ecs-dotnet a git tag/release trigger is already in place and owned by @elastic/observablt-robots.

This allows the unified release to do build and tag automatically but the actual publish goes through the observability infrastructure using key/secret management that @elastic/observablt-robots owns and manages.

But should 8.x loggers also be compatible with 7.last or even 7.x versions of the stack?

For Elasticsearch currently the mapping is client M == server M with work on the way to make sure client M can also speak with server M+1.

The reverse is currently not planned, client M+1 talking to client M will result in an error.

Not saying ecs loggers should follow this but making sure this precedent is known.

Its easier to constrict now and expand our bwc support later then visa versa.

[SergeyKleyman]

we'd still need to release minor and bugfix versions in sync with the stack release even if there are no changes in a logger

What about the opposite direction - if we need to release a bugfix version of a logger will it require releasing a bugfix version of the whole stack?

[Mpdreamz]

What about the opposite direction - if we need to release a bugfix version of a logger will it require releasing a bugfix version of the whole stack?

The stack release schedule always includes several patch releases in its schedule. Enough for most cases to hitch a ride on.

Critical bug fixes will need to be raised and meet the bar for the whole stack to move. For these integrations often downgrading is a suitable temporary measure.

That said jumping on the stack release train raises the bar on the integration branches in that they should be green and releasable. This is a good thing IMO.