search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.58k stars 24.63k forks source link

Create new index & transport versions on minor releases #105295

Open thecoop opened 7 months ago

thecoop commented 7 months ago

Relates #104620

Whenever we release a new minor version, the release automation should bump index and transport versions on main after the release is branched. This helps define rollback & compatibility boundaries as well as helping with mapping from version ids to release versions.

We should not automatically bump versions on patch releases - if we do so we're inserting patch versions into the version ids, which should only be used if we really have to.

This has a few consequences:

  1. The CCR check failing in #104620 needs to either be rewritten to allow different patch versions of the same minor to use CCR, and updating docs accordingly, or to use some other mechanism (equality on the string representation of the build version?) to strictly enforce CCR compatibility
  2. Release version mapping needs to output sensible values for different patch versions with the same version id
  3. There are probably other boundaries somewhere that will no longer be enforced as strictly as they are now. We should check this.
elasticsearchmachine commented 7 months ago

Pinging @elastic/es-core-infra (Team:Core/Infra)

elasticsearchmachine commented 7 months ago

Pinging @elastic/es-distributed (Team:Distributed)

thecoop commented 7 months ago

I'm looking at using an equality check on build version within RestoreService, but the general check in RestoreService is a semantic check - it can restore snapshots from previous versions, but not future versions. Build version is just a string, and can only be used for an equality check. We would need to have a specific comparison check...somewhere...just for CCR. I think our options here are:

  1. Loosen CCR compatibility to allow patch versions (at least those with the same IndexVersion) to interop. We may or may not actually publicise this fact.
  2. Keep storing and comparing Version in snapshot info, and use that in RestoreService

@DaveCTurner @rjernst Thoughts?

mosche commented 6 months ago

I had a brief discussion with @thecoop on the topic and started looking if there's a more CCR specific alternative to reverting back to storing the build version in snapshot info (Simon's 2nd point in the previous comment).

How about we add a CCR specific build version check in CcrRepository.getSnapshotInfo to enforce version constraints for patch versions. That way RestoreService remains generic and only needs to check based on index version, as it should be.

When testing locally that seems to work well, but I'm pretty sure there's a million use cases I haven't covered. Are you aware of any other use case that would be negatively impacted by this?

@thecoop @rjernst @DaveCTurner