search

hazelcast / hz-docs

Source content for the Hazelcast Platform documentation
10 stars 97 forks source link

Unclear rolling upgrade compatibility guarantees for Jet #724

Open vbekiaris opened 1 year ago

vbekiaris commented 1 year ago

According to https://docs.hazelcast.com/hazelcast/latest/deploy/versioning-compatibility#api-compatibility-exceptions, Jet classes only provide patch-level compatibility guarantees.

This contradicts item 2 in Compatibility Guarantees section of Rolling Upgrades which states that "each minor version is compatible with the previous one" (in the context of rolling upgrades for Hazelcast Enterprise).

This should be clarified and proper guidance should be given in the rolling upgrades section. Thanks @burakgok for bringing this to my attention. /cc @k-jamroz

k-jamroz commented 1 year ago

Jet compatibility is a tricky topic. Current approach is that:

  1. To run Jet jobs, entire cluster must be in the same version, otherwise jobs will not be started (i.e. during rolling upgrade Jet Jobs do not run). See JobCoordinationService.shouldStartJobs. It think that also cluster version must be the same as code version (so the RU fully completed), but can't find the check.
  2. Jobs submitted in previous version will have to be resubmitted after upgrade (see MasterContext.resolveDag). Jet uses Java serialization.
  3. Job metadata (including SQL catalog!) are stored in IMaps, some are Versioned, but usually they are not upgraded eagerly unless we implement special procedure for that. Versioned support provides reasonable compatibility, but we should not remove compatibility with older versions from code (data may remain for a long time, especially in SQL catalog).
  4. Changes to some Jet IMaps are almost impossible due to compatibility with Management Center