[NEW] Rolling downgrade, forward compatibility

[zuiderkwast]

The problem/use-case that the feature addresses

After a rolling upgrade of a cluster, if some problem is found, such as CPU or memory usage or some bug, the administrator wants to be able to a rolling downgrade again and take some time to investigate the problem.

When adding new nodes to a cluster, the standard procedure is to add new nodes as replicas to existing nodes. For replication to work, we require that a replica's RDB version >= primary's RDB version. This works for rolling upgrade, but it doesn't work for downgrading if the RDB version has changed.

Some system that I've been made aware of is stuck on Redis 6 (RDB 9) because of the requirement to be able to do rolling downgrades.

The cluster bus is already backward and forward compatible AFAIC. It would be good to avoid breaking that. :)

Note that we're only talking about adding new nodes of an old version to a cluster. We don't need to start an old node with a config file from a new version such as nodes.conf, nor do we want to reuse a node-id when replacing nodes in the cluster.

Description of the feature

Possibility to generate RDB in an older version than the latest, if no keys require the newer RDB version.

An example is the bump from RDB 10 to 11 (that I'm guilty of by introducing listpack encoding for sets). If we configure the node to generate RDB 10, the node can easily be made to store this listpack as a RDB_TYPE_SET (which is just each element as a string) instead of RDB_TYPE_SET_LISTPACK (a listpack dump). The bump from 9 to 10 is similar.

Alternatives you've considered

59 = Full sync without RDB preamble, although it could be combined with a REPLCONF to check if the replica supports the current RDB version and fallback to AOF without preamble. (RDB is a bit faster and smaller in size.)

Additional information

Redis never supported downgrades. This would be a Valkey feature that would make Valkey superior in terms of compatibility.

Valkey/Redis	RDB	Why bump
Redis 5.0	9
Redis 6.0	9
Redis 6.2	9
Redis 7.0	10	Change ziplist to listpack for hashes, sets, etc.
Valkey 7.2	11	Add listpack encoding to sets
Valkey 8.0	11

[madolson]

The major decisions:

1. Do we want to be able to support this downgrade.
2. How far back to we want to support restoring into older versions? (At least one version, maybe much further)
3. How will we document this.
4. What is the format we will use to support forward compatibility. One option is to have the primary send an older RDB compatible object. Another option is to send it in AOF format.

Consensus from core meeting seems to say yes for 1.

[madolson]

We also want to create a separate issue outlining how we can add guardrails to prevent usage of new features which would break forward compatibility. The discussed example was for hash field expiration, which must necessarily change the RDB format.

[zuiderkwast]

A pre-condition we seemed to agree on is that a downgrade is only possible if none of the new features in the new version are used, for example a new key type that doesn't exist in the old version.

It seems pretty strait-forward to generate RDB of an older version. How to select the RDB version to output? There are some possibilities:

1. A new config rdb-version, default to "latest". The admin needs to set it to the required version before connecting the downgraded nodes.
2. Autodetect which RDB version to use. Since Valkey 8.0, replicas send REPLCONF version. If they don't, we can assume they're using a very old version, so we can pick let's say RDB 9, and we can use this for the full sync. There are some corner cases here though:
   • What if a new replica is already performing a full sync using the latest RDB and a downgraded replica connects? It can't share the already created RDB dump. Shall we create a new fork to do a new RDB dump with RDB 9, or shall we return some "Try again later" error?
   • How to handle RDB dump on disk? Which version should they use?

Option 1 (using a config) seems like the easiest and most explicit way.