etcd-io / etcd

Distributed reliable key-value store for the most critical data of a distributed system
https://etcd.io
Apache License 2.0
47.71k stars 9.76k forks source link

Implement storage versioning #13168

Open serathius opened 3 years ago

serathius commented 3 years ago

Implementing a backward compatibility for etcd data directory is a prerequisite to implement downgrades (https://github.com/etcd-io/etcd/issues/11716)

Proposal https://docs.google.com/document/d/1yD0GDkxqWBPAax6jLZ97clwAz2Gp0Gux6xaTrtJ6wHE/edit?usp=sharing

Milestone 1

Storage version is available for snapshots.

Milestone 2

Etcd code is version aware making downgrade implementation possible

Milestone 3

Downgrades can be implemented based on storage versioning

Followups

lilic commented 3 years ago

Thanks for the detailed document and this issue! 🎉 Looks good overall to me.

What is the diff between this in milestone 1:

Add etcdutl migrate --force as an escape hatch

and this task in milestone 2?

Implement etcdutl migrate

Also, could we have some issues open for contributions? For example if someone new or existing contributors want to contribute one or two of these tasks maybe?

serathius commented 3 years ago

What is the diff between this in milestone 1:

Add etcdutl migrate --force as an escape hatch

and this task in milestone 2?

Implement etcdutl migrate

First milestone introduces storage version that should be used to prevent using outdated tools/etcd to load storage data. For example etcd v3.6 should not be able to start from data left by v3.7 without doing explicit downgrade of this data.

This is technically a breaking change as currently you could technically load etcd v3.4 data by etcd v3.0 and vice versa. Official we only support only one by one upgrade (v3.0 -> v3.1 -> v3.2 -> v3.3 -> v3.4) and no downgrades (use backup), but there is no mechanism that would prevent user from shutting themselves in a foot.

Milestone 1 introduces mechanism to explicitly prevent this, so we need to introduce an escape hatch for users that still want to do an downgrade, but they should do it explicitly by running migrate --force.

Difference between Milestone 1 and 3 etcdutl migrate is that the first one just implements the escape hatch, and the second whole schema upgrades and downgrades.

serathius commented 3 years ago

Also, could we have some issues open for contributions? For example if someone new or existing contributors want to contribute one or two of these tasks maybe?

As implementing versioning of storage touches basically whole server codebase I was worried about potential code conflicts before we update the codebase structure. To minimize future conflicts I have started from large scale refactors to restructure code to accommodate versioning, but got bottle-necked on reviews.

Currently I have local going couple of PR in future as my published PRs are waiting for review from a week. I don't think adding new contributors will help the effort or be a good experience as they would have the same problem.

I'm happy to start creating issues when the ground work will be done.

serathius commented 3 years ago

@YoyinZyc @hexfusion @wenjiaswe @ptabor Could you do a review the design and give an approval on the design, so we have a community consensus and can move forward with implementing the breaking changes.