rabernat
commented
2 years ago Jeremy--thanks spending time thinking through the process. I absolutely agree that it is necessary to understand why we are doing this before moving forward. And I agree 💯 with the idea that we should not "waste" the V3 opportunity to make breaking changes. So overall I support this part of your proposal:
I would propose instead that ZEP 1 be used to establish agreement on the problems to be solved in a backwards-incompatible way, and then in subsequent ZEPs we could modify the spec to actually address those problems.
I think this is the correct order of operations.
At the same time, a ton of work has already gone into the V3 spec and its multiple implementations. So we should do this while recognizing that there is already a lot of consensus and momentum around V3. I hope that the outcome of this discussion will be to formalize this consensus and move forward with V3 in a form that is very similar to how it looks now. It has to be mentioned that we have basically spent a large fraction of our EOSS grant on refactoring zarr-python to support V3. We do not have to stick with it just because of that (sunk cost fallacy); however, changing course strongly now would mean that we have wasted resources by putting the horse (V3 implementation) before the cart (clear mission and consensus). If that turns out to be the case, the blame is on us (Zarr SC).
If agree to follow the sequence you proposed, it means we can defer the actual technical discussion of breaking changes to ZEP-1. That will be the place to go into detail on the many thought-provoking points you raised in your comment. However, I do want to push back against one thing:
The v3 spec reorganizes metadata, but it is not clear what problem is solved by that reorganization or how the reorganization would be helpful for high-latency storage.
The placement of metadata in a separate tree from chunks will have immediate and significant improvements for at least two scenarios:
- Users of cloud storage (e.g. S3), where listing a path with millions of objects in it can be extremely slow
- Users of HPC filesystems that have similar issues with listing directories
Yes, there are ways to work around this with V2 (consolidated metadata or separate chunk store). But IMO this improvement is significant enough to justify the breaking change. But I look forward to continuing that discussion in ZEP-1.
I think we should try to keep up the momentum and engagement from this week's community meeting, and try to resolve this quickly. That means getting https://github.com/zarr-developers/governance/pull/16 finalized and merged asap. then moving to ZEP-1.
jbms
joshmoore
jakirkham
jstriebel
From our discussions on v3 my sense is that we need a clear mission statement for v3.
Creating a new storage format provides the opportunity to address problems of existing formats in a backwards-incompatible way --- in my mind that is the only reason to have a zarr v3 rather than just extending zarr v2.
But once v3 is properly "released", that opportunity will have been spent and is no longer available for future improvements. As noted by @rabernat in the community meeting on 2022-04-20, having a large number of incompatible zarr specification versions and implementations may likely just drive people away from zarr altogether.
Reviewing the current v3 spec proposed by @alimanfoo, I see the following 3 things highlighted as the motivation for v3:
However, in my mind it is not clear that these are the most pressing issues to address with a backwards-incompatible change, and furthermore I don't think the current v3 spec actually addresses these issues:
Additionally, the one change that has primarily motivated the renewed interest in zarr v3, sharding, does not require a breaking change, and has been suggested to be deferred until after the initial release of the v3 spec.
I think the zarr steering council has done great work in trying to establish a process by which the v3 spec can be designed and evolved, and I think it would be a mistake to release v3 (and lose the opportunity to make backwards-incompatible changes) without actually taking advantage of that process.
I also think we can potentially distinguish between zarr v3 the format as a breaking change to zarr v2, and this new process of evolving zarr (where there is input from implementations and the user community rather than just a decision from @alimanfoo).
In the community meeting on 2022-04-20, @joshmoore proposed that ZEP 1 should be the release of zarr v3, and my impression is that this was intended to be a relatively small change to the existing spec, just the removal of the extension mechanism. I would propose instead that ZEP 1 be used to establish agreement on the problems to be solved in a backwards-incompatible way, and then in subsequent ZEPs we could modify the spec to actually address those problems.
To be clear, a lot of changes that have been discussed in the context of v3 are purely additional features that can be added on top without requiring any breaking changes to existing usage:
I don't think it is necessary to reach agreement on these types of changes before releasing v3. On the other hand, I think it is important for v3 to have at least one significant improvement as otherwise there is no reason to adopt it.