search

zarr-developers / zeps

Zarr Enhancement Proposals
https://zarr.dev/zeps
Creative Commons Zero v1.0 Universal
12 stars 15 forks source link

Revise ZEP0 #59

Open MSanKeys963 opened 7 months ago

MSanKeys963 commented 7 months ago

Hi all. 👋🏻

This PR was last updated on: 10/15/2024

I'm proposing changes to our ZEP process, i.e. ZEP0, based on the discussions at #44, #52 and #55. Here are the major changes I'm proposing through this PR:

  1. Addition of new ZEP type under Specification ZEPs category → Protocol Extension ZEPs → Removed in https://github.com/zarr-developers/zeps/pull/59/commits/3ae1fc5833cc95220e87370c308b20619f4ab931. Please check https://github.com/zarr-developers/zeps/pull/59#issuecomment-2414350461.

    Until now, we have only been using Core ZEPs to make changes to the specification. Extension ZEPs would help define and clearly add new extensions to the specification via the extension mechanism.

  2. Adding a new ZEP category → Lean ZEPs → Removed in https://github.com/zarr-developers/zeps/pull/59/commits/81b7ca2c9c554b77c0587ace192f0cbacb255a0b. Please check https://github.com/zarr-developers/zeps/pull/59#issuecomment-2403341658.

    This category is suitable for adding significantly small and straightforward changes to the Zarr specification directly via a pull request.

  3. The process of submitting a ZEP

    Until now, if anyone wishes to submit a ZEP proposal, they'd need to create two PRs, first in zeps repository and the other in zarr-specs repository. I've proposed simplifying this process by combining the zeps and zarr-specs repositories. Now, the author could submit only one PR to submit a ZEP proposal for the Zarr specification.

  4. Discussing a ZEP

    As pointed out in #55, discussing a ZEP proposal in multiple issues and PR fragments the conversation, leading to a loss of motivation and momentum. I've simplified this and proposed that all the discussions occur in a single PR. I've also kept an option to open separate issues if the conversation becomes too long and hard to follow, but this depends on the need and the author's choice. But please keep in mind that all the conversation and issues will happen in zarr-specs repository.

  5. How does a ZEP become accepted?

    This is mostly formalising the various phases I've outlined in https://github.com/zarr-developers/zeps/issues/55#issuecomment-1905157744. Until now, we have been using a voting mechanism to decide on the fate of the ZEP proposals. Over the past couple of years, we've realised we need more involved participation from the stakeholders. I've proposed to add a couple of phases before we start voting on the ZEPs.

  6. ZEP Format

    I've proposed that the ZEP be in .rst instead of .md format, similar to the specification text format.

There is also a question regarding the versioning of the ZEP proposal, which are modified after acceptance, for example ZEP0. There's a suggestion of using semantic versioning by @LDeakin in https://github.com/zarr-developers/zeps/issues/55#issuecomment-1962288366, which is something we can discuss and adopt. PS. If we adopt semantic versioning, this update would be ZEP0.1. :)

Many thanks to @joshmoore, @rabernat, @jbms and others for the initial discussions regarding the ZEP0 revision.

Also, I've been fiddling around to merge the zeps and zarr-specs repository and have come up with: https://docs-test-sanket.readthedocs.io/en/latest/. I'd appreciate any feedback or comments on the new webpage design.

Please let me know what you think of this proposal. Also, please send this to folks interested in this conversation.

Preview: https://zeps--59.org.readthedocs.build/en/59/active/ZEP0000.html

CC: @zarr-developers/implementation-council @zarr-developers/steering-council

normanrz commented 7 months ago

I agree with @rabernat. Lean ZEPs sound great!

My learning from the last ZEPs was that we need to engage the stakeholders earlier in the process and not only when presenting them with a final vote. Having the feedback earlier increases buy-in and reduces the number of post-vote changes. I don't think the solution to that problem is more phases and more votes in the process. On the contrary, I think we need to make the conversation easier to follow and actively reach out to ZIC and ZSC. That is why the Lean ZEP approach is appealing. It simplifies the process/paperwork and streamlines the conversation. In any case, the ZEP process only works with the expectation that members of the ZIC and ZSC bring enough time and attention to follow the conversations.

I still think we need a mechanism of forcing a vote, though. Consensus seems ill-defined in a distributed, volunteer-driven community.

normanrz commented 7 months ago

There is also a question regarding the versioning of the ZEP proposal, which are modified after acceptance, for example ZEP0. There's a suggestion of using semantic versioning by @LDeakin in #55 (comment), which is something we can discuss and adopt. PS. If we adopt semantic versioning, this update would be ZEP0.1. :)

In the spirit of copying PRs, I would advocate for keeping linear ZEP numbering. The ZEP process is its own artifact that gets created and modified through ZEPs. ZEP 0 introduced the process. This proposed modification should become its own ZEP.

The wish for ZEP versioning almost seems like an artifact of a too complex ZEP process. If it were as easy to make and approve ZEPs like a PR, we could just introduce changes like the sharding "index_location" in a new ZEP.

jbms commented 7 months ago

Regarding versioning, as I see it, a "ZEP" would just be the additional process taken for reviewing a non-editorial/non-trivial spec change (i.e. PR). Therefore, there would be no reason to version a "ZEP" except perhaps in its draft state, where you could potentially identify versions by git commit hashes, because once it is accepted it is merged into the spec and any further revisions would be done as a separate PR. In fact I would propose that the ZEP number be assigned based on the PR number.

For example, let's say we add a "pack_bits" codec in PR 1234, that initially only supports "bool" data type. Then someone in PR 1255 adds support for a "uint4" data type and modifies the description of the "pack_bits" codec to also support "uint4". When we document what features are supported by each implementation, if not already explicit we could at this point update the descriptions to indicate that certain implementations support pack_bits with bool only while others support both bool and uint4.

It might be useful to define as part of the spec a precise way to indicate support for various features. For example we could say that support for a given data type can be indicated by the "codecs/pack_bits/" feature identifier. I'm not sure how important it is to specify these feature flags precisely, though.

normanrz commented 7 months ago

I think it is important to separate support for a feature of the Zarr specification in an implemention from the ZEP that introduced the feature.

It might be useful to define as part of the spec a precise way to indicate support for various features. For example we could say that support for a given data type can be indicated by the "codecs/pack_bits/" feature identifier. I'm not sure how important it is to specify these feature flags precisely, though.

I think an implementation needs to be able to decide during metadata parsing whether it can correctly understand a Zarr array/group. If it encounters unknown attributes, values or identifiers that would already indicate a lack of support. Basically, that means that new changes can only be additive (e.g. new attributes on existing codecs, new values for attributes, new codecs).

jbms commented 7 months ago

I think it is important to separate support for a feature of the Zarr specification in an implemention from the ZEP that introduced the feature.

It might be useful to define as part of the spec a precise way to indicate support for various features. For example we could say that support for a given data type can be indicated by the "codecs/pack_bits/" feature identifier. I'm not sure how important it is to specify these feature flags precisely, though.

I think an implementation needs to be able to decide during metadata parsing whether it can correctly understand a Zarr array/group. If it encounters unknown attributes, values or identifiers that would already indicate a lack of support. Basically, that means that new changes can only be additive (e.g. new attributes on existing codecs, new values for attributes, new codecs).

To be clear, I was imagining that these feature identifiers would be solely for documentation purposes, similar to caniuse.com, not that they would appear in the metadata anywhere.

MSanKeys963 commented 7 months ago

Thanks all for the feedback so far.

Here are my responses:

Who will be the arbiter of whether a ZEP is minor enough to qualify for a Lean ZEP? What will happen if someone submits a Lean ZEP which some people feel is scoped too large? Who will enforce turning it into a regular (old-style) ZEP?

@rabernat: The changes which are minor and non-controversial would fall under the Lean ZEP category, e.g. https://github.com/zarr-developers/zarr-specs/pull/256 or adding a new data type. The ZEP, which proposes a significant change to the specification and requires a reference implementation of the sort, should not be submitted under Lean ZEP, e.g. major specification version update (V3 → V4), extensions like sharding, variable chunking, etc. In the ZEP workflow section, the author has various options to ask around if their changes are minor enough to be considered under a Lean ZEP.

Categorising all ZEPs under Lean might lead to less involved participation in an important proposal from everyone involved. The reading and implementation phases ensure that the stakeholders are given enough time to read and work on implementing the proposal before it is finalised. I've outlined various issues we've encountered here: https://github.com/zarr-developers/zeps/issues/55#issuecomment-1905156595.

I've also proposed removing the option for multiple repo/PR processes so that everything would take place in the zarr-specs repository.

I don't think the solution to that problem is more phases and more votes in the process.

@normanrz: Why do you think a multi-phase approach would not help have more engaged participation from the stakeholders?

normanrz commented 7 months ago

e.g. zarr-developers/zarr-specs#256 or adding a new data type

Adding data types or codecs can be controversial, too. I don't think it is possible to know a priori. That is why a community discussion is needed for every ZEP. PRs are a great tool for that.

requires a reference implementation of the sort

Shouldn't all ZEPs come with a reference implementation or at least a proof-of-concept?

I don't think the solution to that problem is more phases and more votes in the process.

@normanrz: Why do you think a multi-phase approach would not help have more engaged participation from the stakeholders?

Everybody in the Zarr community is volunteering and very busy. Multi-phase processes require a lot of effort to understand and to follow. I might be wrong but I think that more complex processes can be intimidating and annoying both on the side of the ZEP authors, who need to chase the feedback/votes, and on the side of the community members, who are frequently pressured to issue feedback/votes.

MSanKeys963 commented 2 months ago

Resurrecting this PR after having discussions and feedback from various community and ZEP meetings. I have extended the scope of Lean ZEPs to add breaking changes if there is unanimous approval from participating stakeholders. This change is specifically targeted to include https://github.com/zarr-developers/zarr-specs/pull/292 and other similar changes in the future.

@normanrz, here are my responses to your last comment —

Adding data types or codecs can be controversial, too. I don't think it is possible to know a priori. That is why a community discussion is needed for every ZEP. PRs are a great tool for that.

It can be. That's why it's recommended and preferred that an issue in the zarr-specs repo be created to have a community discussion before working on the actual PR for a ZEP; see here - 3rd para.

Shouldn't all ZEPs come with a reference implementation or at least a proof-of-concept?

ZEPs which add new functionality by changing or extending the existing specification should be accompanied by an implementation/POC. Other ZEPs like Lean, which includes minor changes (addition or removal of non-controversial changes) or Informational or Process, don't require a POC.

complex processes can be intimidating and annoying both on the side of the ZEP authors, who need to chase the feedback/votes, and on the side of the community members, who are frequently pressured to issue feedback/votes.

I understand that the phased approach requires involved participation from the active maintainers and implementors. I've outlined issues we've faced with the existing version of the ZEP process at https://github.com/zarr-developers/zeps/issues/55#issuecomment-1905156595 during ZEP1. We need a mechanism to avoid repeating those issues. I think the phased approach is one solution that could help us. If you have suggestions, let's discuss them. I'm all ears. :)

Also, having all the discussions and comments in a single PR for the entirety of a ZEP proposal would be convenient for the author to facilitate feedback and votes from everyone involved.

normanrz commented 2 months ago

Thanks for addressing some of our feedback and revising the ZEP.

I think the lean ZEPs help to move the process in the right direction. However, a main problem we are currently facing is lack of (timely) responses from the council members. Without solving that problem, I don't think we will see significant improvements to the process. So, I think it is important to also move https://github.com/zarr-developers/governance/pull/44 forward.

d-v-b commented 2 months ago

It seems like a "lean ZEP" as defined in this PR is a spec change that gets approved without a formal ZEP process, and so I would argue that a "lean ZEP" is is not a ZEP, or at least it communicates very little to categorize such changes as ZEPs.

Instead of requiring that every spec change fall into some ZEP category, It would be simpler to reserve the concept of "ZEP" for formal proposals that do observe a ZEP process. Instead of defining a "lean ZEP", this document can simply state that not all spec changes happen via ZEPs -- some spec changes, like typo fixes, or even semantically significant changes, can be proposed, discussed, and accepted via github PRs provided there is an exceptional level of consensus, bypassing the ZEP process completely.

@normanrz wrote:

I think the lean ZEPs help to move the process in the right direction. However, a main problem we are currently facing is lack of (timely) responses from the council members. Without solving that problem, I don't think we will see significant improvements to the process. So, I think it is important to also move zarr-developers/governance#44 forward.

I agree with this completely. We are not bottlenecked by the lack of new ZEP categories (although it's important to define those carefully). We are bottlenecked because there is not enough developer time going into the spec.

jhamman commented 2 months ago

We are not bottlenecked by the lack of new ZEP categories (although it's important to define those carefully).

Extrapolating on this a bit, let me introduce a new idea here. Rather than adding additional ZEP categories, we do away with all but core-protocol ZEPs, reserving these for major and/or breaking changes to the specification. All other changes (e.g. bug fixes, clarifications) and/or extensions to the specification can be done as pull requests directly to the spec repo and merged through a much simpler process (ideally managed entirely by the spec core team -- not the ZEP process).

joshmoore commented 1 month ago

I think the primary additions I'd make to your comment, @jhamman, would be:

MSanKeys963 commented 1 month ago

Thanks, everyone, for your feedback.

I understand the importance of adding changes to the spec via GitHub PRs. I'm in favour, but I'd like to keep the Lean ZEPs category in the process, as it'll help us maintain a record of changes that have gone in the spec in a readable manner — the original idea I proposed in this PR to combine zarr-specs and ZEP repository and maintain a single website for both of them (sample here). The webpage would host and display a list of Lean ZEPs (which would essentially grab text from the PRs, i.e. lean ZEPs).

Let me know what you think. ^

The discussion at https://github.com/zarr-developers/governance/pull/44 is important, and I like what Josh proposed in https://github.com/zarr-developers/governance/pull/44#issuecomment-2352788442.

I'd like to propose merging this PR unless strong objections exist to this current version. Successive PRs can be added for more changes or to reflect the progress/conclusion from https://github.com/zarr-developers/governance/pull/44.

d-v-b commented 1 month ago

I'd like to propose merging this PR unless strong objections exist to this current version.

I think @jhamman's proposal to slim down the ZEP process counts for me as a strong objection. We should at least discuss that as a viable alternative to this PR.

@MSanKeys963 if the main goal is to record all spec changes on the zeps web page, then this can be done without introducing a new ZEP category -- we could display ZEPs and PRs together on that site, with a clear indication of which is which.

MSanKeys963 commented 2 weeks ago

As per our @jhamman's https://github.com/zarr-developers/zeps/pull/59#issuecomment-2310819674, and discussion in the last community and ZEP meeting (notes here) with @joshmoore and @d-v-b — I've removed Lean ZEPs category in https://github.com/zarr-developers/zeps/pull/59/commits/81b7ca2c9c554b77c0587ace192f0cbacb255a0b.

Also, see: https://github.com/zarr-developers/zeps/pull/59#discussion_r1794148489.

We discussed that https://github.com/zarr-developers/governance/pull/44, when finalised, would mention when the Zarr-Specs core devs think it's necessary to follow the ZEP0 process to change the specification. The definition of these changes is yet to be decided, but they're mostly going to revolve around major changes like incrementing the specification version, change that involves participation from multiple implementations, and similar things.

All other changes to the specifications would be done simply via a GitHub PRs. These PRs would be approved and merged by the Zarr-Specs core devs team (when formed).

I've added multiple comments in the markdown document like: <!-- Mention Zarr-Spec devs in line: x-y, after zarr-developers/governance/#44 is finalised --> to change the text when the time comes.

Please review. Thanks!

joshmoore commented 2 weeks ago

@rabernat, @jakirkham, @alimanfoo, @ryan-williams: are there any changes you would like to see here?

MSanKeys963 commented 1 week ago

Thanks for your reviews, @joshmoore and @rabernat!

As per your feedback, I've removed extension ZEPs entirely as we're working on defining a new way to add extensions — currently under discussion in https://github.com/zarr-developers/zarr-specs/pull/312#issuecomment-2407444223 and https://github.com/zarr-developers/zarr-specs/issues/316.

I've also removed the SPEC ZEPs categorisation as we previously had core and extension ZEPs under the category. Now, there is only Core ZEPs.

Also, every core ZEP (removed the term 'protocol') will lead to a major version increment of the Zarr specification.

Meta question: should this change be done via a ZEP? 🤣

@rabernat: Instead of a new ZEP, let's use this PR to add the changes so we can move ahead! ;)