Governance and organization for this repo and for the spec development process

[rabernat]

Now that we have a shared place to work on this spec, we also need a process by which to evolve it. If we want to move forward without the official OGC SWG process, we will have to answer some basic questions about how we will operate.

• How will we make decisions about how to evolve the spec? We should strive for consensus. But what if we can't reach it? Will we vote? Who gets to vote?
• Who will have commit rights on this repo? I see this more as a technical question rather than a governance question. If we can answer the governance question above, then merging PRs is just a technical step--but someone will still need to do it. For the time being, I have created the @zarr-developers/geozarr group, which contains myself, @christophenoel, and @briannapagan.
• How will we involve the community? There has been a lot of discussion on twitter. But we need to move this to the issue tracker and work to enumerate the specific suggestions that have been made about evolving the spec?

There are many other questions, but these are the main ones that come to mind.

[christophenoel]

For the decision making method , I recommend applying principles similar to OGC SWG. Creating a SWG (soon or later) is the final goal that initiated all these discussions.

It would also be beneficial to have a repository to store and collaborate on documents (e.g. a report on yesterday's discussion).

Finally, the involvement of xArray is essential, as this is nearly the only tool available to demonstrate the applications of Zarr in the spatial context.

[briannapagan]

I am about to send some follow up emails from last week's discussion, but want to do this as openly as possible, and it touches on some of your questions @rabernat. I am going to send a doodle poll to those who attended and others mentioned at the end of this meeting: https://hackmd.io/@MSBYE-SmSS-O706S4WXH0Q/geozarr-spec-swg-20230119. From this I will select a standing bi-weekly time and post information on how to join these meetings.

I would like to better understand how the SWGs work when it comes to voting/consensus. This can be a topic of discussion for our first biweekly meeting.

[rabernat]

Brianna, has this doodle gone out? I didn't see it in my email.

[briannapagan]

@rabernat With so many people I just selected a day/time, you should have received an invitation last week for bi-weekly meeting Wednesdays at 11AM EST. It was sent to your Columbia email.

[rabernat]

Ok thanks for this. I did get it (but had not realized).

The last 30 minutes conflict with the Xarray developers' meetings. But I think that's okay for now.

I just looked at the invite list. There are potentially over 30 people attending. That is a HUGE group. I think we have to be extremely clear about the goal of the meeting and super disciplined about the structure. 30 people cannot all engage in a dialog--it will end up being dominated by just a few voices.

My $0.02 is that the goal of the meeting should be informational. The actual work on discussing and implementing the spec will be async. At the meeting, we should focus on conveying information to interested parties about how the spec process will work. This will require significant prep work. It might be useful to have a few short presentations. For example.

• Scott or someone from OGC could explain the SWG process
• @matthewhanson could explain how the STAC process has worked
• I could explain how the Zarr spec and convention process works
• Someone (maybe @christophenoel) could give an overview of where GeoZarr stands today and what are some of the challenges / open questions that have to be addressed

[christophenoel]

I believe the agenda should not be overloaded and should focus on the discussion about the OGC SWG process. I don't know if Scott provided feedback if he may attend the meeting. I'm not against introducing GeoZarr for the next one, but I need some time to prepare a short presentation as I haven't worked on this topic for months now.

[briannapagan]

Information for how to join and notes from meetings will be kept here: https://hackmd.io/@briannapagan/geozarr-spec-swg

[christophenoel]

It would be great to collect the brainstorming ideas and be able to work on that during the week.

[dblodgett-usgs]

Given what I think is consensus on pursuing an OGC standard along the "full standard" track, which I've summarized in the open PR #13 -- the outstanding issue (for this issue) is whether we will continue to work in this space or take up development of an OGC specification under the opengeospatial github organization? Perhaps we develop a SWG charter using this repository / issue tracker and move to the OGC github for SWG activities?

[christophenoel]

For your information, I'm still waiting for confirmation from Scott Simmons regarding whether our activities are suitable for a SWG.

[briannapagan]

Scott is able to join for the first 30 minutes tomorrow at our biweekly meeting. I would encourage folks to come with any open questions.

[christophenoel]

Just to share comments from Scott:

Yes, the objectives are definitely in line with OGC intent and the ZEP-4 document includes information that is suitable for an OGC Standard. BUT, OGC would still need a formal Standard document to reference that looks quite different from the ZEP-4 document.

Here is how I can see this working. The content of a ZEP would need to be described as clearly-defined requirements that are testable. The OGC template for a Standard [1] would then be populated from the ZEP text and requirement(s). The Standard could be very short - no need to write hundreds of pages if very few are needed. Finally, the enhancement (which I suppose would be extended, but optional functionality for Zarr) could also be described as is best for the Zarr community as an included Annex in the Standard so that Zarr users see what they are used to and OGC Standard readers also see what they expect.

OGC could publish the documentation in more developer-friendly ways than just the ugly Standard document itself, see OGC API - Features [2].

I suppose that adding a few more requirements to the ZEP-4 convention is acceptable.

[dblodgett-usgs]

We should develop a CONTRIBUTING.md that outlines the answers to @rabernat's OP.

I think we are going to move forward with this as an opt-in consensus (objection to unanimous consent / vote and debate) process akin to the OGC. How we identify voting members is not clear, but doesn't need to be complicated if we don't want it to be. Does the ZARR community governance offer any guidance?

We need to answer the "who has commit rights" and who is moderating question. This is a coalition of the willing / able and we should move ahead with a light weight and transparent approach.

IMHO, we should notify the ZARR/xarray community, the CF community, and the OGC community of this effort but keep development here with a target deadline to have a ZARR convention formatted as an OGC community standard.

[christophenoel]

Following the recent progress and discussions from our bi-monthly meeting, I would like to respectfully share my thoughts and insights on two key subjects that have been frequently discussed and debated within the project. I acknowledge that not everyone may share my opinions, and I appreciate the diversity of thoughts that contribute to the success of the project.

GeoZarr: Scope

First, I would like to address the focus areas for GeoZarr. It is crucial that we first establish a common understanding of our goals before delving into highly specific technical details, such as encoding new coordinate types. From our perspective, the primary objectives (which are way beyond CF conventions) include:

• standardizing the encoding of geospatial data in Zarr to facilitate seamless data interpretation and processing by any client through a unified set of metadata and encoding conventions (including coordinate encoding as the purpose is that anybody can read the data in standard way).
• fully support the range of functionalities offered by cloud-native formats like COG, we aim to encompass map visualization with the necessary provisions for multiple projections and resolutions, accommodate various geospatial data types such as hyperspectral, multispectral, and 4D+ ARD data, and leverage Zarr's multidimensional array format by enabling support for multiple chunking versions.
• drawing from established conventions like CF but potentially others, where relevant, but overall identifying and prioritizing the most crucial properties and levels for our project.

Refining the CF conventions data model, although advantageous to numerous users, is not the primary focus of the GeoZarr project. While we recognize the potential benefits of such enhancements, they are not critical to the quality or success of GeoZarr. Furthermore, it is important to recognize that the acceptance and adoption of changes to the CF conventions may involve a more complex and extensive process across various stakeholders.

Standardization Approach

Secondly, I would like to address the working framework debate between the OGC SWG approach and an independent community-based approach (with the intention of proposing the result as an OGC Community Standard).

I suggest that developing conventions in isolation may not be advantageous, as it could lead to a less collaborative and less widely accepted outcome. This approach poses challenges, as it tends to enforce a finalized product rather than fostering deeper consensus among a more extensive group of stakeholders. Additionally, such a framework may be characterized by a lack of structure, and efficiency (and thus far, our progress has been limited)

Instead, an OGC SWG framework fosters a more structured and collaborative environment, enabling a broad range of stakeholders to provide valuable feedback and contribute to the development of a high-quality specification. Furthermore, the OGC SWG approach offers the added benefits of hardware infrastructure, organizational support, and increased visibility.

"if we want to go with SWG we have to write a charter, get it approved and only then we can start working, so could also delay things"

As we are already actively working on the project, I don't see how developing a clear charter would impede our progress; instead, it could provide a solid framework that aligns our efforts and enhances our overall efficiency.

In conclusion, we appreciate the diverse perspectives, and I hope that we can engage in a constructive dialogue to address these key topics. The initial intention of Spacebel and ESA was to support the OGC SWG framework, which promotes collaboration and a structured approach, ultimately leading to a successful and widely adopted GeoZarr standard.

Thank you for considering my thoughts and for your continued dedication to the GeoZarr project.

[dblodgett-usgs]

I can offer a second to what you are proposing, @christophenoel. If you would like USGS to be a co-convener of the SWG, please feel free to add my name to the charter and add me as a reviewer when you are ready for review. If and when the SWG kicks off, I will join as a member and take part in the process.

I would suggest modifying the README here to indicate that this decision has been taken and that activity will move to the opengeospatial venue in the future so it is clear to others that this will not be developed in the zarr-developers organization.

[rabernat]

will move to the opengeospatial venue in the future so it is clear to others that this will not be developed in the zarr-developers organization.

Let's not move the repo again. We can write the charter in way that allows us to keep the repo here. This is a technical issue and separate from the broader organizational questions.

[dblodgett-usgs]

I'm not familiar with any OGC activities taking that approach, but would support it if the community is willing to accept that practice.

[christophenoel]

Make sense to close because of creation of the SWG.