Publish Zarr spec with OGC

[rabernat]

http://www.opengeospatial.org/standards:

OGC(R) standards are technical documents that detail interfaces or encodings. Software developers use these documents to build open interfaces and encodings into their products and services. These standards are the main "products" of the Open Geospatial Consortium and have been developed by the membership to address specific interoperability challenges. Ideally, when OGC standards are implemented in products or online services by two different software engineers working independently, the resulting components plug and play, that is, they work together without further debugging.

Considering the fact that zarr already has a mature spec, I believe it would be advantageous for us to register that spec with OGC. The effort required would be minimal, since the spec is well written. But it would give a certain level of credibility with certain communities.

cc @percivall, who first made me aware of OGC standards in https://github.com/pangeo-data/pangeo/issues/450

[percivall]

Thanks @rabernat for continuing this discussion here. Considering zarr is already stable makes it an excellent candidate for standardization in OGC. How shall we proceed? Some discussion either F2F or via telecon might be most efficient. Will folks be at the ESIP Federation meeting in Bethesda to discuss this?

[jhamman]

Hi @percivall. I'm sorry this was dropped back in January. Can you outline the process that Zarr would need to go through in order to find standardization in OGC? There are weekly Zarr developer calls (Wed. 1p PT, 4p ET). Perhaps we can add this subject to the agenda of an upcoming meeting and you can join?

[percivall]

Thanks @jhamman for keeping this going. Busy times for a few weeks. I could join a Zarr telecon on Wednesday April 17th, 1p PT, 4p ET. Does that work for you guys?

[forman]

I am currently working in an ESA project in which one of the task is to push the specifications of possible data cube APIs and data models. If ZARR goes OGC I'd wish to see something like specification extension or "application profiles" in which geo-spatio-temporal data cubes could be specified and where my project can contribute to. This would also include the multi-resolution use case.

[percivall]

On June 26th the OGC Big Data Domain Working Group agreed to distributed "Geospatial Coverages Data Cube Community Practice" as a Public Request for Comments (RFC). The Community Practice defines 13 requirements based on the common aspects of 7 data cube implementations. The requirements are either core or extension: Access and Data model are in the core; Analytics are extensions as there is less commonality. The discussion in the Big Data DWG about analytics regarded use of WCPS, WPS, Python/Juypter, ZARR and others. Initiating ZARR as an open standard in OGC would be timely over the next few months as we continue the discussion of something like: "server-side analytics on geospatial data cubes."

A good opportunity for ZARR discussion would be at the next OGC meeting in Banff Canada, September 9-12. http://ogcmeet.org/

[meggart]

This is great news. Would you mind to share a link to the Public Request for Comments, since I could not find it after a short web search.

[percivall]

The document has been approved for release after editing (which is my job). The document should be ready in a couple weeks. I should have been more clear in my previous post - my bad. I will share the link here when it is ready.

[percivall]

Planning for ESIP and OGC API - Coverage/Arrays Analytics Sprint January 6-7, 2020 | Bethesda, MD Draft description of the sprint:

Note in particular the discussion about engaging the ZARR communities in this event. Please invite comment on the description of the Sprint Registration will open soon, but looking to get an early indication of interest for planning. The event will be discussed next week at the OGC Technical Committee meeting, Banff Canada.

[jeffdlb]

George et al.-

It seems that an OGC Community Standard (https://www.opengeospatial.org/standards/community) rather than a formal OGC® Standard would be a good way forward, if you are still doing those. See for example the LAS spec for LIDAR point cloud data format on that page.

-Jeff DLB

[percivall]

The OGC EDR API sprint is currently testing the EDR API on Zarr data structures. https://www.ogc.org/ogcevents/environmental-data-retrieval-edr-api-sprint

[rabernat]

Hi @percivall. Just following up on this long-standing issue.

Can you clarify what steps we would need to register the Zarr spec as an OGC community standard? I could not figure this out by reading the OGC website.

[percivall]

A telecon with Scott Simmons, lead of OGC Standards Program, would be a good way to discuss the OGC Community Standard process. Switching to email or other way to identify date and time may be best: percivall@ogc.org

[rabernat]

Email sent.

That said, it would be lovely if we could conduct whatever discussions are needed asynchronously and transparently, in a public forum such as this one.

[percivall]

Email received: introductions of Ryan and Scott accomplished.

Information about OGC Community Standards:

• An OGC Community Standard is an official position of the OGC endorsing a specification or standard developed external to the OGC. A Community standard is considered to be a normative standard by OGC membership and part of the OGC Standards Baseline.
• List of current OGC Community Standards
• Policies and Procedures for adoption of OGC Community Standards are part of the OGC Consensus Standards Process. The specific process for Community Standards are here:

[rabernat]

In consultation with George and Scott, I have prepared a first draft of our OGC Community Standard Justification application.

https://docs.google.com/document/d/1TWs4NjddROo6aRTuabqDVaJAfIOJ2MjoQeM7iL2x3C4/edit?usp=sharing

I would like to share this at today's meeting and, if possible, obtain unanimous support from @zarr-developers/steering-council.

[joshmoore]

:+1: from me.

[alimanfoo]

:+1: from me too, thanks @rabernat.

[meggart]

Linking the discussion here, if anyone feels like commenting: https://www.ogc.org/standards/requests/214

[jakirkham]

cc @DennisHeimbigner @WardF