Appendix A, common formats list: should scientific formats be included?

[lvdbrink]

Should scientific formats be included in Appendix A, common formats list?

Is there any reason from across the group why the current matrix of spatial data formats doesn't include formats such as GRIB, HDF4, HDF5, netCDF3, netCDF4, etc? These are used pervasively throughout the sciences.

See discussion here: https://lists.w3.org/Archives/Public/public-sdw-wg/2016Mar/0083.html

[lvdbrink]

Criteria: proper open standard (in UK gov sense); usable on the web (i.e. no need to download it in order to work with it).

@jonblower would you take a look at this issue?

[lvdbrink]

The formats mentioned above seem to be open but not webby.

[6a6d74]

@jonblower - could you indicate whether these formats can be used "on the Web" (rather than having to download and use offline) ... this is why you've developed CovJSON!?

[jonblower]

Well just to clarify a possibly-obvious point, even with JSON you are still downloading the data - it's just that it's easier to work with directly in the web browser as opposed to handing off to a separate tool. I'm not sure that there is an equivalence between "on the web" and "easily processable in a web browser" so we might have to be careful with our choice of words.

But anyway, all those formats are binary formats that typically require the user to download them and use in an alternative tool. Technically, there would be nothing stopping someone writing a GRIB decoder for Javascript - it's just that it doesn't exist yet AFAIK.

I'm not sure what a good definition of a "webby" format ought to be, but some useful characteristics that favour in-browser processing might include:

• Text-based, avoiding the need for specialist codecs
• Individual files small enough to load entirely into the browser's memory
• Mechanism to link between files, so that large datasets can be split into smaller linked files
• Open, non-proprietary specification

(And yes @6a6d74, these were the kinds of things we had in mind when we developed CovJSON!)

Formats designed for efficient archiving (or bulk offline processing) would typically not have all of these characteristics.

[6a6d74]

Talking with Chris Little, we suggest that this topic is discussed in the upcoming OGC TC meeting (St. Johns, Canada - June 2017) ... there's an Earth Systems Science DWG that could provide an appropriate home for the discussion.

If we want to add scientific formats based on a discussion there, we can add them in a "final" version - but we'll leave this issue pending for the next Working Draft release.

[ghost]

Agree lets discuss at the TC... in my view the focus needs to be on mainstream publishers for which scientific data may not be relevant..

[lieberjosh]

There is a Geosemantics DWG session scheduled for both present deliverables and upcoming activities

--Josh

On May 3, 2017, at 6:11 AM, Jeremy Tandy notifications@github.com wrote:

Talking with Chris Little, we suggest that this topic is discussed in the upcoming OGC TC meeting (St. Johns, Canada - late June) ... there's an Earth Systems Science DWG that could provide an appropriate home for the discussion.

If we want to add scientific formats based on a discussion there, we can add them in a "final" version - but we'll leave this issue pending for the next Working Draft release.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/w3c/sdw/issues/237#issuecomment-298871731, or mute the thread https://github.com/notifications/unsubscribe-auth/AExWhhuy1_YG6s0qe7tI_3frluxA9rYZks5r2FLWgaJpZM4H6f0O.

[6a6d74]

I am not aware of any resolution of this issue coming from the OGC TC meeting. Will add a Note to the document summarising the points that @jonblower made, and referencing CovJSON.