Consolidation of `cf-json` and `nco-json`

[kwilcox]

Hey folks, I think most of you were involved in the conversation about a year and a half ago regarding CovJSON, cf-json and nco-json. Most was on the cf-metadata mailing list and carried over to https://github.com/covjson/specification/issues/86.

I'd like to focus in this issue on resolving the differences between cf-json and nco-json so we can focus on developing and promoting a single spec. I myself use nco-json extensively. Here are what I consider the main differences and please post any objections or clarifications as it has been a long while since we had this conversation

1. data is required in cf-json and optional in nco-json
2. nco-json supports groups. I've barely kept up with CF-2.0 discussions but supporting this now might be a good idea?
3. Support for flattening the data arrays. The client is then responsible for reshaping to the defined variable shape
4. Support for typing both variables and attributes

@czender @BobSimons @ChrisBarker-NOAA @rsignell-usgs @pedro-vicente @davemetocean @aportagain

[czender]

Kyle, First a general comment and then some clarifications. NCO-JSON can losslessly express in JSON any CF-compliant file so long as CF is based on netCDF (i.e., presumably for a long time). My understanding (someone please correct me) is that CF-JSON is/was intended to deal with higher level structures that could benefit from additional abstraction, e.g., a CRS object (?). In any case it is unclear that CF-JSON was designed or intended to represent the entire netCDF feature space (as opposed to the CF subset of that space), so I do not think NCO-JSON and CF-JSON are necessarily in conflict. In fact, my impression was that CF-JSON did adopt NCO-JSON about two years ago, and that now CF-JSON has no differences from NCO-JSON in pedantic mode aside from possibly some CF-specific higher level abstractions. I hope someone from CF-JSON clarifies this.

1. NCO-JSON requires data for variables always, but in non-pedantic mode omits data from attributes. Is this what you're referring to? All the early adopters of NCO-JSON that I am aware of seem to prefer pedantic mode, which requires data for attributes too. I'm having second thoughts about making non-pedantic mode the default for NCO, and am considering make pedantic mode the default. Then it could be said that NCO, like CF-JSON, requires data (unless the user opts-in to the non-pedantic mode). Thoughts?
2. Groups seem to be on their way into CF. Maybe in CF-1.8. Maybe even this year.
3. NCO-JSON supports flattening arrays but structures them by default. Flattening is enabled by adding four to the base NCO-JSON format, e.g., --jsn_fmt=4. I do not think flattening by default is a good idea.
4. NCO-JSON requires typing for all attribute types only in pedantic mode. Point 1 above would also address this. More thoughts on this? Best, Charlie

[kwilcox]

In fact, my impression was that CF-JSON did adopt NCO-JSON about two years ago, and that now CF-JSON has no differences from NCO-JSON in pedantic mode aside from possibly some CF-specific higher level abstractions. I hope someone from CF-JSON clarifies this.

Also hoping for clarification on if cf-json features were intended to be a sub-set, super-set or the same set as nco-json. It seems to me like we should all advocate together for the same format and I don't quite understand why we have two formats!

1. I was only referring to the data field on variables which contains the data in array form. The cf-json spec says it is required. It is not required in nco-json (I can dump as JSON using ncks -mM --json and it is omitted). I'm advocating for this to remain optional. My usage of nco-json revolves around allowing analysts to curate metadata records (attributes) externally to the data. It is stored and indexed in nco-json. The nco-json is applied to netCDF files later on in a pipeline. I don't require the data field on variables, this use-case of nco-json could be called "attributes-only" or "data-less".

2. Excellent, let the confusion begin!

3. When I do use the data field on a variable I always flatten because it is more efficient to load as a flat array and reshape into the already-defined-in-the-json shape of the variable, but I agree, it shouldn't be the default.

4. I would like to see cf-json support typing of attributes like nco-json does in levels 2 and 1 of the JSON output (pedantic and pedantic-light mode). I use level 1 with a custom rule that I specifically specify the type unless it is a NC_STRING.

[ChrisBarker-NOAA]

As I see it, CF was/is built on the netcdf data model. Thus you should be able to "do" CF in any format that fully support netcdf. With that in mind, ideally, "CF JSON" would simply be:

"Use nco-json, and follow all the same rules that CF has"

And it sounds like we are close to that, except that maybe it's:

"Use nco-json in pedantic mode, and follow all the same rules that CF has".

So ideally, if there are any discrepancies still, we hash those out for nco-json.

On to some points:

I was only referring to the data field on variables which contains the data in array form. The cf-json spec says it is required. It is not required in nco-json (I can dump as JSON using ncks -mM --json and it is omitted). I'm advocating for this to remain optional. My usage of nco-json revolves around allowing analysts to curate metadata records (attributes) externally to the data. It is stored and indexed in nco-json. The nco-json is applied to netCDF files later on in a pipeline. I don't require the data field on variables, this use-case of nco-json could be called "attributes-only" or "data-less".

Interesting -- I had thought of "optional data" as being when you have a variable only in order to store the attributes, but don't really have any meaningful data to attach. IN that case, while I think netcdf simply does not allow data-less variables, JSON does, so we should allow it -- no data means, well, no data that I care about.

But this use case -- there IS real data, but I want a meta-data-only representation is tricky. In theory, you could want that with netcdf as well -- but I suspect there is any CF legal way to specify metadata-only.

Frankly, I think the the way to handle this is to alter CF, rather than being JSON specific, this is off the cuff, but something like:

""" If a variable has an attribute "meta-data-only" then that attribute should be set to a description of the data type and shape of the data that is associated with the variable, but the actual data will be a minimal length 1 array. """

In any case, whether in all of CF or jsut JSON, it needs to be clear whether no data means there is no data, or if it means I've left the data out -- and if you've left the data out, it's probably a good idea to specify the data type and shape (dimensions?)

3. When I do use the data field on a variable I always flatten because it is more efficient to load as a flat array and reshape into the already-defined-in-the-json shape of the variable

Really? why is that? makes no sense to me -- I would expect the JSON parser in a JS interpreter to be about as efficient as possible, and if you are going to generate all those sub-arrays anyway, there is no memory advantage. Personally, I'd much rather "one way to do it", and that way should be keeping the structure in the JSON.

[ChrisBarker-NOAA]

@czender: what does this have to do with the Consolidation of cf-json and nco-json?

Did you accidentally post to the wrong issue?

[czender]

Ooops! Ignore my TMTS (Too-many-tab-syndrome) behavior

[czender]

I thought there would be unanimous endorsement of making pedantic (jsn_fmt=2) NCO-JSON the default, as I offered above. Kyle uses a variation where all attributes except NC_STRING need to be explicitly typed, and clearly there are other variations that NCO could though currently does not accept as different format variations. More importantly, Kyle's usage points to the tension between the design of JSON, which embodies minimalism and not overcomplicating things (e.g., why type strings?) with the desire of some for completely lossless representations (which ERDDAP likes). Rather than declaring one the winner and THE standard, I would prefer that the standard encompass both and allow some variations (as NCO currently implements with the jsn_fmt parameter). Thoughts? Does that make sense or should there be ONLY one netCDF JSON standard and any variation is non-standard?

[kwilcox]

I'm in favor of a more open ended spec allowing all of current jsn_fmt options to be correct. I can see the use-case for non-pedantic files and don't want to restrict someone who doesn't need to save the attribute types from reducing the complexity of their JSON. I don't think we really need to specify the type of format the file is in... parsers and clients of the spec will have to deal with both typed attributes and non-typed attributes and act accordingly. The spec could specifically state that non-typed attributes should be interpreted as string, float or int (similar to jsn_fmt=1).

[czender]

It sounds like @kwilcox and I agree that it should all be valid. I guess I'm still looking for guidance on whether the NCO default should be jsn_fmt=0, 1, or 2. Ideally that choice would be consistent with the wishes of the largest fraction of the userbase. It's almost a guess at this point, yet based on my informal polling at AGU in December, I think the ERDDAP folks generate the most NCO-JSON. Am I wrong? And Kyle, would it be helpful if NCO added a new jsn_fmt=3 that types all attributes except NC_STRING?

[ChrisBarker-NOAA]

@czender wrote:

the tension between the design of JSON, which embodies minimalism and not overcomplicating things (e.g., why type strings?) with the desire of some for completely lossless representations (which ERDDAP likes).

I"m not sure that quite captures the difference -- I don't know that JSON embodies minimalism, but rather it IS a format with minimal complexity. It was originally designed to be a serialization for Javascript, which is also a pretty simple language (or at least has a limited type system):

• There is one only type of number.
• There is only one text type.
• There is only one "array" type.
• There is only one "mapping" type.

netcdf and CF support a wider set of types, so there is a mismatch. It does seem that there are enough use-cases for simpler JSON that having only one full featured mode is not going to fly.

But I wonder if we can more clearly define the "simpler" use cases, and thus have semi-lossless conversion without being pedantic -- that is, there is a subset of netcdf data that can expressed in a simpler form, and it is clearly defined how to go from JSON to netcdf without a full spec. I"ve lost track a bit, but somethign like:

If there is no datatype specified, then numerical data are doubles, and string data are unicode text with a standard encoding. (Sorry, I've really lost track of how to express that in netcdf :-) )

If there is no dimensionality provided for an array, it is the dimensionality it is presented in -- that is, 1D is 1D, nested for higher dimensions.

If there is no data, then the data is irrelevant (that is, an attribute only variable) -- in that case.

[note: we could maybe make a distinction between integers and floats if we wanted]

When going from netcdf => JSON, we could also have well defined rules:

All numeric types are JSON numbers All text types are JSON strings All ND arrays are displayed as nested All empty data arrays are left off.

This would allow almost loss-free round tripping ....

In fact, the JSON writer would check for values that are out of bounds and raise a warning or error. Though now that I think about it, out of bounds values are going to be a problem anyway...

Finally, maybe the default behaviour could be the less pedantic mode when there is no ambiguity.

-CHB

[czender]

@ChrisBarker-NOAA These are helpful suggestions, and they appeal to my minimalistic side. Moreover, they largely though not completely consistent with how the default NCO implementation, aka jsn_fmt=0, prints attributes. Level 0 formats attributes without types if possible. One difference is that NCO currently has no way of printing variables that way. It calls for a new formatting level which simplifies the printing of variable data similar to attribute data. I think it is worth doing because, as you say, it preserves the essential data in the sprit of JSON, with minimal overhead.

[ChrisBarker-NOAA]

Level 0 formats attributes without types if possible.

How often are non-string attributes actually used? For my part, I always thought that strings were the only option for attributes (until the conversation about JSON...)

But anyway, it is fully specified and lossless to have default types for JSON. That is, specifying that:

[1.0, 2.3, 4.5]

is an array or doubles is exactly the same as not specifying the type.

So could we say that anywhere a type is not specified it IS the JSON type?

Then folks converting netcdf to JSON would, by default get "untyped" JSON for doubles, and could opt in for untyped (uptyping) for floats, for example.

I'm still a bit stuck on strings, as I am very confused about how one is supposed to handle unicode in netcdf.

[kwilcox]

How often are non-string attributes actually used? For my part, I always thought that strings were the only option for attributes (until the conversation about JSON...)

Non-string attributes are essential. _FillValue, missing_value, valid_min|max|range, scale_factor, add_offset are some "standard" based one that need to be typed to be used correctly.

I also wanted to clear up that the intention of cf-json and nco-json was to be a a convenient language-agnostic, text-based, and easily-parse-able (in contrast to CDL) data format. While JSON may have its roots in JavaScript, its usage is well beyond that.

1. data is required in cf-json and optional in nco-json

I'm glad this spurred more ideas and thoughts than I thought it would. I've noticed other groups talking about a generic text-based metadata format and nco-json without a variable's data included is what I've been using for a year or so. I'd like to avoid having to call this kyle-json and somehow build this into the spec. I have never really thought about how someone would "know" if it was data-less or not, and even if that is something that would be required. Someone could just check for the existence of the "data" attribute if they need it. And if they don't need it then can just ignore it?

2. nco-json supports groups. I've barely kept up with CF-2.0 discussions but supporting this now might be a good idea?

I'd like to see cf-json be a formalized nco-json, i.e. the nco-json of an already CF compliant netCDF file. So as CF starts thinking about groups, cf-json needs to adapt how nco-json does groups accordingly. A netCDF file is to nco-json as CF netCDF file is to cf-json. cf-json is still compliant nco-json.

3. Support for flattening the data arrays. The client is then responsible for reshaping to the defined variable shape

I'm good with always keeping the data in the shape of the variable. We decided something already. High fives. ✋

4. Support for typing both variables and attributes

For a lossless round trip of netCDF -> JSON -> netCDF, both the attributes and variables need to be typed. There is no alternative. If cf/nco-json wants to support round-tripping (I think it should), then being able to specify variable and attribute types is a requirement. I think we agree that the ability to type is required, but it should also be optional if someones use-case does not require a 1:1 representation. Type or don't type, it is up to the user. I'd prefer if variables required a type.

[aportagain]

Thanks @kwilcox for bringing this discussion here, and everyone else for their contributions so far! And apologies for not getting involved earlier, unfortunate timing, this issue was opened the day after I went on annual leave and I just got back yesterday :) I'll have a chat with some of our (internal) active users of CF-JSON over the next few days to get their opinions and then reply in more detail.

[aportagain]

Just one quick question from my side in the meantime: @czender, in addition to what's explained in the NCO User Guide (http://nco.sourceforge.net/nco.html#index-_002d_002djson) and some of your recent posters / slides (https://www.essoar.org/doi/pdf/10.1002/essoar.10500689.1 , http://dust.ess.uci.edu/smn/smn_json_esip_201807.pdf), is there any separate, single-document "specification" describing the NCO-JSON format (and levels)? I feel like this could be very valuable, for people like us who are now comparing multiple specs, but maybe even more so for other producer / consumer developers to code against?

[czender]

@aportagain You have mentioned the major written documentation sources. Those cover most but not all of the bases. I am (still) working on a manuscript to document NCO-JSON. It's almost ready to share, and I will share it once it is, as feedback would be valuable. The NCO implementation (e.g., ncks --jsn_fmt=2 in.nc) demonstrates all the current features.

[aportagain]

I've had discussions with our various internal stakeholders, and also had a look at the 2018 email thread (http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2017/009423.html) again that @davemetocean initiated before I myself got involved (he's taken some extended time off to sail around the world so probably can't comment himself anytime soon). Seems to me we just kind of lost momentum back then? So the good news in summary from my perspective: from the cf-json side nothing much has changed, so I'm pretty optimistic we can either still merge with some slight variation / new format level of nco-json, or at least redefine cf-json as a very thin format spec (or maybe rather just "convention"?) on top of this and CF.

[aportagain]

In the interest of one-issue-per-issue I'd probably prefer to split this discussion up into multiple individual issues now, and keep this one for overarching questions if that's alright for everyone... so unless someone objects. Otherwise I'm worried it might get a bit overwhelming for anyone potentially joining or reading up on this later :)

[aportagain]

I'd also be happy to try to organise a conference call, if people think that might be more efficient please let me know. Otherwise I'll prep my various comments within the next week. Sorry for the delay, busy days...

[aportagain]

Alright, I'll start with an easy one, coming back to @kwilcox 's original four points:

1. `data` is required in `cf-json` and optional in `nco-json`

My own perspective in summary: if making data in some way optional has advantages for other people's use cases I'm sure we can deal with that for ours :)

Split out to: https://github.com/cf-json/cf-json.github.io/issues/11

[aportagain]

And another from my perspective easy one:

3. Support for flattening the `data` arrays. The client is then responsible for reshaping to the defined variable shape

I'm pretty certain for our typical current cf-json use cases we'd very much prefer not to (have to) support this, and rather keep it simple, even if in some cases there may be a performance penalty.

Split out to: https://github.com/cf-json/cf-json.github.io/issues/12

[aportagain]

Regarding

2. `nco-json` supports `groups`. I've barely kept up with CF-2.0 discussions but supporting this now might be a good idea?

I guess cf-json currently implicitly only supports the "classic" / netCDF3 data model, while nco-json supports most of netCDF4, just not opaque types yet if I understand correctly (guess we'd mainly have to agree on an encoding for the binary data for that)? Optional support for some or all of the netCDF4 bits sounds great, and the way nco-json does it so far looks good to me.

Having said that, I do think we'd like to preserve a clearly defined (and easily referenced) as-simple-as-possible format... most of our use cases really are quite simple, and we do not want to put extra burden on developers that know they'll only ever process the simple cases (especially on the parser / consumer side). I think we have at least three options that have fairly high-level effects on the data structure / schema: classic or netCDF4 "structure" (maybe without opaque types), pedantic or non-pedantic handling of types (or level 0 / 1 / 2 / ...), metadata-only or metadata-and-data. Most of our use cases somehow interact with what I'd vaguely call the "web space", so for us it's very valuable to have a clear association of expected data structure / schema with some agreed upon MIME / media type, and ideally also being able to represent it in OpenAPI, which in turn is somewhat tied to JSON SCHEMA ("uses an extended subset of"... yet another somewhat messy coupling of standards). I haven't quite yet wrapped my head around how one format with multiple "format levels", and then possibly additional "options" like metadata-only would translate to this space...

[aportagain]

Like @ChrisBarker-NOAA and @kwilcox have mentioned, I too wish that for our cf-json purposes we could just use some variation of nco-json as the actual format with a global Conventions attribute of CF-X.Y... Maybe not impossible, but right now one deal breaker remaining seems to be time coordinate variables: the current CF working draft to my understanding still only allows numerical values, but we really want to keep strings as an option for this, so technically cf-json "breaks" CF compliance in this respect. I'm not entirely across the whole udunits "coupling"... CF still seems to do this, but maybe the Common Data Model version 4 actually doesn't? I found a related trac ticket from over a decade ago: https://cf-trac.llnl.gov/trac/ticket/14 , and funny enough this has recently come up again in a calendar issue: https://github.com/cf-convention/cf-conventions/issues/148 so maybe there's hope?

[aportagain]

The last separate issue that I can think of at the moment concerns JSON null. We've found this quite handy, JSON parsers and generators in multiple languages deal well with it, and I think it's also very much in the as-simple-as-possible spirit. This is more like an item for a wish list for a new / revised format level of nco-json... should I still split it out as a separate issue in this repo here at least for now? Or would it be better to wait until @czender has a draft "format spec" document ready so that we could create issues and potentially pull requests in direct relation to that?

That's probably all from my side before the weekend... :) Let me know what you guys think, and also if you think any of these should be split up further obviously feel free to create new / separate issues!

[ChrisBarker-NOAA]

Maybe not impossible, but right now one deal breaker remaining seems to be time coordinate variables: the current CF working draft to my understanding still only allows numerical values, but we really want to keep strings as an option for this, so technically cf-json "breaks" CF compliance in this respect.

No more than storing time with a different encoding in netCDF breaks CF.

In order to work with datetimes in any environment, you need a decent datetime library — expecting that in a consumer of the data is quite reasonable.

Ideally, we have one spec that maps the netCDF data model to JSON, and one that maps CF to JSON (which is probably almost “just do the same thing as you do in netCDF” we should not extend CF just to make it a little easier for readers without the right tools.

( and most ( I think JavaScript as well) datetime software works with C style seconds-since-an-ephoc under the hood anyway.)

I'm not entirely across the whole udunits "coupling"... CF still seems to do this, but maybe the Common Data Model version 4 actually doesn't?

I hope not — I hate that too. But again, that’s an argument for CF — it’s a bad Idea to change/extend CF itself just for JSON.

And much as I don’t like the Units coupling, if you see it as; “unit handling and definitions for CF are specified in this other doc” it’s not so bad.

And funny enough this has recently come up again in a calendar issue: cf-convention/cf-conventions#148 https://github.com/cf-convention/cf-conventions/issues/148 so maybe there's hope?

Maybe — but that’s kinda stalled out.

-CHB

[ChrisBarker-NOAA]

The last separate issue that I can think of at the moment concerns JSON null .

What would you use it for? Is null supported in netCDF?

If not, do you see it as an alternative to special “missing values” or Fill_Value?

-CHB

[ChrisBarker-NOAA]

TL;DR

I don’t think cf-json should be used as a way to extend CF. As frustrating as it is, the place to move forward is CF is CF itself.

2. nco-json supports groups. I've barely kept up with CF-2.0 discussions but supporting this now might be a good idea?

I guess cf-json currently implicitly only supports the "classic" / netCDF3 data model, while nco-json supports most of netCDF4,

Again, CF is CF, if CF doesn’t support groups, neither does CF-JSON. Whether nco-json supports groups or not is irrelevant.

-CHB

[BobSimons]

Well, anyone can do whatever they want. But there are advantages to sticking to the standard and (in this case) not writing String times as dimension values in files that are supposed to be CF compliant (until the CF standard says it's okay and how to do it): While the format/meaning of the time strings may be obvious to you or a human and may even follow a standard like ISO 8601:2004(E), there is no standard way in CF to specify the format of a time string (e.g., Java's "yyyy-MM-dd'T'HH:mm:ssZ"), so there is no way for software written to follow the CF specification to deal with String dimension values and know what the format is (how to parse them). There are literally 1000's of time formats in use in scientific data files. Some of them can't even be deciphered by humans because 1 or 2-digit year values make the values ambiguous. Let's avoid this problem or deal with it properly (in CF). One of the big advantages of following a standard is that software can work with the files automatically. Otherwise, everyone has to write custom software to deal with each of the non-standard file variants.

On Thu, Mar 28, 2019 at 10:52 PM Alex Port notifications@github.com wrote:

Like @ChrisBarker-NOAA https://github.com/ChrisBarker-NOAA and @kwilcox https://github.com/kwilcox have mentioned, I too wish that for our cf-json purposes we could just use some variation of nco-json as the actual format with a global Conventions attribute of CF-X.Y... Maybe not impossible, but right now one deal breaker remaining seems to be time coordinate variables: the current CF working draft to my understanding still only allows numerical values, but we really want to keep strings as an option for this, so technically cf-json "breaks" CF compliance in this respect. I'm not entirely across the whole udunits "coupling"... CF still seems to do this, but maybe the Common Data Model version 4 actually doesn't? I found a related trac ticket from over a decade ago: https://cf-trac.llnl.gov/trac/ticket/14 , and funny enough this has recently come up again in a calendar issue: cf-convention/cf-conventions#148 https://github.com/cf-convention/cf-conventions/issues/148 so maybe there's hope?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cf-json/cf-json.github.io/issues/10#issuecomment-477877123, or mute the thread https://github.com/notifications/unsubscribe-auth/ABarOADavrf3u_wonUp4YAxtkMSLwv_hks5vbaoMgaJpZM4a5i02 .

-- Sincerely,

Bob Simons IT Specialist Environmental Research Division NOAA Southwest Fisheries Science Center 99 Pacific St., Suite 255A Monterey, CA 93940 Phone: (831)333-9878 Fax: (831)648-8440 Email: bob.simons@noaa.gov

The opinions in this message are mine personally and do not necessarily reflect any position of the U.S. Government or the National Oceanic and Atmospheric Administration. <>< <>< <>< <>< <>< <>< <>< <>< <><

[ChrisBarker-NOAA]

On data types and "levels":

I finally looked at the docs in NCO (thanks Charlie!)

I think there is a distinction between what a file means and what you write. And for the spec, ideally we don't have a bunch of "levels". AND I think we can, in fact, do that by using essentially the level-1 spec:

If the type is not specified, then it is the base JSON type: int, float64, string. If the type is specified, then it is (Obviously) the type specified :-)

So a READER can simply use that spec -- it only has to read one thing. It does make reader code a bit more complex, 'cause it has to check whether or not the type info is there, but once that code is written, you're done.

writers, on the other hand, do need to know how they want to write the data -- essentially whether it's lossy or not (with regard to type), but either way, they are always writing to a single spec.

maybe we could call it:

"type_complete"

or

"type_lossy"

Or something like that?

NOTE: I'm a bit confused about the need for pedantic mode. If we say, for instance, that no type spec always means float64, then there is no ambiguity, is there?

Though the way I'm thinking of it, a "pedantic" file would conform to the spec in any case. The only danger is that a reader that expects pedantic files will not be able to read not-quite-pedantic files.

[aportagain]

The last separate issue that I can think of at the moment concerns JSON null.

What would you use it for? Is null supported in netCDF? If not, do you see it as an alternative to special “missing values” or Fill_Value?

Yup, "alternative" sounds right: since JSON conveniently does provide null as a primitive data type, we use it as a simpler alternative to the values-defined-by-attributes way of identifying missing data.

Split out to: https://github.com/cf-json/cf-json.github.io/issues/13

[aportagain]

I've also split out "time coordinate values as strings": https://github.com/cf-json/cf-json.github.io/issues/14 . It's obviously a can of worms... no surprise there... :)