search

covjson / specification

CoverageJSON specification
https://covjson.org/spec/
45 stars 6 forks source link

Quality flags #65

Closed letmaik closed 8 years ago

letmaik commented 8 years ago

I'm just looking at some typical quality flag encodings, e.g. https://lpdaac.usgs.gov/sites/default/files/public/modis/docs/MODIS_LP_QA_Tutorial-2.pdf and wondered how that could be done in CovJSON.

Obviously all these flags can be represented as categorical parameters, in the example above that would be 9 of those, each having between 2 and 11 categories.

As binary bit masks each grid cell takes 2 bytes for representing all flags. With CovJSON's categorical parameters we'd need at least one to two bytes per flag, so in total around 18 bytes per grid cell for flags.

I have a feeling that replicating this bit mask encoding in CovJSON would go too far in terms of simplicity. On the other hand, embedding such quality flags as categorical parameters/ranges (if there are many) would likely not be a good idea since the document would get too big, so the relevant ranges would have to be linked, which would maybe rule out some use cases like the standalone fully embedded output for WCS or others, but maybe not, since you can do parameter subsetting in WCS and could only request the flags you're interested in.

So, I just wanted to put that on the table. I don't think we should complicate this and just use categorical parameters as they are. Any opinions?

jonblower commented 8 years ago

I agree, we should not complicate the spec for this, unless it turns out to be a significant problem in real use. You could ask @GerardoLopez for typical use cases - maybe most use cases don't need all the quality flags as you say?

letmaik commented 8 years ago

Just to follow up, Gerardo said you should always use all available quality flags if you do serious scientific processing on a dataset. But I think our focus is slightly different here, so I would suggest we leave it at that for now and rely on linked ranges and WCS parameter subsetting or similar techniques if the full set of quality parameters have to be exposed.

jonblower commented 8 years ago

Yes, I think bit masks would be a bit too complex for a web use case. We can still expose all the quality flags, but would have to do so as separate ranges I think, if we really needed to. Can we use ParameterGroups to express the links between the ranges?

letmaik commented 8 years ago

Possibly, there could be a higher-level observed property called "quality flags" that the parameter group could be associated to. Out of scope for us I think.