stcorp / harp

Data harmonization toolset for scientific earth observation data
http://stcorp.github.io/harp/doc/html/index.html
BSD 3-Clause "New" or "Revised" License
55 stars 18 forks source link

Develop HARP ingestion standard #306

Open StevenCompernolle opened 1 week ago

StevenCompernolle commented 1 week ago

HARP conventions are being used more and more not only for in-memory representation, but also to archive data. Data storage encoding/decoding optimization techniques are however not meant to be included in the HARP conventions

In this more specific issue, #305 (about encoding pressure profile with hybrid sigma-pressure coefficients), Sander formulated as follows

I do realise that the HARP conventions are now getting considered more and more as an actual storage/archive format. And for archival/distribution purposes, I understand the need for this kind of storage optimisation. However, this would not be something for the actual interface conventions themselves. If we would introduce some kind of format that would combine HARP conventions and specific compression/encoding techniques, then this would have to become its own kind of sub-standard, with a special importer in the HARP software (similar to all the other foreign-format importers) that would decode and remove all these compressed elements automatically as soon as they are read. Defining such a standard would take some consideration though. We should then look into further cases that should be supported here. Introducing such a standard purely for the pressure profile compression would be a bit overkill.

The current issue is meant to cover this more general question of a HARP ingestion standard.