search

NOAA-PMEL / Ferret

The Ferret program from NOAA/PMEL
https://ferret.pmel.noaa.gov/Ferret/
The Unlicense
55 stars 20 forks source link

Python module for interfacing with ncStream #1194

Open karlmsmith opened 6 years ago

karlmsmith commented 6 years ago

Reported by @karlmsmith on 27 Feb 2012 18:20 UTC Investigate and implement a Python module for reading ncStream data and creating something in Python that contains the metadata and the data array. (Investigate == see if someone is already working on this; if not, bounce ideas off some folks.) The target of this module would be something that pyferret.putdata or pyferret.put could use to make the data available to Ferret.

From http://www.unidata.ucar.edu/software/netcdf-java/stream/NcStream.html

NetCDF Streaming Format (ncstream) is a write-optimized encoding 
of CDM datasets. Ncstream consists of a series of header and data 
messages, in any order. Writes are always appended. Later messages 
override earlier ones whenever they overlap or conflict. To add or 
modify structural metadata, simply append a new header message. Each 
data message identifies the variable and the section (rectangular 
subset) of data it contains. A variable's data thus consists of the 
collection of data messages for it, if any.

Migrated-From: http://dunkel.pmel.noaa.gov/trac/ferret/ticket/1922

karlmsmith commented 6 years ago

Comment by steven.c.hankin on 27 Feb 2012 21:07 UTC One challenge to keep in mind (arguably a "version 2" consideration): to the degree that ncStream is truly "stream" oriented, the size of the result set from a query cannot be known in advance. It may prove so big that it needs to be aborted. In such cases one assumes the desired behavior of t a plotting client would be to (say) plot the data progressively as it is read and give the user the power to interrupt it. It would be worth thinking about how an effect like this might be achieved in Ferret. If the ncStream reads could be handled in a separate (asychronous) thread, then Ferret might be given a partial result and allowed to plot it as the read continued. Would then need a way for ferret to inform the user (or script) that further data was available, and to (somehow) perform an "update" that extended the array sized of the previously "put" data.