Make nc_open_mem functionality available thru Java.

[DennisHeimbigner]

esupport issue: DMF-819954

I need to read a netcdf from a stream (the netcdf is stored in HDFS, there is no way to access the underlying local file system). How can I do this ? The file is small, about 20Mb, I guess I can store it in a byte array but how can I use the NetCDF java API to access the content ?

We can provide this functionality using nc_open_mem in the netcdf-c library. This will require some work to expose it and to force use of the jna netcdf reader.

[lesserwhirls]

Is it possible that this could be done as an IOSP rather than forcing all netCDF (I assume netCDF-4) reading to be done through the JNA netCDF reader? I assume this would intended for netCDF-Java, and not for use in the TDS, given the performance hit we've seen when using the jna netCDF reader?

[DennisHeimbigner]

Yes, I had assumed this would be only for client-side java. Also, I would expect the user to have to do something specific to enable this functionality. Probably using the -D argument that Christian set up to force use of the Netcdf Iosp over HDF5. Also, what performance issues are you referring to. Seems like those need addressing.

[lesserwhirls]

I recall that @cwardgar added the -D ability when there was an issue reported in the HDF5 IOSP. The user reported performance issues when reading through the netCDF IOSP. @cwardgar - do you remember any of those details?

[cwardgar]

the -D argument that Christian set up to force use of the Netcdf Iosp over HDF5.

In #558, I added the Netcdf4Clibrary.useForReading property to threddsConfig.xml and nj22Config.xml. There is no command-line switch for it. There probably should be, but we'd need to add some code to make sure that the 3 separate means of setting that property don't interfere with each other.

given the performance hit we've seen when using the jna netCDF reader?

I'm not aware of any performance hit when using Nc4Iosp vs H5iosp. However, as we saw in #518, Nc4Iosp will produce the expected result from some datasets that H5iosp won't.

We can provide this functionality using nc_open_mem in the netcdf-c library.

We already have something similar. There should be no need to add nc_open_mem, right?

[lesserwhirls]

The performance issue I was referring to is in Unidata/thredds#518, specifically this comment, although it seems to be a caching issue not necessarily tied to anything JNA specific.

[DennisHeimbigner]

The openInMemory RAF will only work with H5Iosp, not Nc4Iosp, so if you can live with that, then that would solve this problem.