search

Unidata / netcdf-java

The Unidata netcdf-java library
https://docs.unidata.ucar.edu/netcdf-java/current/userguide/index.html
BSD 3-Clause "New" or "Revised" License
146 stars 70 forks source link

Processing openInMemory fails for non-filesystem datasets #482

Closed brentlemons closed 3 years ago

brentlemons commented 4 years ago

version: 5.3.3 Java 11 linux and macos filesystems

Call in question: NetcdfFile ncf = NetcdfFiles.openInMemory("s3test", data);

Exception:

No records found in dataset s3test: java.lang.IllegalStateException
java.lang.IllegalStateException: No records found in dataset s3test
    at ucar.nc2.grib.collection.Grib2CollectionBuilder.makeGroups(Grib2CollectionBuilder.java:139)
    at ucar.nc2.grib.collection.GribCollectionBuilder.createMultipleRuntimeCollections(GribCollectionBuilder.java:128)
    at ucar.nc2.grib.collection.GribCollectionBuilder.createIndex(GribCollectionBuilder.java:120)
    at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:830)
    at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromDataFile(GribCdmIndex.java:807)
    at ucar.nc2.grib.collection.GribCdmIndex.openGribCollectionFromRaf(GribCdmIndex.java:777)
    at ucar.nc2.grib.collection.GribIosp.build(GribIosp.java:148)
    at ucar.nc2.NetcdfFiles.build(NetcdfFiles.java:787)
    at ucar.nc2.NetcdfFiles.open(NetcdfFiles.java:726)
    at ucar.nc2.NetcdfFiles.openInMemory(NetcdfFiles.java:674)
    at com.stuff.noaa.hrrr.HRRRParseFunctionHandler.handleRequest(HRRRParseFunctionHandler.java:122)
    at com.stuff.noaa.hrrr.HRRRParseFunctionHandler.handleRequest(HRRRParseFunctionHandler.java:68)

openInMemory creates an InMemoryRandomAccessFile. The filename property in the InMemoryRandomAccessFile is set to the name of the dataset. When passing a byte array, this does not represent a file on the filesystem.

The raf is passed to openGribCollectionFromRaf(...). From there it is passed to openGribCollectionFromDataFile(...). Within this method, the following occurs:

String filename = dataRaf.getLocation();
File dataFile = new File(filename);

MFile mfile = new MFileOS(dataFile);
return openGribCollectionFromDataFile(isGrib1, mfile, updateType, config, errlog, logger);

dataRaf.getLocation() only points to a made up name, therefore there is no file to open. As soon as it tries to use the "file", it exceptions out with the above error.

This occurs with openInMemory using byte[] as well as URI. It doesn't happen when you openInMemory with a filename because it is later able to open the file. However, that means it is actually loading the file into memory, or randomly accessing later, the same file/data.

JohnLCaron commented 3 years ago

What is in the data array?

The problem with GRIB is that it needs the GRIB index (gbx9) file in addition to the binary GRIB file. So there would have to be a special API for that, telling it to put the gbx file into memory also I suppose.

JohnLCaron commented 3 years ago

See Issue #638