Unlimited dimension cannot work without a specified length

ddshd commented 8 months ago

Versions impacted by the bug

netcdfAll 5.5.3

What went wrong?

The following dimension in ncml file cannot be parsed by

FileReader reader = new FileReader("/tmp/tempNCML.ncml");
NetcdfDataset ncmlFile = NetcdfDatasets.openNcmlDataset(reader, "/tmp/tempNCML.ncml", null);

ncml:

<dimension name="sample" isUnlimited="true"/>

but adding

length="<some_n>" to the above allows it to work, but negates the point of unlimited.

<?xml version="1.0" encoding="UTF-8"?>
<netcdf xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2  http://www.unidata.ucar.edu/schemas/netcdf/ncml-2.2.xsd" xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
    <attribute name="Conventions" value="CF-1.7" type="string"/>
    <attribute name="Metadata_Conventions" value="Unidata Dataset Discovery v1.0" type="string"/>
    <attribute name="standard_name_vocabulary" value="CF Standard Name Table (v35, 20 July 2016)" type="string"/>

    <dimension name="sample" isUnlimited="true"/>
    <variable name="time" type="double" shape="sample">
        <attribute name="long_name" value="sample" type="string"/>
        <attribute name="standard_name" value="time" type="string"/>
        <attribute name="units" value="test" type="string"/>
        <attribute name="axis" value="T" type="string"/>
    </variable>
</netcdf>

Relevant stack trace

java.lang.IllegalArgumentException: Dimension sample does not exist
        at ucar.nc2.Dimensions.makeDimensionsList(Dimensions.java:125)
        at ucar.nc2.Group$Builder.makeDimensionsList(Group.java:1252)
        at ucar.nc2.Variable$Builder.setDimensionsByName(Variable.java:2025)
        at ucar.nc2.internal.ncml.NcmlReader.readVariableNew(NcmlReader.java:1062)
        at ucar.nc2.internal.ncml.NcmlReader.readVariable(NcmlReader.java:940)
        at ucar.nc2.internal.ncml.NcmlReader.readGroup(NcmlReader.java:625)
        at ucar.nc2.internal.ncml.NcmlReader.readNetcdf(NcmlReader.java:529)
        at ucar.nc2.internal.ncml.NcmlReader.readNcml(NcmlReader.java:478)
        at ucar.nc2.internal.ncml.NcmlReader.readNcml(NcmlReader.java:336)
        at ucar.nc2.dataset.NetcdfDatasets.openNcmlDataset(NetcdfDatasets.java:170)
...

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed? Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

[x] I agree to follow the UCAR/Unidata Code of Conduct

tdrwenski commented 8 months ago

I believe isUnlimited means that the dimension has a size but can grow and isVariableLength means the size is not currently known, see here. So perhaps isVariableLength is more what you are looking for?

tdrwenski commented 8 months ago

Actually, let me walk back my last comment a bit-- I don't know if you should use isVariableLength here-- it is usually used as the dimension for a variable that is a "ragged array" (see "VLEN" section here if you are interested).

But regardless, the behavior of isUnlimited that you show is correct-- you need to specify a starting length for the dimension, though it is allowed to grow.

ddshd commented 8 months ago

But regardless, the behavior of isUnlimited that you show is correct-- you need to specify a starting length for the dimension, though it is allowed to grow.

Is it possible to grow the variable after the file is already out of define mode? I'm facing an issue where it's not possible to write an array of 1D data greater than the size of the unlimited variable's initial variable.

Is it expected that we must know the max size of data we need to write (and to set it as the initial value of the unlimited variable) before going out of define mode?

Unidata / netcdf-java