search

OPENDAP / bes

The BES framework, which forms the basis for the Hyrax server
GNU Lesser General Public License v3.0
13 stars 19 forks source link

Missing zlib (deflate) compression level from dmrpp files #954

Open ayushnag opened 2 months ago

ayushnag commented 2 months ago

I am working on parsing dmrpp metadata files to match the format of zarr metadata files (parser PR here). The only piece of missing information I have encountered in dmrpp files is the zlib (deflate) compression level. It seems like the opendap dmrpp generator has this logic to retrieve and store this attribute as deflateLevel (see example below) but it is not implemented across many files. Many NASA datasets including SWOT L2 SSH, MUR L4 SST, and Ice-SAT are missing this field.

Example dmrpp file WITH zlib level

Example dmrpp file WITHOUT zlib level

ndp-opendap commented 2 months ago

You might check the version of the build_dmrpp application that was used to build the dmr++ files that lack the information you are seeking. This should be available in the metadata of the dmr++:

    <Attribute name="build_dmrpp_metadata" type="Container">
        <Attribute name="build_dmrpp" type="String">
            <Value>3.20.13-310</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.20.13-310</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.20.11-93</Value>
     </Attribute>

It may be the case that these dmr++ files were created prior to the deflateLevel being included in the output. If so, we would like to know which NASA collections you encounter that lack this, as we may be able to get them to rebuild those files with the most recent build_dmrpp application.

ndp-opendap commented 2 months ago

After reviewing the examples I think we'll need to appeal to @kyang2014 for his insights on this.

ayushnag commented 2 months ago

I can point to some examples of NASA datasets/dmrpp that have this issue but I'm not sure how to find all of them. The file I linked earlier with deflateLevel was build_dmrpp version 3.20.13. Some of the files below are above that version number and still missing deflateLevel. Could it be a DAAC/collection level issue?

https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20240101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc.dmrpp

metadata:

        <Attribute name="build_dmrpp" type="String">
            <Value>3.20.13-664</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.20.13-664</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.20.11-198</Value>
        </Attribute>

https://archive.swot.podaac.earthdata.nasa.gov/podaac-swot-ops-cumulus-protected/SWOT_L2_LR_SSH_2.0/SWOT_L2_LR_SSH_Expert_017_463_20240705T134333_20240705T143458_PIC0_01.nc.dmrpp

metadata:

        <Attribute name="build_dmrpp" type="String">
            <Value>3.21.0-272</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.21.0-272</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.21.0-70</Value>
        </Attribute>

This file below does have deflateLevel

https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/VNP14A1.002/VNP14A1.A2024189.h00v08.002.2024190091116/VNP14A1.A2024189.h00v08.002.2024190091116.h5.dmrpp

metadata:

        <Attribute name="build_dmrpp" type="String">
            <Value>3.21.0-46</Value>
        </Attribute>
        <Attribute name="bes" type="String">
            <Value>3.21.0-46</Value>
        </Attribute>
        <Attribute name="libdap" type="String">
            <Value>libdap-3.21.0-27</Value>
        </Attribute>
jgallagher59701 commented 2 months ago

Kent: There is a question about DMR++ files and the deflateLevel information. Can you answer it?

On Jul 8, 2024, at 4:02 PM, Ayush Nag @.***> wrote:

I can point to some examples of NASA datasets/dmrpp that have this issue but I'm not sure how to find all of them. The file I linked earlier with deflateLevel was build_dmrpp version 3.20.13. Some of the files below are above that version number and still missing deflateLevel. Could it be a DAAC/collection level issue?

I don’t know the exact version where the deflateLevel starts appearing, but we added that when we implemented a specific feature that required the information. To simply decompress those chunk, you don’t need this information, but we have a feature where chunks from an HDF5 file can be directly passed through to a netCDF4 file so that the HDF5 file can be subset with the result stored in a netCDF4 file without the expensive decompression and compression operations being performed. For this to work, we need to set the deflate level of the chunks as they are written to the netCDF4 file.

James

PS. Nathan is right, we should ask Kent about which versions of the DMR++ builder have this feature. They will be recent versions.

https://archive.podaac.earthdata.nasa.gov/podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1/20240101090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1.nc.dmrpp

metadata:

    <Attribute name="build_dmrpp" type="String">
        <Value>3.20.13-664</Value>
    </Attribute>
    <Attribute name="bes" type="String">
        <Value>3.20.13-664</Value>
    </Attribute>
    <Attribute name="libdap" type="String">
        <Value>libdap-3.20.11-198</Value>
    </Attribute>

https://archive.swot.podaac.earthdata.nasa.gov/podaac-swot-ops-cumulus-protected/SWOT_L2_LR_SSH_2.0/SWOT_L2_LR_SSH_Expert_017_463_20240705T134333_20240705T143458_PIC0_01.nc.dmrpp

metadata:

    <Attribute name="build_dmrpp" type="String">
        <Value>3.21.0-272</Value>
    </Attribute>
    <Attribute name="bes" type="String">
        <Value>3.21.0-272</Value>
    </Attribute>
    <Attribute name="libdap" type="String">
        <Value>libdap-3.21.0-70</Value>
    </Attribute>

This file below does have deflateLevel

https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/VNP14A1.002/VNP14A1.A2024189.h00v08.002.2024190091116/VNP14A1.A2024189.h00v08.002.2024190091116.h5.dmrpp

metadata:

    <Attribute name="build_dmrpp" type="String">
        <Value>3.21.0-46</Value>
    </Attribute>
    <Attribute name="bes" type="String">
        <Value>3.21.0-46</Value>
    </Attribute>
    <Attribute name="libdap" type="String">
        <Value>libdap-3.21.0-27</Value>
    </Attribute>

— Reply to this email directly, view it on GitHub https://github.com/OPENDAP/bes/issues/954#issuecomment-2215423474, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7Q4KV24TVGQEKWUOMK7ELZLMEAHAVCNFSM6AAAAABKRHRIHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJVGQZDGNBXGQ. You are receiving this because you are subscribed to this thread.

kyang2014 commented 1 month ago

I double checked the date I added the deflateLevel to the dmrpp file. It is Sep. 5th, 2023 with https://github.com/OPENDAP/bes/pull/809

And the tag for this pull request is https://github.com/OPENDAP/bes/releases/tag/3.20.13-845. which is bes-3.20.13-845. So BES 3.20.13-664, the one listed as without having the deflateLevel, was generated before the time I added this feature. The other two examples that have deflateLevels were generated after the time I added this feature.

jgallagher59701 commented 1 month ago

Thanks Kent.

On Jul 10, 2024, at 9:18 AM, kyang2014 @.***> wrote:

I double checked the date I added the deflateLevel to the dmrpp file. It is Sep. 5th, 2023 with #809 https://github.com/OPENDAP/bes/pull/809 And the tag for this pull request is https://github.com/OPENDAP/bes/releases/tag/3.20.13-845. which is bes-3.20.13-845. So BES 3.20.13-664, the one listed as without having the deflateLevel, is before the time I added this feature. The other two examples that have deflateLevels are after the time I added this feature.

— Reply to this email directly, view it on GitHub https://github.com/OPENDAP/bes/issues/954#issuecomment-2220490468, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB7Q4KWZQE4JA7G6SL3BL33ZLUYB5AVCNFSM6AAAAABKRHRIHOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQGQ4TANBWHA. You are receiving this because you commented.