search

isce-framework / nisar-workflows

3 stars 1 forks source link

Update public sample data product #4

Closed alex-s-gardner closed 2 months ago

alex-s-gardner commented 7 months ago

I've found issues with reading the new HDF products that @vbrancat with various tools and I would like to work through these with product developers. I'm not able to replicate several of the issues with the sample data products that are posted on the project website: https://nisar.jpl.nasa.gov/data/sample-data/

Would it be possible to update the sample data products to the output from the latest version of the processor. This would help facilitate getting the tooling in place and better identify bugs in the products (where/if they exist).

hfattahi commented 7 months ago

Thanks @alex-s-gardner for your comment. Short answer is that yes we will provide new version of the sample products on NISAR website soon. In the meantime would you provide more details on which your side? Which products you have had issue with and with which tool and how that has been with the sample products on the NISAR website?

alex-s-gardner commented 7 months ago

It seems only 2 or 5 datasets can be read by NCDatasets.jl, this is the default utility for reading CF compliant nc and hdf5 datasets into Julia.. it also is the backbone to other packages like Rasters.jl.

ROFF and RUNW can be read without issue.

GOFF, GUNW and RIFG all throw “Key not found” errors.

ScienceCat18 commented 7 months ago

@alex-s-gardner to clarify, are these issues related to the sample products recently provided to ST offline, generated by Max, Virginia and I (using PCM)? or the sample products on the NISAR website?

alex-s-gardner commented 7 months ago

@ScienceCat18 this is for the files that were provided to ST offline. Sample NISAR products on website seem to work fine.

vbrancat commented 7 months ago

Providing background information on the issue posted by @alex-s-gardner.

The NISAR InSAR sample products available on the NISAR website can be opened in Julia using the packages HDF5.jl and NCDataset.jl. The RIFG, GUNW, and GOFF products generated with the latest version of the ISCE3 software cannot be opened (and manipulated) with NCDataset.jl but they can still be opened using HDF5.jl.

I managed to track down the source of the issue for the RIFG and GUNW products. Both these products contain a complex HDF5 Dataset which is the complex wrapped interferogram. This Dataset contains an HDF5 Attribute called _FillValue whose value is used to label pixels with missing or undefined data. The CF-1.7 convention requires that the data type of the _FillValue Attribute must be the same data type as its associated Dataset (see CF Conformance Requirements and Recommendation). Complex Datasets within HDF5 NISAR products will have a_FillValue = np.nan + 1j * np.nan. The 'NCDataset.jl' seems to do not recognize this and throws a sort of value error. Given the above, it seems that there is an issue related to how 'Dataset.jl' recognizes and manipulates complex data.

vbrancat commented 2 months ago

Closing this issue as it has been addressed.

NISAR sample products are routinely updated at every official release of the ISCE3 software. They are available on the NISAR website.