search

NasjonaltBakkeSegment / safe_to_netcdf

Python scripts for converting specific Copernicus Sentinel products in Standard Archive Format for Europe (SAFE) to NetCDF/CF
GNU General Public License v3.0
6 stars 5 forks source link

Corrupted jp2 file #21

Closed ElodieFZ closed 1 year ago

ElodieFZ commented 1 year ago

For product S2A_MSIL1C_20220808T140751_N0400_R053_T35XNJ_20220808T174921 , one jp2 file is empty (GRANULE/L1C_T35XNJ_A037230_20220808T140746/QI_DATA/MSK_DETFOO_B03.jp2) so the processing failed.

Traceback (most recent call last): File "rasterio/_base.pyx", line 260, in rasterio._base.DatasetBase.init File "rasterio/_shim.pyx", line 78, in rasterio._shim.open_dataset File "rasterio/_err.pyx", line 215, in rasterio._err.exc_wrap_pointer rasterio._err.CPLE_OpenFailedError: '/lustre/storeB/project/NBS2/sentinel/production/NorwAREA/netCDFNBS_work/production/products_being_processed/S2A_MSIL1C_20220808T140751_N0400_R053_T35XNJ_20220808T174921/S2A_MSIL1C_20220808T140751_N0400_R053_T35XNJ_20220808T174921.SAFE/GRANULE/L1C_T35XNJ_A037230_20220808T140746/QI_DATA/MSK_DETFOO_B03.jp2' not recognized as a supported file format.

We should catch the exception and process the product anyway if it's just mask information that is missing. Best is probably to keep the variable in the nc file but fill it with NaN? So that users are aware that this information is missing.

ElodieFZ commented 1 year ago

After closer inspection, there is nothing wrong with the product. A complete rerun, including downloading from colhub, worked fine. The odd thing is that the md5sum of the SAFE archive was fine and no error appeared in the logs either when downloading or archiving. Maybe some I/0 error on lustre? A similar issue happened at the same time (#22 )