search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

Error: More than one input filename have been found for the File_Type 'GIP_L2WATV' #52

Closed octoflar closed 4 years ago

octoflar commented 4 years ago

I tried to make a test run of start-maja on a few Sentinel-2 tiles for Hamburg. I use Python 2.7 and CentOS 7. When I execute start_maja.py with the command

python ./start_maja.py -f myfolders.txt -g ../gipp/SENTINEL2_NATIF -l ../luts/LUT_MAJA_3_TM_CAMS -t 32UNE -s Hamburg -d 20190714 -e 20190717

the script start fine. I also managed to create a DTM file for Hamburg before the test run. After a few seconds I obtain the following error message:

2020-04-02T09:02:40.829532 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] vns::Data::ERROR: DataApplicationHandler(0x2006a00): DataApplicationHandler::GetGippFilename: More than one input filename have been found for the File_Type 'GIP_L2WATV' and for the satellite 'SENTINEL-2B'.

See details further below. Many thanks and best wishes, Ralf.

The files in ../proc/tmp/maja/Hamburg/gipp/SENTINEL2_NATIF/in are

S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE
S2B_MSIL1C_20190717T104029_N0208_R008_T32UNE_20190717T142616.SAFE
S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR    
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
S2__TEST_AUX_REFDE2_T32UNE_0001.DBL.DIR
S2__TEST_AUX_REFDE2_T32UNE_0001.HDR
S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF

The complete log output is:

2020-04-02 09:02:22,368 - Start-Maja - INFO - repCAMS is missing from configuration file. Needed : repCode, repWork, repL1, repL2, repMaja
2020-04-02 09:02:22,368 - Start-Maja - INFO - Processing without CAMS
2020-04-02 09:02:22,587 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190714.DBL.DIR not found
2020-04-02 09:02:22,587 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190714-*_T32UNE_C_V* not found
2020-04-02 09:02:22,590 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190717.DBL.DIR not found
2020-04-02 09:02:22,590 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190717-*_T32UNE_C_V* not found
2020-04-02 09:02:22,590 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-02 09:02:22,611 - Start-Maja - INFO - => processing date 20190714
2020-04-02 09:02:22,614 - Start-Maja - INFO - dates to process in backward mode :
2020-04-02 09:02:22,614 - Start-Maja - INFO - -- 20190714 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE
2020-04-02 09:02:22,617 - Start-Maja - INFO - -- 20190717 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190717T104029_N0208_R008_T32UNE_20190717T142616.SAFE
2020-04-02 09:02:23,108 - Start-Maja - INFO - #################################
2020-04-02 09:02:23,108 - Start-Maja - INFO - #################################
2020-04-02 09:02:23,108 - Start-Maja - INFO - processing /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE in backward mode
2020-04-02 09:02:23,109 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-02 09:02:23,109 - Start-Maja - INFO - MAJA logfile: /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log
2020-04-02 09:02:23,109 - Start-Maja - INFO - #################################
2020-04-02T09:02:40.829532 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] vns::Data::ERROR: DataApplicationHandler(0x2006a00): DataApplicationHandler::GetGippFilename: More than one input filename have been found for the File_Type 'GIP_L2WATV' and for the satellite 'SENTINEL-2B'.

2020-04-02T09:02:40.833214 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T09:02:40.836307 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x2006a00): DataApplicationHandler::GetGippFilename: More than one input filename have been found for the File_Type 'GIP_L2WATV' and for the satellite 'SENTINEL-2B'.

2020-04-02T09:02:40.839316 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E]   [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T09:02:40.842070 janwevers.uservm.rscloud.vito.be maja-processing-3.3.5 3.3 [000000018205] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]

2020-04-02 09:02:40,879 - Start-Maja - INFO - #######################################
2020-04-02 09:02:40,879 - Start-Maja - INFO - Error detected, see: /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log
2020-04-02 09:02:40,879 - Start-Maja - INFO - #######################################

The file S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log is attached.

S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log

olivierhagolle commented 4 years ago

Thanks, I'll check that. There seems to be an issue with this GIPP package. However, our recommendation is to use the TM version of MAJA and the corresponding GIPP, as it uses the same format as the production of MUSCATE (and we will not maintain that format for long). Olivier

petket-5 commented 4 years ago

Hi,

For a reason that is unknown to me yet, you have two sets of *CONTINEN* models in your execution folder(s). Inside of your gipp/SENTINEL2_NATIF folder (or alternatively in each ../proc/tmp/maja/Hamburg/gipp/SENTINEL2_NATIF/in), can you please delete all CONTINEN_1* files? So at the end you should have only one set of CONTINEN_00001_*HDR/DBL.DIR each.

Kind regards, Peter

octoflar commented 4 years ago

The two sets are indeed only within the temporary directory, but not in the gipp directory. Somehow files like

S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR

are copied into the temporary directory from the LUT directory LUT_MAJA_3_TM_CAMS. Does this make sense? Shall I delete the GIP files from the LUT directory?

In fact, my LUT directory is full of GIPfiles:

S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.HDR
S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.HDR
S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.HDR
S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_DUST_____50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.HDR
S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR
S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR

I fetched the LUT files from https://zenodo.org/record/2636694 as explained in the Start-Maja README file.

petket-5 commented 4 years ago

Well, it does not make much sense to have two sets of the same aerosol model. Can you please show me an ls of each of the LUT and GIPP folders? With that info, we should be able to find out which files to delete and which ones to keep.

octoflar commented 4 years ago

Hi, I included an lsof the LUT folder with my comment above, which I had been editing while you where replying to it. Below is an ls of the GIP folder. All files there a newer than those in the LUT folder.

S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR
S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR
S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF
octoflar commented 4 years ago

The files in the GIP directory were fetched from http://tully.ups-tlse.fr/olivier/gipp_maja.git and copied from the SENTINEL2_NATIF folder. The file set was not complete, however, and I fetched the remaining files from https://zenodo.org/record/3368623/files/20190626_LUT_MAJA_SENTINEL2_NATIF_NOCAMS.zip as explained in the README included with the SENTINEL2_NATIF folder.

petket-5 commented 4 years ago

Ahh I have a suspicion where the error comes from.

First to the solution though: Your GIP folder should not contain any .HDR or .DBL.DIR files. You have to delete all of those. Afterwards, only one set of WATV, TOCR, DIRT and DIFT files for the constant aerosol model (named CONTINEN) should be existing inside the LUT directory

And now to the reason why: Some time ago, we used to have all of the GIP and LUT files inside the same git repository. That must have been when you downloaded the given GIPs for the SENTINEL2_NATIF plugin. Some months back, we split apart the two for volume reasons, which is why we put the LUTs (which are less often subject to change) into a zenodo archive. If you check the same folder in the git repo now, you will see that theres no more HDR and DBL.DIR files: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/SENTINEL2_NATIF

And to finally come back to Oliviers remark: We highly recommend that you switch to the SENTINEL2_TM plugin instead :) With it, you will be able to work on the same format as the tiles already processed by Theia (https://theia.cnes.fr/atdistrib/rocket/#/home) and PEPS on-demand!

Please tell me if the solution proposed worked for you.

Kind regards, Peter

octoflar commented 4 years ago

Thank you very much, this resolved the issue. I'll check if the project does agree to using the TM version :-)