search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO #79

Open meule opened 3 years ago

meule commented 3 years ago

Hey!

I'm processing T12UUB (Canada) and T47QLU (Myanmar) tiles for May-September 2020 and 2019 years with Maja 4.2.0 and I constantly get the error:

2020-11-24 21:37:56 [1655858] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_c37355cbe3907efb5a90a856e4e5d4e5/S2A_MSIL1C_20200616T035541_N0209_R004_T47QLU_20200616T070547.SAFE/GRANULE/L1C_T47QLU_A026027_20200616T040014/../../GRANULE/L1C_T47QLU_A026027_20200616T040014/QI_DATA/MSK_DETFOO_B02.gml !

How to fix that?

olivierhagolle commented 3 years ago

Hi, a few, rare, ill-formed L1C products lack an information needed by MAJA, to accurately compute the viewing angles, and MAJA is not able to process them. All we can recommend, when it happens, is to remove the product from the list of products to be processed and go on with the other ones. Sorry for that, Olivier

meule commented 3 years ago

Thank you Olivier! Is it possible, that all L1C products for a certain tile are invalid? I'm trying to run python3 ./Start_maja.py -y --overwrite --type_dem srtm --nbackward 16 -f /home/ubuntu/d/maja_folders.txt -t 12UUB -s MM -d 2020-05-01 -e 2020-10-31 and python3 ./Start_maja.py -y --overwrite --type_dem srtm --nbackward 16 -f /home/ubuntu/d/maja_folders.txt -t 12UUB -s MM -d 2019-05-01 -e 2019-10-31 And I get error messages for all dates, like:

2020-11-24 17:31:57,686 [INFO ] Beginning workplan execution.                                                                                                                                      2020-11-24 17:31:57,686 [INFO ] Executing workplan #1/75
2020-11-24 17:31:59 [1074840] PROGRESS MAJA 4.2 initialize:476 Starting MAJA_L2_BACKWARD_CHAIN                                                                                                     2020-11-24 17:34:32 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2A_MSIL1C_20200607T182921_N0209_R027_T12UUB_20200607T2206
33.SAFE/GRANULE/L1C_T12UUB_A025907_20200607T183828/../../GRANULE/L1C_T12UUB_A025907_20200607T183828/QI_DATA/MSK_DETFOO_B02.gml !
...
2020-11-24 17:35:29 [1074840] PROGRESS MAJA 4.2 process_one_product:854 System infos (RAM;DISK;HOUR;MIN;SEC;LOAD) : 6326;10883;0;3;29;63                                                           2020-11-24 17:36:10 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a
constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2B_MSIL1C_20200605T183919_N0209_R070_T12UUB_20200605T221123.SAFE/GRANULE/L1C_T12UUB_A016970_20200605T184217/../../GRANULE/L1C_T12UUB_A016970_20200605T184217/QI_DATA/MSK_DETFOO_B03.gml !
...
2020-11-24 17:37:45 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2B_MSIL1C_20200602T182919_N0209_R027_T12UUB_20200602T220412.SAFE/GRANULE/L1C_T12UUB_A016927_20200602T183556/../../GRANULE/L1C_T12UUB_A016927_20200602T183556/QI_DATA/MSK_DETFOO_B03.gml !
...
2020-11-24 17:39:34 [1074840] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a
constant shift with the feature indices in the MSK_DETFOO /home/ubuntu/d/MAJA/StartMaja/work/Start_maja_26a40a4e3195c2d27717d1e9eed465df/S2A_MSIL1C_20200531T183921_N0209_R070_T12UUB_20200531T221758.SAFE/GRANULE/L1C_T12UUB_A025807_20200531T184459/../../GRANULE/L1C_T12UUB_A025807_20200531T184459/QI_DATA/MSK_DETFOO_B03.gml !
...

and so on for all the dates between May and October. And the same for tile 47QLU

olivierhagolle commented 3 years ago

Glups, I never met such a case, up to now, we only met this rarely. We will have to do something about it Sorry for that Olivier

c00kiemon5ter commented 3 years ago

Hello, I'm trying out MAJA and seeing this, too. The following is the beginning of the process I run with the same error:

The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO

I am using MAJA v4.2.1.

2020-12-03 19:39:33,410 [INFO ] =============This is Start_Maja v4.2.0==============
2020-12-03 19:39:33,415 [WARNI] repCAMS is missing. Processing without CAMS
2020-12-03 19:39:33,418 [INFO ] Detecting input products...
2020-12-03 19:39:33,427 [INFO ] 16 L1C product(s) detected for tile 34SEH in /vagrant/34SEH/L1C/34SEH
2020-12-03 19:39:33,427 [WARNI] No L2A products detected for tile 34SEH in /vagrant/34SEH/L2A/34SEH
2020-12-03 19:39:33,427 [INFO ] Skipping CAMS file detection.
2020-12-03 19:39:33,427 [INFO ] Checking GIPP files
2020-12-03 19:39:33,428 [INFO ] Setting up GIPP folder: /vagrant/34SEH/foo/gipp
2020-12-03 19:39:33,445 [INFO ] Searching for DTM
2020-12-03 19:39:33,448 [INFO ] Found DTM: /vagrant/34SEH/foo/dtm/S2__TEST_AUX_REFDE2_34SEH_1001.HDR
2020-12-03 19:39:33,465 [INFO ] GIPP Creation succeeded for SENTINEL2_NATIF
2020-12-03 19:39:33,467 [INFO ] 13 workplan(s) successfully created:
               DATE |       TILE |     MODE |                                                             L1-PRODUCT | ADDITIONAL INFO
2019-02-08 09:21:41 |      34SEH | BACKWARD |      S2A_MSIL1C_20190208T092141_N0207_R093_T34SEH_20190208T112117.SAFE | Backward of 8 products
2019-02-18 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190218T092031_N0207_R093_T34SEH_20190218T095107.SAFE | L2 from previous
2019-02-28 09:20:21 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190228T092021_N0207_R093_T34SEH_20190228T105030.SAFE | L2 from previous
2019-03-05 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190305T092029_N0207_R093_T34SEH_20190305T132100.SAFE | L2 from previous
2019-03-10 09:20:21 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190310T092021_N0207_R093_T34SEH_20190310T112345.SAFE | L2 from previous
2019-03-15 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190315T092029_N0207_R093_T34SEH_20190315T120923.SAFE | L2 from previous
2019-03-25 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190325T092029_N0207_R093_T34SEH_20190325T121158.SAFE | L2 from previous
2019-03-30 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE | L2 from previous
2019-04-04 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190404T092029_N0207_R093_T34SEH_20190404T112331.SAFE | L2 from previous
2019-04-04 09:20:29 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190404T092029_N0207_R093_T34SEH_20190404T114927.SAFE | L2 from previous
2019-04-14 09:20:39 |      34SEH |  NOMINAL |      S2B_MSIL1C_20190414T092039_N0207_R093_T34SEH_20190414T112240.SAFE | L2 from previous
2019-04-19 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190419T092031_N0207_R093_T34SEH_20190419T112226.SAFE | L2 from previous
2019-04-29 09:20:31 |      34SEH |  NOMINAL |      S2A_MSIL1C_20190429T092031_N0207_R093_T34SEH_20190429T115143.SAFE | L2 from previous
Press Enter to continue...

2020-12-03 19:40:01,109 [INFO ] Beginning workplan execution.
2020-12-03 19:40:01,109 [INFO ] Executing workplan #1/13
2020-12-03 19:40:02 [19612] PROGRESS MAJA 4.2 initialize:476 Starting MAJA_L2_BACKWARD_CHAIN
2020-12-03 19:42:09 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B02.gml !
2020-12-03 19:42:21 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B03.gml !
2020-12-03 19:42:32 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B04.gml !
2020-12-03 19:42:43 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B08.gml !
2020-12-03 19:42:57 [19612] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /vagrant/34SEH/foo/work/Start_maja_03d07c9e683466a4adaaeaa939025e4d/S2A_MSIL1C_20190330T092031_N0207_R093_T34SEH_20190330T105736.SAFE/GRANULE/L1C_T34SEH_A019681_20190330T092028/../../GRANULE/L1C_T34SEH_A019681_20190330T092028/QI_DATA/MSK_DETFOO_B05.gml !
...
jprankl commented 3 years ago

Hi, same for me. With MAJA 4.2 I get the index_shift error, which I did not get with MAJA 3.x

2020-12-04 11:14:43 [20938] WARNING MAJA 4.2 get_detfoo_index_shift:925 Sentinel2L1ImageFileReaderBase::get_detfoo_index_shift: The detector indices are not in ascending order or do not have a constant shift with the feature indices in the MSK_DETFOO /data_nvme/tmp_work-test/Start_maja_eaef298c96a5c2596bf70447af4ac908/S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.SAFE/GRANULE/L1C_T33UWP_A023671_20200103T095402/../../GRANULE/L1C_T33UWP_A023671_20200103T095402/QI_DATA/MSK_DETFOO_B05.gml !

Products look somehow correct, (at least the cloud masks), but it seems that shadow masks are missing. Bests, Johann

jerome-colin commented 3 years ago

Hi Konstantin, I tried to process T12UUB starting from May 2020 and indeed got some products skipped with 'DETFOO'-related 'WARNING', but strictly speaking, it doesn't end in error state, just moves up to the next product. At the time of writing, I got L2A results for May 1st, 16th, 26th and 28th, and it keeps running on June. Does it breaks in error state for you ? Thanks, Jerome

jprankl commented 3 years ago

Sorry, in my post before I wrote "index_shift error", but Jerome you are right, in my case, it is also a warning and it produces a product.
Bests,
Hannes

olivierhagolle commented 3 years ago

Hi @Jprankl, about your note on undetected cloud shadows, this can happen if the multi-temporal methods could not be used. For instance, if the initialization with backward mode had only one image, or at least one usable image (because of the index shift error or because of a very large cloud cover). If you tell us more about the case where you observe that, we might help. Olivier

jprankl commented 3 years ago

Hi Oliver, thanks, that might be the case. It was initialization in a cloudy season. Actually, the first product ended with an error (which I mentioned in #78), and for the next product (SENTINEL2A_20200103-095716-566_L2A_T33UWP_C_V1-0), MAJA computed an L2A product, but without a cloud mask and with get_detfoo_index_shift warnings. S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.log Thanks for your explanation! Bests! Johann