search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

ERROR: No GIPP of type GIP_CKEXTL has been detected for the Mission <SENTINEL-2A> #24

Closed DariaLudtke closed 4 years ago

DariaLudtke commented 5 years ago

I am trying to run MAJA 3.3.0 without CAMs data for the test region around Avignon, but am blocked by the following error:

vns::Data::ERROR: DataApplicationHandler(0xa235d0): No GIPP of type GIP_CKEXTL has been detected for the Mission in the input directory! [vnsDataApplicationHandler.cxx:1501] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:131]

I downloaded the additional look-up tables from https://zenodo.org/record/2636694 though CAMs should be deactivated, but either way I get blocked by the error. Any help on how to proceed would be great, thanks!

olivierhagolle commented 5 years ago

Hi Daria, I am lacking information. Please send the command line, folders.txt and full log. Did you get the GIPP directory from gitlab, which one ? Thanks, Olivier

DariaLudtke commented 5 years ago

Thanks for your quick reply, Olivier.

The command line used in the Start-MAJA folder is ./start_maja.py -f folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t T31TFJ -s avignon -d 20190314 -e 20190326

The GIPP directory used is GIPP_S2_MAJA_3.3_TM (https://github.com/CNES/Start-MAJA/files/3349574/S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.txt) downloaded from http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_S2_MAJA_3.3_TM

Here is the folders.txt and the log file. folders.txt S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log

Thanks, Daria

olivierhagolle commented 5 years ago

Thanks Daria, there was an error in 4 GIPP files which must contain <Mission>SENTINEL-2A</Mission> or <Mission>SENTINEL-2B</Mission> but contained <Mission>SENTINEL2A</Mission> or <Mission>SENTINEL2B</Mission>

S2A_TEST_GIP_CKEXTL_S_31TJF10001_20150703_21000101.EEF S2A_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF S2B_TEST_GIP_CKEXTL_S_31TJF10001_20150703_21000101.EEF S2B_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF

While the other GIPP files must not contain the dashes. I hope we will be able to remove this unneeded complexity in MAJA 4.0.

You can download again the GIPP folder, I have updated it. Sorry for this error. Olivier

DariaLudtke commented 5 years ago

Hello Olivier, I totally missed that error indeed. When downloading the altered GIPP files from http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master , I see that you updated GIPP_S2_MAJA_3_TM for MAJA 3.1 standalone version. I am currently using MAJA 3.3 and manually added the dashes in the four GIPP files in my Maja-3.3.0-noTM, but this unfortunately resulted in another error:

"ERROR - First backward processing was unsuccessful, check MAJA installation"

Interestingly, my .log file ends with: "Process OK ;-) [code return: 0]" and I have a variety of .TIF, .HDR and a .JPEG as outputs.

The .log file is the following: S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log

Could you give me any insights about this? Thank you, Daria

olivierhagolle commented 5 years ago

Hi, I am not sure, but the parameters you provided : "-d 20190314 -e 20190326" indicate a very short period of time. MAJA is a multi-temporal processor, and needs to process time series. It is possible that in that period there is no cloud free pixel. Could you please try to download more dates and process a longer time period ? Usually we use a minimum period of one month. Olivier

DariaLudtke commented 5 years ago

Hello Olivier,

I re-ran MAJA 3.3 noTM without CAMS with a month of input data and got the same result: Again, the .log file gives me "Process OK ;-) [code return: 0]" as the last line while the terminal returns the error of "ERROR - First backward processing was unsuccessful, check MAJA installation".

As far as the outputs are concerned, they are (apart of the folder structure) consistent with the expected native format published here: http://www.cesbio.ups-tlse.fr/multitemp/?page_id=10464#English .

(To be specific what I mean when I talk about the difference in folder structure is that the masks (QLT, CLD, MSK) are provided in the same directory as FRE, SRE, and ATB; while the MSK_DETFOO folder contains one .gml file per band and there's an additional PRIVATE folder whose use I don't really understand.)

Do you think the outputs are correct despite the error in the terminal? I loaded the TIFFs in a GIS and they all have the correct spatial resolution, spatial extent, and the right number of bands.

Thanks, Daria

olivierhagolle commented 5 years ago

Hi Daria,

I am sorry for the long process, I have too many meetings, but it's good to comeback to coding at the end of the week.

Yes the log you sent clearly shows that MAJA works fine, and the L2A product you have is fine. But start_maja.py is supposed to process the whole time series for you, and it seems it stops after the first production in backward mode, because it does not detect the L2A product. The problem is that I cannot reproduce the error, it works on my side.

Did you find the l2a product in this folder ? /home/darialuedtke/MAJA/data/sentinel-2/l2a_maja Could you also give me the log which is printed on your terminal ?

I am also running a new tests, result in 30 minutes... Olivier

olivierhagolle commented 5 years ago

Oups, backward is two hours, result on Monday... Olivier

olivierhagolle commented 4 years ago

My test was unsuccessful in reproducing what you observe. Please send the information requested above. Olivier

DariaLudtke commented 4 years ago

Hello Olivier,

Sorry for the late reply. I am on my vacation until Friday and didn't bring my computer. I will provide you what you need first thing Friday morning and ask my colleague who had the same observation if he can share with you today.

Best, Daria

olivierhagolle commented 4 years ago

All right, enjoy your holidays !

I will be on holidays myself at the end of the week, but I'll see if someone else in the team can help you. Olivier

alvarezWegaw commented 4 years ago

Hello Olivier, I am Gonzalo, Daria's colleague. I have reproduced the error that Daria had, with the same setup.

We have installed Maja-3.3.0-noTM and executed "./start_maja.py -f folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t T31TFJ -s avignon -d 20190314 -e 20190326" (I have excuted maja with only 3 different L1C files for the sake of speed, seeing that the error I have is the same as Daria's with a full month of data).

The GIPP files and LUTs are the following (downloaded from Cesbio's repositories and after applying the corrections to the mission SENTINEL-2A). GIPP downloaded from: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_S2_MAJA_3.3_TM LUTs downloaded from: https://zenodo.org/record/2636694

My folders.txt file is the following: folders.txt

The output log file from the latest execution is: S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log

The result on my L2 folder (/mnt/data/sentinel2/L2) is the following: . └── Avignon └── 31TFJ └── GIPP_S2_MAJA_3.3_TM ├── S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log ├── S2A_OPER_PMC_L2REPT_31TFJ20190314.EEF ├── S2A_OPER_SSC_L2VALD_31TFJ20190314.DBL ├── S2A_OPER_SSC_L2VALD_31TFJ20190314.DBL.DIR │   ├── MSK_DETFOO │   │   ├── MSK_DETFOO_B01.gml │   │   ├── MSK_DETFOO_B02.gml │   │   ├── MSK_DETFOO_B03.gml │   │   ├── MSK_DETFOO_B04.gml │   │   ├── MSK_DETFOO_B05.gml │   │   ├── MSK_DETFOO_B06.gml │   │   ├── MSK_DETFOO_B07.gml │   │   ├── MSK_DETFOO_B08.gml │   │   ├── MSK_DETFOO_B09.gml │   │   ├── MSK_DETFOO_B10.gml │   │   ├── MSK_DETFOO_B11.gml │   │   ├── MSK_DETFOO_B12.gml │   │   └── MSK_DETFOO_B8A.gml │   ├── PRIVATE │   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ20190314_LTC.DBL.DIR │   │   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ20190314_LTC.DBL.mha │   │   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ20190321_LTC.DBL.mha │   │   │   └── S2B_OPER_SSC_LUTANX_L2VALD_31TFJ20190326_LTC.DBL.mha │   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ20190314_LTC.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_CLD.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_CLD.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_NDT.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_NDT.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_PXD.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_PXD.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_RCR.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_RCR.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_RTA.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_RTA.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_RTC.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_RTC.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_STO.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_STO.HDR │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_WAM.DBL.TIF │   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_WAM.HDR │   │   └── S2A_OPER_TEC_L2VALD_31TFJ20190314.EEF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_ATB_R1.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_ATB_R1.HDR │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_ATB_R2.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_ATB_R2.HDR │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_CLD_R1.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_CLD_R1.HDR │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_CLD_R2.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_CLD_R2.HDR │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_MSK_R1.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_MSK_R1.HDR │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_MSK_R2.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_MSK_R2.HDR │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_QLT_R1.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_QLT_R1.HDR │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ20190314_QLT_R2.DBL.TIF │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_QLT_R2.HDR │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ20190314_FRE_R1.DBL.TIF │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ20190314_FRE_R1.HDR │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_FRE_R2.DBL.TIF │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ20190314_FRE_R2.HDR │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ20190314_SRE_R1.DBL.TIF │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_SRE_R1.HDR │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ20190314_SRE_R2.DBL.TIF │   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ20190314_SRE_R2.HDR │   ├── S2A_OPER_SSC_PDTQLK_L2VALD_31TFJ____20190314.DBL.JPG │   └── S2A_OPER_SSC_PDTQLK_L2VALD_31TFJ20190314.HDR └── S2A_OPER_SSC_L2VALD_31TFJ____20190314.HDR (Here in image format, it may be easyer to read: output )

The full log from terminal is the following: root@610433098ef8:/Start-MAJA# ./start_maja.py -f folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t 31TFJ -s Avignon -d 20190314 -e 20190326

2019-07-08 11:03:07,396 - Start-Maja - INFO - No existing L2 product, we start with backward mode 2019-07-08 11:03:07,396 - Start-Maja - INFO - => processing date 20190314 2019-07-08 11:03:07,399 - Start-Maja - INFO - dates to process in backward mode : 2019-07-08 11:03:07,400 - Start-Maja - INFO - -- 20190314 : /mnt/data/sentinel2/L1/Avignon/S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE 2019-07-08 11:03:07,400 - Start-Maja - INFO - -- 20190321 : /mnt/data/sentinel2/L1/Avignon/S2A_MSIL1C_20190321T103021_N0207_R108_T31TFJ_20190321T173032.SAFE 2019-07-08 11:03:07,400 - Start-Maja - INFO - -- 20190326 : /mnt/data/sentinel2/L1/Avignon/S2B_MSIL1C_20190326T103029_N0207_R108_T31TFJ_20190326T155314.SAFE 2019-07-08 11:03:07,415 - Start-Maja - INFO - ################################# 2019-07-08 11:03:07,415 - Start-Maja - INFO - ################################# 2019-07-08 11:03:07,415 - Start-Maja - INFO - processing /mnt/data/sentinel2/L1/Avignon/S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE in backward mode 2019-07-08 11:03:07,415 - Start-Maja - INFO - Initialisation mode with backward is longer 2019-07-08 11:03:07,415 - Start-Maja - INFO - MAJA logfile: /mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log 2019-07-08 11:03:07,416 - Start-Maja - INFO - ################################# 2019-07-08 12:03:48,299 - Start-Maja - INFO - => processing date 20190321 2019-07-08 12:03:48,320 - Start-Maja - INFO - MAJA command failed : /maja/bin/maja -i /mnt/data/sentinel2/work/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//in -o /mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM/ -m L2BACKWARD -ucs /mnt/data/sentinel2/work/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//userconf --TileId 31TFJ >/mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log 2>&1 2019-07-08 12:03:48,321 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation

I hope this information is all you needed and it is clear enough, but feel free to ask for any other test or complementary information you may need.

Thanks in advance, Gonzalo

olivierhagolle commented 4 years ago

Hi, thanks for providing more information ! I still do not understand the cause of failure. Everything seems fine in the log file, the L2 is generated, but Start_Maja does not find the L2A product to produce the next one. I have made a few modifications within start_maja.py to print in the console log the L2A file name MAJA is searching for. If you could update your start_maja.py version and run again the command line.

The best would be to erase this directory before running again: /mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM

Thanks, Olivier

alvarezWegaw commented 4 years ago

Hi! I have not had the time to test with the new version of Start-MAJA. But I have tested installing and using Start-MAJA with Maja-3.3.0-TM and this time the process is able to finish and it creates all the L2 files

I will retest with MAJA-noTM and the new version of Start-MAJA as soon as possible and let you know the results. Thanks, Gonzalo

olivierhagolle commented 4 years ago

Thanks for the good news. I had not noticed you were using the no-TM version (although Daria had written it). The problem is that you are using TM GIPP and LUTS with the No-TM version of MAJA.

My advice is too use the TM plugin. The no-TM is only provided for compatibility for previous users of MAJA. It will be probably abandoned at the end of 2019.

Olivier

DariaLudtke commented 4 years ago

Hello Olivier,

I hope you are enjoying your vacation. Gonzalo and I have had a chat to update me what happened during my absence. MAJA is still causing several issues for us.

One of them is the processing time and the space required. I tried to run it on one month's worth of data (9 images) and we provided around 30GB of disc space, which wasn't sufficient. It produced an error after over 1 hour running time due to lack of space. Could you give us an estimation how much disk space and time is needed for processing one tile? Alternatively, is there a way to reduce the disk space needed for the processing?

Secondly, since we want to investigate a larger region, we aim at reusing the flat surface reflectances so we don't have to go through the entire multi-temporal processing chain every time a new image for a tile comes in. Could you please give us an idea how we can reuse it and how it will affect the processing parameters (disk space/ time)?

Thanks for your help! Daria

RubenValDin commented 4 years ago

Hello everyone,

I have got the same error (No GIPP of type GIP_CKEXTL has been detected for the Mission ). However, in my case I am running MAJA-3.2.2_TM.run using GIPP_S2_MAJA_3_TM.

Olivier, you have commented that is already fixed and when I open the file I can see the change so I don't understand what is happening.

The .log file is the following: S2B_MSIL1C_20180803T112109_N0206_R037_T29TPH_20180803T152545.SAFE.log

Thank you very much Ruben

alvarezWegaw commented 4 years ago

Hello Ruben, We have found out after testing the noTM and TM versions that there is a discrepancy between what they expect in the <Mission> field on the GIP_CKEXTL files.

For the TM version, Start-MAJA expect to have the<Mission> parameter without a dash. In order to correct your problem, you can modify the following files: S2A_TEST_GIP_CKEXTL_S_31TJF10001_20150703_21000101.EEF S2A_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF S2B_TEST_GIP_CKEXTL_S_31TJF10001_20150703_21000101.EEF S2B_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF

You will have to change the parameter so it shows <Mission>SENTINEL2A</Mission> or <Mission>SENTINEL2B</Mission>. With this changes, MAJA should work.

Olivier, I don't think you where aware of this problem, we just found it out.

I hope this helps!

jerome-colin commented 4 years ago

Dear Ruben, Unless you have very specific reason to use version 3.2.2, I would recommend you to download the latest 3.3.0[TM] version and re-try. You may well keep version 3.2.2 if needed and still have 3.3.0 installed beside, provided that you point to the proper version of the executable in your 'folder.txt' file.

Please let us know if you succeed, Best, Jerome

RubenValDin commented 4 years ago

Hello both,

@alvarezWegaw Thanks. I have changed the parameter and it seems to be working now. @jerome-colin I will test the new version 3.3.0 in coming weeks, for now I will just change the parameter.

Thanks Ruben

olivierhagolle commented 4 years ago

I like that when users help each other (thank you @alvarezWegaw !) Sorry for the mistake, this error in the parameters keeps coming back although I have chased it. Thanks for telling us, I will try to correct it. Olivier