search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
141 stars 25 forks source link

Error running start_maja.py / Doubts about folder-file structure #51

Closed intiluna closed 4 years ago

intiluna commented 4 years ago

Platform and versions: • Ubuntu 18.04.4 LTS • Python 3.7.6 (default, Jan 8 2020, 19:59:22) [GCC 7.3.0] :: Anaconda, Inc. on linux • GDAL 2.4.2, released 2019/06/28

Dear Maja Users,

I have some problems with the folder and files structure required to properly run Maja. To start, I want to first process without CAMS. The steps are described below:

1-Downloaded and installed Maja_3.3.2_TM.run

bash MAJA-3.2.2_TM.run –target ~/maja

2.Clone the current repository to get start_maja.py

git clone https://github.com/CNES/Start-MAJA.git

3. Prepare folders.txt

repCode= ~/maja/Start-MAJA repWork= /media/ernesto/cuatrot/maja_procesos/temporal repL1 = /media/ernesto/cuatrot/maja_procesos/S2/L1C/L1CCH repL2 = /media/ernesto/cuatrot/maja_procesos/S2/L2A/L2ACH repMaja= ~/maja/bin/maja repCAMS= /media/ernesto/cuatrot/maja_procesos/cams

4. Retrieve Sentinel-2 L1C data Using peps_download.py, I have downloaded 188 images for 4 tiles (PDU,PDV,PEU,PEV) for the period from 2019-08-01 to 2020-03-20 . An example of the commands used to download images for one tile is presented:

python /media/ernesto/cuatrot/peps_download/peps_download-master/peps_download.py -c S2ST -t 16PDU -o 140 -a /media/ernesto/cuatrot/peps_download/peps_download-master/peps.txt -d 2019-08-01 -f 2020-03-20

Unzip L1C data path=media/ernesto/cuatrot/maja_procesos/S2/L1C/L1CCH

5. Parameters

path=~/maja/Start-MAJA/userconf

The files in this folder are: MAJAUserConfig_FORMOSAT_MUSCATE_PROTO.xml MAJAUserConfig_LANDSAT8_MUSCATE_PROTO.xml MAJAUserConfig_LANDSAT8_MUSCATE.xml MAJAUserConfig_LANDSAT8.xml MAJAUserConfig_LANDSAT_MUSCATE_PROTO.xml MAJAUserConfig_LANDSAT_MUSCATE.xml MAJAUserConfig_SENTINEL2_GPP.xml MAJAUserConfig_SENTINEL2_MUSCATE.xml MAJAUserConfig_SENTINEL2_TM.xml MAJAUserConfig_SENTINEL2.xml MAJAUserConfig_SPOT4_MUSCATE_PROTO.xml MAJAUserConfigSystem.xml MAJAUserConfig_VENUS.xml

path=~/maja/Start-MAJA/GIPP/SENTINEL2_TM The files in this folder are: README.md S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF S2A_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF S2A_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF S2A_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF

I have a doubt regarding the use of GIPP without CAMS. According to the instructions, if we want to use Parameters without CAMS, we should use the folder SENTINEL2_TM_CAMS:

However, when we read the README.md for http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/SENTINEL2_TM_CAMS. It clearly states that CAMS option enabled. And the README.md for: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/SENTINEL2_TM, we get the message that CAMS option disabled.

So, I am confused. I assumed that the README files are correct and I should use SENTINEL2_TM when processing without CAMS.

I downloaded the LUT from the zenodo server and I choose to use: 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS path=~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs

Files in this folder: S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR

6. DTM preparation After some problems with the code to prepare the dtm. It was suggested to use another branch of the DTM preparation code (https://github.com/CNES/Start-MAJA/tree/reprog-rc1/prepare_mnt) and using this updated code it works fine.

path=~/maja/Start-MAJA/DTM

Since I use 4 tiles I produced 4 DTMs. This folder contains 2 files per tile: S2TEST_AUX_REFDE2_16PDU_0001.DBL.DIR S2TEST_AUX_REFDE2_16PEU_0001.DBL.DIR S2TEST_AUX_REFDE2_16PDU_0001.HDR S2TEST_AUX_REFDE2_16PEU_0001.HDR S2TEST_AUX_REFDE2_16PDV_0001.DBL.DIR S2TEST_AUX_REFDE2_16PEV_0001.DBL.DIR S2TEST_AUX_REFDE2_16PDV_0001.HDR S2TEST_AUX_REFDE2_16PEV_0001.HDR

7. CAMS data

As stated in the beginning currently I want to use MAJA without CAMS but I would like to be able to process with CAMS in the future.

Following instructions from https://github.com/CNES/Start-MAJA/tree/master/cams_download At this moment I have registered at ECMWF and I have API key.

I do not know what it means with: Create the file '.ecmwfapirc' in your home and type the API key. How you do that. Is it a text in which I hould put the API key? And referring to home folder you mean Home (~) or it means the “cams_download” folder in the Statr-Maja folder? path= ~/maja/Start-MAJA/cams_download

8. Execute start_maja.py

In the start-maja folder (~/maja/Start-MAJA), I open terminal and run:

python start_maja.py -f ~/maja/Start-MAJA/folders.txt -g ~/maja/Start-MAJA/GIPP/SENTINEL2_TM -l ~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS -t 16PDU -s ser -d 20190801 -e 202000320

Running the above commands produce the following errors: image

What am I missing or doing wrong? Many thanks for your help. Kind regards,

olivierhagolle commented 4 years ago

Hi, this one is easy, start_maja is meant for python 2.7, and what you observe is typically an error when running it witth python3. Come on, you're almost there ! Olivier

intiluna commented 4 years ago

Hi Olivier, Thanks a lot. I want to get there soon :)

I installed a python 2.7 environment with conda:

conda create --name py2 python=2.7 conda activate py2

and the run the code again:

python start_maja.py -f ~/maja/Start-MAJA/folders.txt -g ~/maja/Start-MAJA/GIPP/SENTINEL2_TM -l ~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS -t 16PDU -s ser -d 20190801 -e 202000320

I get the following error of missing folders/files

py2_first_try

However I do have the files. Here is the tree of folder/files in html for easy visualization: https://www.dropbox.com/s/t5glnhrht7zcm6f/tree_folders.html?dl=0 image

Thanks again!!!! :D

olivierhagolle commented 4 years ago

Hi Inti, the message means that some of the folders you defined in folders.txt are not- existing. Could you please check them ? repCode = ~/maja/bin/Start-MAJA repMaja = ~/maja/bin/maja

One possibility is that python does not recognise "~", maybe try to replace by the whole path.

(it's now time for me to disconnect, but I can try to answer new questions tomorrow) Olivier

intiluna commented 4 years ago

Dear Olivier,

I managed to get MAJA script finding correctly the paths and files. It took some tries to realize that when running the start-maja.py we should think of a additive path (folders.txt path + additional path for GIPP, LUT and site name)

Full path to LUT: "/home/ernesto/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs" Full path to GIP:"/home/ernesto/maja/Start-MAJA/GIPP/SENTINEL2_TM" Full path to L1C data: "/media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA"

In my case, the folders.txt is like this now:

repCode= /home/ernesto/maja/Start-MAJA repWork= /media/ernesto/cuatrot/maja_procesos/temporal repL1 = /media/ernesto/cuatrot/maja_procesos/S2/L1C repL2 = /media/ernesto/cuatrot/maja_procesos/S2/L2A repMaja= /home/ernesto/maja/bin/maja repCAMS= /media/ernesto/cuatrot/maja_procesos/cams

So, when running the script I just have add the rest of the path like this: python start_maja.py -f folders.txt -g GIPP/SENTINEL2_TM -l 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs -t 16PDU -s CHINANDEGA -d 20190801 -e 202000320

Later I run it but I get some others errors: ` ernesto@ernestomacpro:~/maja/Start-MAJA$ python start_maja.py -f folders.txt -g GIPP/SENTINEL2_TM -l 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs -t 16PDU -s CHINANDEGA -d 20190801 -e 202000320

2020-04-02 11:48:46,254 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDUT20190805.DBL.DIR not found 2020-04-02 11:48:46,254 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190805-_T16PDU_C_V not found 2020-04-02 11:48:46,255 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDUT20190810.DBL.DIR not found 2020-04-02 11:48:46,255 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190810-_T16PDU_C_V not found 2020-04-02 11:48:46,258 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20190815.DBL.DIR not found 2020-04-02 11:48:46,258 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190815-_T16PDU_C_V not found ....cut some rows just like the above part but for another dates....

2020-04-02 11:48:46,319 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDUT20191223.DBL.DIR not found 2020-04-02 11:48:46,319 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20191223-_T16PDU_C_V not found 2020-04-02 11:48:46,321 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDUT20191228.DBL.DIR not found 2020-04-02 11:48:46,321 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20191228-_T16PDU_C_V not found 2020-04-02 11:48:46,321 - Start-Maja - INFO - No existing L2 product, we start with backward mode 2020-04-02 11:48:46,322 - Start-Maja - INFO - => processing date 20190805 2020-04-02 11:48:46,337 - Start-Maja - INFO - dates to process in backward mode : 2020-04-02 11:48:46,337 - Start-Maja - INFO - -- 20190805 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE 2020-04-02 11:48:46,337 - Start-Maja - INFO - -- 20190810 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190810T160911_N0208_R140_T16PDU_20190810T212513.SAFE 2020-04-02 11:48:46,338 - Start-Maja - INFO - -- 20190815 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190815T160829_N0208_R140_T16PDU_20190815T211555.SAFE 2020-04-02 11:48:46,338 - Start-Maja - INFO - -- 20190820 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190820T160901_N0208_R140_T16PDU_20190820T212247.SAFE 2020-04-02 11:48:46,339 - Start-Maja - INFO - -- 20190825 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190825T160829_N0208_R140_T16PDU_20190825T193312.SAFE 2020-04-02 11:48:46,339 - Start-Maja - INFO - -- 20190830 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190830T160901_N0208_R140_T16PDU_20190830T212354.SAFE 2020-04-02 11:48:46,340 - Start-Maja - INFO - -- 20190904 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190904T160829_N0208_R140_T16PDU_20190904T193256.SAFE 2020-04-02 11:48:46,341 - Start-Maja - INFO - -- 20190909 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190909T160901_N0208_R140_T16PDU_20190909T212411.SAFE 2020-04-02 11:48:46,362 - Start-Maja - INFO - ################################# 2020-04-02 11:48:46,362 - Start-Maja - INFO - ################################# 2020-04-02 11:48:46,362 - Start-Maja - INFO - processing /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE in backward mode 2020-04-02 11:48:46,362 - Start-Maja - INFO - Initialisation mode with backward is longer 2020-04-02 11:48:46,362 - Start-Maja - INFO - MAJA logfile: /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE.log 2020-04-02 11:48:46,362 - Start-Maja - INFO - ################################# 2020-04-02T11:48:47.644243 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] vns::Data::ERROR: DataApplicationHandler(0x1a655d0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:GetListOfGippFilenames:1699]

2020-04-02T11:48:47.645796 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] ** [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.647014 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x1a655d0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:1699] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.648199 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.649346 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] ** [vnsMajaMainProcessor.cxx:main:129]

2020-04-02 11:48:47,666 - Start-Maja - INFO - ####################################### 2020-04-02 11:48:47,666 - Start-Maja - INFO - Error detected, see: /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE.log 2020-04-02 11:48:47,666 - Start-Maja - INFO - ####################################### (py2) ernesto@ernestomacpro:~/maja/Start-MAJA$ ` What is this file type (S2?_OPER_SSC_L2VALD_16PDU____T20190805.DBL.DIR not found)missing?

In my GIP folder I have: image

and in my LUT folder: image

Thank you for your help, Inti

olivierhagolle commented 4 years ago

Hi Inti, The message "In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! " means that the DTM was not found.

It has to be stored in the DTM folder, where you launch start_maja from. Olivier

intiluna commented 4 years ago

Hi Olivier, It is strange because I do have a DTM folder inside star-maja folder in which I run the script as it can be seen here:

startmajafolder

and inside the DTM folder I have the corresponding DTM files (2 files per tile) created with the preparation tools:

image

As you can see there are the files required for this tile: S2TEST_AUX_REFDE2_16PDU_0001.DBL.DIR
S2TEST_AUX_REFDE2_16PDU_0001.HDR

What else could it be? Thank you!

intiluna commented 4 years ago

Hi Olivier, I have to say that the DTM files were created using the dtm preparation tools suggested by Peter (petket-5) in the previous post:https://github.com/CNES/Start-MAJA/issues/50 that are located in https://github.com/CNES/Start-MAJA/tree/reprog-rc1/prepare_mnt.

Afterwards, I just moved the created dtm files to the DTM folder inside original start-maja folder (coming from https://github.com/CNES/Start-MAJA). I hope this information can help find the error cause.

Thanks,

olivierhagolle commented 4 years ago

Hi inti, I checked, and (sorry to have forgotten this part), the DBL.DIR and HDR files are not stored in DTM folder. There is another folder in between:

[hagolle@visu02 Start_Maja]$ ls -1 DTM
S2__TEST_AUX_REFDE2_T11SPC_0001 
S2__TEST_AUX_REFDE2_T29TPH_0001  
S2__TEST_AUX_REFDE2_T31TFJ_0002  
S2__TEST_AUX_REFDE2_T32TNS_0001  
S2__TEST_AUX_REFDE2_T35JPM_0001
[hagolle@visu02 Start_Maja]$ ls DTM/S2__TEST_AUX_REFDE2_T11SPC_0001
S2__TEST_AUX_REFDE2_T11SPC_0001.DBL.DIR  S2__TEST_AUX_REFDE2_T11SPC_0001.HDR

I hope it's the last one ! Olivier

intiluna commented 4 years ago

Hi Olivier, Thanks, that last hint of the folder in between the DTM and the DTM files (.DBL.DIR .HDR) was what was missing and now it worked fine and I was able to process images for 5 months of one tile since yesterday.

Thanks a lot, Inti

olivierhagolle commented 4 years ago

I told you you were close to succeed, but that's a good news, thanks for letting us know ! Peter @petket-5 has developped a new version of start_maja, which should be more user friendly. It is in its final tests.