search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
141 stars 25 forks source link

ERROR (Start_MAJA) - First backward processing was unsuccessful, check MAJA installation #54

Open egregr opened 4 years ago

egregr commented 4 years ago
  1. Install MAJA Version: 3.3.2 [TM]
  2. Files in Start-MAJA/ Common/ , folders.txt , prepare_mnt , start_maja.py , userconf/ , cams_download/ , prepare_dtm , Readme.md

  3. Folders and files added in Start-MAJA

    • DTM/S2TEST_AUX_REFDE2_T31TFJ_0001 containing S2TEST_AUX_REFDE2_T31TFJ_0001.DBL.DIR S2__TEST_AUX_REFDE2_T31TFJ_0001.HDR These are the result of the 'Create DTM' step applied on L1C data The L1C data was taken by performing python ./peps_download.py -c S2ST -l --lon 4.8 --lat 43.97 -a peps.txt -d 2015-11-01 -f 2015-12-01 -w /var/local/maja/L1C_DATA/4-8_43-97 The folder /var/local/maja/L1C_DATA/4-8_43-97 contains S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.zip S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.SAFE S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.zip S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.SAFE S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.zip S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.SAFE S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.zip

    • GIPP_S2AS2B/SENTINEL2_TM/ containing files S2B_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF S2A_TEST_GIP_CKEXTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF S2A_TEST_GIP_CKEXTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF S2A_TEST_GIP_CKQLTL_S_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF S2A_TEST_GIP_CKQLTL_S_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF S2A_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF S2A_TEST_GIP_L2COMM_L_ALLSITES_10001_20190626_21000101.EEF S2B_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF S2A_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF

    • LUT_SENT2_NOCAMS/LUTs containing (https://zenodo.org/record/3368623) S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.HDR S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL.DIR S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR S2B_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.HDR

    • TMP_FILES/

This is the folder.txt file: repCode=/var/local/maja/Start-MAJA repWork=/var/local/maja/Start-MAJA/TMP_FILES repL1 =/var/local/maja/L1C_DATA repL2 =/var/local/maja/L2A_DATA repMaja=/var/local/maja/bin/maja repCAMS=/var/local/maja/Start-MAJA/CAMS

I encounter an error when performing start_maja.py

python ./start_maja.py -f folders.txt -g GIPP_S2AS2B/SENTINEL2_TM -l LUT_SENT2_NOCAMS/LUTs -t 31TFJ -s 4-8_43-97 -d 20151101 -e 20151201

2020-04-02 15:59:24,574 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJT20151117.DBL.DIR not found 2020-04-02 15:59:24,574 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151117-_T31TFJ_C_V not found 2020-04-02 15:59:24,575 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJT20151120.DBL.DIR not found 2020-04-02 15:59:24,575 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151120-_T31TFJ_C_V not found 2020-04-02 15:59:24,576 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJT20151127.DBL.DIR not found 2020-04-02 15:59:24,576 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151127-_T31TFJ_C_V not found 2020-04-02 15:59:24,577 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_31TFJT20151130.DBL.DIR not found 2020-04-02 15:59:24,577 - Start-Maja - INFO - /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//SENTINEL2?_20151130-_T31TFJ_C_V not found 2020-04-02 15:59:24,577 - Start-Maja - INFO - No existing L2 product, we start with backward mode 2020-04-02 15:59:24,577 - Start-Maja - INFO - => processing date 20151117 2020-04-02 15:59:24,577 - Start-Maja - INFO - dates to process in backward mode : 2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151117 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE 2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151120 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151120T104332_N0204_R008_T31TFJ_20151120T104750.SAFE 2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151127 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151127T103352_N0204_R108_T31TFJ_20151127T103440.SAFE 2020-04-02 15:59:24,577 - Start-Maja - INFO - -- 20151130 : /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151130T104412_N0204_R008_T31TFJ_20151130T104646.SAFE 2020-04-02 15:59:24,581 - Start-Maja - INFO - ################################# 2020-04-02 15:59:24,581 - Start-Maja - INFO - ################################# 2020-04-02 15:59:24,581 - Start-Maja - INFO - processing /var/local/maja/L1C_DATA/4-8_43-97/S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE in backward mode 2020-04-02 15:59:24,581 - Start-Maja - INFO - Initialisation mode with backward is longer 2020-04-02 15:59:24,581 - Start-Maja - INFO - MAJA logfile: /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log 2020-04-02 15:59:24,582 - Start-Maja - INFO - ################################# 2020-04-02 15:59:24,586 - Start-Maja - INFO - => processing date 20151120 2020-04-02 15:59:24,587 - Start-Maja - INFO - MAJA command failed : /var/local/maja/bin/maja -i /var/local/maja/Start-MAJA/TMP_FILES/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//in -o /var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM/ -m L2BACKWARD -ucs /var/local/maja/Start-MAJA/TMP_FILES/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//userconf --TileId 31TFJ >/var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log 2>&1 2020-04-02 15:59:24,587 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation

octoflar commented 4 years ago

I had the same issue two weeks ago. Meanwhile Maja 3.3.5 is available but I could not test yet whether the same issue occurs again.

olivierhagolle commented 4 years ago

Hi, thanks for submitting the issue.

MAJA 3.3.5 will not solve the issue, it only improves performances on some Linux distributions such as RH7. Could you please provide the log file from MAJA ?

/var/local/maja/L2A_DATA/4-8_43-97/31TFJ/GIPP_S2AS2B/SENTINEL2_TM//S2A_MSIL1C_20151117T103312_N0204_R108_T31TFJ_20151117T103313.SAFE.log

Thanks, Olivier

egregr commented 4 years ago

/var/local/maja/bin/maja-processing-3.3.2: error while loading shared libraries: libMajaChain.so.3.3: cannot open shared object file

petket-5 commented 4 years ago

There seems to be some problem with your maja installation. Do you have any additional PATH or LD_LIBRARY_PATH variables set (maybe in your ~/.bashrc)?

To debug, please run ldd /var/local/maja/lib/libMajaChain.so.3.3 and post the output here. Also, what OS are you running on?

Kind regards, Peter

octoflar commented 4 years ago

I have the same issue. Typing

python ./start_maja.py -f myfolders.txt -g ../gipp/SENTINEL2_NATIF -l ../luts/LUT_MAJA_3_TM_CAMS -t 32UNE -s Hamburg -d 20190714 -e 20190717

processes the first date successfully, but then aborts with an error:

2020-04-06 08:20:18,076 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190714.DBL.DIR not found
2020-04-06 08:20:18,076 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190714-*_T32UNE_C_V* not found
2020-04-06 08:20:18,079 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2?_OPER_SSC_L2VALD_32UNE____T20190717.DBL.DIR not found
2020-04-06 08:20:18,079 - Start-Maja - INFO -  /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//SENTINEL2?_20190717-*_T32UNE_C_V* not found
2020-04-06 08:20:18,079 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-06 08:20:18,114 - Start-Maja - INFO - => processing date 20190714
2020-04-06 08:20:18,116 - Start-Maja - INFO - dates to process in backward mode :
2020-04-06 08:20:18,116 - Start-Maja - INFO - -- 20190714 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE
2020-04-06 08:20:18,118 - Start-Maja - INFO - -- 20190717 : /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190717T104029_N0208_R008_T32UNE_20190717T142616.SAFE
2020-04-06 08:20:18,622 - Start-Maja - INFO - #################################
2020-04-06 08:20:18,622 - Start-Maja - INFO - #################################
2020-04-06 08:20:18,622 - Start-Maja - INFO - processing /home/janwevers/worldcover/proc/l1c/Hamburg/S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE in backward mode
2020-04-06 08:20:18,622 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-06 08:20:18,622 - Start-Maja - INFO - MAJA logfile: /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log
2020-04-06 08:20:18,622 - Start-Maja - INFO - #################################
2020-04-06 08:53:12,119 - Start-Maja - INFO - => processing date 20190717
2020-04-06 08:53:12,721 - Start-Maja - INFO - MAJA command failed : /home/janwevers/worldcover/maja/bin/maja  -i /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//in -o /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF/ -m L2BACKWARD -ucs /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//userconf --TileId 32UNE >/home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log 2>&1
2020-04-06 08:53:12,871 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation

The log file from processing the first date is attached.

S2B_MSIL1C_20190714T103029_N0208_R108_T32UNE_20190714T122554.SAFE.log

egregr commented 4 years ago

Indeed the problem was that certain soft paths to the libraries were broken i.e. all the libraries were in the folder /var/local/maja/lib/ but were not found

octoflar commented 4 years ago

When I re-issue the failed command manually

/home/janwevers/worldcover/maja/bin/maja  -i /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//in -o /home/janwevers/worldcover/proc/out/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF/ -m L2BACKWARD -ucs /home/janwevers/worldcover/proc/tmp/maja/Hamburg/32UNE/../gipp/SENTINEL2_NATIF//userconf --TileId 32UNE

I obtain the log output attached.

out.log

octoflar commented 4 years ago

The problem does not occur for the TM version, which I use now instead of no_TM.

olivierhagolle commented 4 years ago

Good, we managed to have you using the TM version ;)

I do not think it was related to the version but rather to the way it had been installed. but who knows. Olivier