search

olivierhagolle / Start_maja

To process a Sentinel-2 time series with MAJA cloud detection and atmospheric correction processor
Other
51 stars 15 forks source link

MAJA 3.1.1 does not find the the L1 files #29

Closed borlafgis closed 5 years ago

borlafgis commented 5 years ago

Hello

I am trying to use MAJA 3.1.1. I performed the setup like this:

repCode=/mnt/d/Software/MAJAws/repCode # contains DTM,  GIPP_MAJA_3_1_S2AS2B_MUSCATE_TM and userconf
repWork=/mnt/d/Software/MAJAws/repWork # originally was empty
repL1  =/mnt/d/Software/MAJAws/repL1 # contains a site folder 'SC', the .SAFE files are inside of it
repL2  =/mnt/d/Software/MAJAws/repL2 # originally was empty
repMaja=/mnt/d/Software/MAJAws/repMaja/bin/maja # installed with "MAJA-3.1.1_MuscateOnly.run"
repCAMS=/mnt/d/Software/MAJAws/repCAMS # originally it was empry

When I run the script on ubuntu I get the following error:

biom7@Biometrie7:/mnt/d/Software/MAJAws/repCode$ python start_maja.py -s SC -t 35TLL -f /mnt/d/Software/MAJAws/folders.txt -c MAJA_3_1_S2AS2B_MUSCATE_TM -d 20170101 -e 20180930
['/mnt/d/Software/MAJAws/repCode', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

2019-01-31 09:35:35,576 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-01-31 09:35:35,577 - Start-Maja - INFO - => processing date 20170116
2019-01-31 09:35:35,581 - Start-Maja - INFO - dates to process in backward mode :
2019-01-31 09:35:35,581 - Start-Maja - INFO - -- 20170116 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170116T091321_N0204_R050_T35TLL_20170116T091315.SAFE
2019-01-31 09:35:35,583 - Start-Maja - INFO - -- 20170119 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170119T092311_N0204_R093_T35TLL_20170119T092308.SAFE
2019-01-31 09:35:35,584 - Start-Maja - INFO - -- 20170126 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170126T091231_N0204_R050_T35TLL_20170126T091606.SAFE
2019-01-31 09:35:35,587 - Start-Maja - INFO - -- 20170129 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170129T092221_N0204_R093_T35TLL_20170129T092325.SAFE
2019-01-31 09:35:35,589 - Start-Maja - INFO - -- 20170205 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170205T091141_N0204_R050_T35TLL_20170205T091403.SAFE
2019-01-31 09:35:35,590 - Start-Maja - INFO - -- 20170208 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170208T092131_N0204_R093_T35TLL_20170208T092143.SAFE
2019-01-31 09:35:35,591 - Start-Maja - INFO - -- 20170215 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170215T091041_N0204_R050_T35TLL_20170215T091040.SAFE
2019-01-31 09:35:35,602 - Start-Maja - INFO - -- 20170218 : /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170218T092031_N0204_R093_T35TLL_20170218T092738.SAFE
2019-01-31 09:35:35,629 - Start-Maja - INFO - #################################
2019-01-31 09:35:35,629 - Start-Maja - INFO - #################################
2019-01-31 09:35:35,644 - Start-Maja - INFO - processing /mnt/d/Software/MAJAws/repL1/SC/S2A_MSIL1C_20170116T091321_N0204_R050_T35TLL_20170116T091315.SAFE in backward mode
2019-01-31 09:35:35,645 - Start-Maja - INFO - Initialisation mode with backward is longer
2019-01-31 09:35:35,662 - Start-Maja - INFO - MAJA logfile: /mnt/d/Software/MAJAws/repL2/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//S2A_MSIL1C_20170116T091321_N0204_R050_T35TLL_20170116T091315.SAFE.log
2019-01-31 09:35:35,673 - Start-Maja - INFO - #################################
2019-01-31 09:35:35,698 - Start-Maja - INFO - => processing date 20170119
Traceback (most recent call last):
  File "start_maja.py", line 586, in <module>
    start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
  File "start_maja.py", line 444, in start_maja
    logger.info("Using %s L2 type" % L2type)
UnboundLocalError: local variable 'L2type' referenced before assignment
biom7@Biometrie7:/mnt/d/Software/MAJAws/repCode$ 2019-01-31T09:35:35.888944 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] MAJA Software Version : 3.1.1
2019-01-31T09:35:35.902977 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] Overloads the conf value in the command line by '/mnt/d/Software/MAJAws/repWork/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//userconf'.
2019-01-31T09:35:35.960114 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] -------------------------------------------------------------------------
2019-01-31T09:35:35.964926 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] Configuration User System parameters:
2019-01-31T09:35:35.969179 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      L2Note                                          : L2 note
2019-01-31T09:35:35.972738 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      L3Note                                          : L3 note
2019-01-31T09:35:35.976560 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      CheckToolsNote                                  : Checktool note
2019-01-31T09:35:35.982580 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      NbThreads                                       : 8
2019-01-31T09:35:35.987643 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      EnableCleaningTemporaryDirectory                : true
2019-01-31T09:35:35.994638 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      EnableCachingConvertReflectanceData             : true
2019-01-31T09:35:36.001588 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      EnableCleaningCachingDirectoryBeforeProcessing  : false
2019-01-31T09:35:36.012420 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      EnableCleaningCachingDirectoryAfterProcessing   : true
2019-01-31T09:35:36.017874 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      EnableL2ResolutionProcessing                    : true
2019-01-31T09:35:36.029099 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      PreProcessingCleanInputZipFiles                 : false
2019-01-31T09:35:36.034485 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      PostProcessingZipFiles                          : false
2019-01-31T09:35:36.048773 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      PostProcessingCleanFiles                        : false
2019-01-31T09:35:36.053706 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      CheckXMLFilesWithSchema                         : false
2019-01-31T09:35:36.064919 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] -------------------------------------------------------------------------
2019-01-31T09:35:36.076210 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] Using default Admin configuration system file.
2019-01-31T09:35:36.193518 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] -------------------------------------------------------------------------
2019-01-31T09:35:36.202946 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] Configuration Admin System parameters:
2019-01-31T09:35:36.211017 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      System                                          : MAJA
2019-01-31T09:35:36.227326 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      Tolerances / Footprint / Coordinate             : 1e-06
2019-01-31T09:35:36.235829 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *           ''          ''    / Direction              : 1e-06
2019-01-31T09:35:36.244708 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      EnablePerformanceMeasureAlgorithms              : false
2019-01-31T09:35:36.258381 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] -------------------------------------------------------------------------
2019-01-31T09:35:36.275743 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] ******************************************************************************
2019-01-31T09:35:36.283445 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] Application information:
2019-01-31T09:35:36.290893 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      Processor Name               : MAJA_L2_BACKWARD_CHAIN
2019-01-31T09:35:36.299376 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      Test mode                    : false
2019-01-31T09:35:36.305939 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      UserConfigSystemFilename     : /mnt/d/Software/MAJAws/repL2/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//.maja-working-directory/MAJAUserConfigSystem.xml
2019-01-31T09:35:36.313992 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      AdminConfigSystemFileName    : /mnt/d/Software/MAJAws/repL2/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//.maja-working-directory/MAJAAdminConfigSystem.xml
2019-01-31T09:35:36.319685 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]     *      Specifics command line parameters:
2019-01-31T09:35:36.327811 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I]           - TileId: 35TLL
2019-01-31T09:35:36.334755 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] ******************************************************************************
2019-01-31T09:35:36.343022 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-01-31T09:35:36.348606 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-01-31T09:35:36.358521 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </mnt/d/Software/MAJAws/repWork/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//in>.  [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150]
2019-01-31T09:35:36.364711 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]
2019-01-31T09:35:36.373447 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </mnt/d/Software/MAJAws/repWork/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception]  [vnsMajaMainProcessor.cxx:main:129]
2019-01-31T09:35:36.379300 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [E]   [vnsMajaMainProcessor.cxx:main:129]
2019-01-31T09:35:36.386613 Biometrie7 maja-processing-3.1.1 3.1 [000000000035] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]

I am doing something wrong?

Kind regards, Ignacio

olivierhagolle commented 5 years ago

Hi Ignacio, I guess you printed the content of the file /mnt/d/Software/MAJAws/repL2/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//S2A_MSIL1C_20170116T091321_N0204_R050_T35TLL_20170116T091315.SAFE.log

The error is "Impossible to detect a L1 product in the directory </mnt/d/Software/MAJAws/repWork/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//in" I do not know what happens, will try to reproduce it on my side. But did you check that folder ? Does it contain all the .SAFE files necessary as input to the backward mode ?

By the way, start-MAJA has moved to a new repository, https://github.com/CNES/Start-MAJA, and you will find there the link to access to MAJA 3.2.2

Sorry for this new error ! Olivier

borlafgis commented 5 years ago

Hello Olivier

That is just the output I get from the console, the file /mnt/d/Software/MAJAws/repL2/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//S2A_MSIL1C_20170116T091321_N0204_R050_T35TLL_20170116T091315.SAFE.log Is completely empty

There are two errors... the first one from python:

Traceback (most recent call last):
  File "start_maja.py", line 586, in <module>
    start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
  File "start_maja.py", line 444, in start_maja
    logger.info("Using %s L2 type" % L2type)

And the next one from MAJA: Impossible to detect a L1 product in the directory </mnt/d/Software/MAJAws/repWork/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//in>.

When ls said directory I get no return, it is entirely empty. Could be that the python script is failing to create the links to the level 1 archive (repL1)? The .SAFE files might be misplaced?

They are on /mnt/d/Software/MAJAws/repL1/SC

ls /mnt/d/Software/MAJAws/repL1/SC
S2A_MSIL1C_20171013T090921_N0205_R050_T35TLL_20171013T091544.SAFE
S2A_MSIL1C_20180928T090731_N0206_R050_T35TLL_20180928T111750.SAFE
S2B_MSIL1C_20180327T090559_N0206_R050_T35TLL_20180327T131514.SAFE
...

Thanks for your kelp! Kind regards, Ignacio

olivierhagolle commented 5 years ago

Hiola Ignacio I was not able to reproduce your error. Same command line works on my side. But the hint is that /mnt/d/Software/MAJAws/repWork/SC/35TLL/MAJA_3_1_S2AS2B_MUSCATE_TM//in is empty.

I only can advise for you to dig in to find why files are not copied or linked there. lo siento, Olivier

borlafgis commented 5 years ago

Hello Olivier

I will take a look and tell you what was wrong. [EDIT] I have been making some tests and the problem seems to be that when os.system(commande) gets called on the line 437, for some reason Python thinks MAJA finished before it actually does (or MAJA returns 0 before finishing). Thus the next loop is started (nominal mode, line 441 onwards), which seems to expect the presence of a L2 product. As MAJA has been unable to finish, it does not find any, causing the length of nomL2init_Natif and nomL2init_MUSCATE to be 0. This, in turn, casuses L2type and derniereDate to be left undefined, causing the script to crash when it reaches the line 443, where logger.info("Using %s L2 type" % L2type) is called.

Also, I tried to download the version 3.2.2, but I have been unable to get the installer. The section "Get MAJA > Get MAJA Sofware" points to the CNES site, where only V1.0 is available. [EDIT2] Just saw that the new version have been uploaded to the CNES site today

Sorry for the trouble

borlafgis commented 5 years ago

Hello Olivier

I have found a solution. Up to this point I was using either the ubuntu subsystem of windows 10 or a ubuntu virtual machine. Unable to find the problem, I tried in centos... which worked. The only explanation I can think is that maybe the different distros catch the output of commande differently.

Kind regards, Ignacio

olivierhagolle commented 5 years ago

Thanks for the feedback, and happy you managed to have it working. I will try to give it a try on a ubuntu computer. But virtual machines are sometimes tricky... Best regards, Olviier