L2TYPE referenced before assignment

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.

https://gitlab.orfeo-toolbox.org/maja/maja

Apache License 2.0

137 stars 25 forks source link

L2TYPE referenced before assignment #5

Closed Greforb closed 5 years ago

Greforb commented 5 years ago

I run MAJA 3.2.2 on Ubuntu 16.04. I get this error in the beginning. However, after this error it seems to continue, but then gives another error (that may be linked to this one) First error:

2019-03-15 14:33:10,116 - Start-Maja - INFO - => processing date 20160607
Traceback (most recent call last):
  File "./start_maja.py", line 613, in <module>
    start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
  File "./start_maja.py", line 468, in start_maja
    logger.info("Using %s L2 type" % L2type)
UnboundLocalError: local variable 'L2type' referenced before assignment

Second error:

2019-03-15T14:33:10.804557 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-15T14:33:10.821243 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-15T14:33:10.830535 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>.  [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150]
2019-03-15T14:33:10.845801 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]
2019-03-15T14:33:10.854450 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception]  [vnsMajaMainProcessor.cxx:main:129]
2019-03-15T14:33:10.865996 D23515 maja-processing-3.2.2 3.2 [000000028370] [E]   [vnsMajaMainProcessor.cxx:main:129]

And yes it's correct that there are no L1 products in this folder, but there shouldn't be. /media/au173212/bigdrive/JEM/Orkideer/temp_maja is the maja working directory. Hope you can help me?

olivierhagolle commented 5 years ago

Hi, I am lacking information. could you provide full log (the one on the terminal, the .log file), and the command line ? Olivier

Greforb commented 5 years ago

It says:

2019-03-15 14:33:10,115 - Start-Maja - INFO - MAJA logfile: /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log

This logfile is empty. Full output, including command line:

au173212@D23515:~/MAJA/Maja322/Start-MAJA$ sudo python ./start_maja.py -f folders.txt -c GIPP_MAJA_3.1.2_TM -t 32UMF -s Denmark -d 20160101 -e 20170101

2019-03-15 14:33:10,110 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-03-15 14:33:10,111 - Start-Maja - INFO - => processing date 20160604
2019-03-15 14:33:10,111 - Start-Maja - INFO - dates to process in backward mode :
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160604 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160607 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160608T022705_R008_V20160607T104026_20160607T104026.SAFE
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160610 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160614T210959_R051_V20160610T105030_20160610T105030.SAFE
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160614 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160615T165512_R108_V20160614T103231_20160614T103231.SAFE
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160620 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160620T203427_R051_V20160620T105336_20160620T105336.SAFE
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160624 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160624T173407_R108_V20160624T103023_20160624T103023.SAFE
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160627 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160627T213517_R008_V20160627T104023_20160627T104023.SAFE
2019-03-15 14:33:10,111 - Start-Maja - INFO - -- 20160630 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160702T033618_R051_V20160630T105028_20160630T105028.SAFE
2019-03-15 14:33:10,114 - Start-Maja - INFO - #################################
2019-03-15 14:33:10,115 - Start-Maja - INFO - #################################
2019-03-15 14:33:10,115 - Start-Maja - INFO - processing /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE in backward mode
2019-03-15 14:33:10,115 - Start-Maja - INFO - Initialisation mode with backward is longer
2019-03-15 14:33:10,115 - Start-Maja - INFO - MAJA logfile: /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log
2019-03-15 14:33:10,115 - Start-Maja - INFO - #################################
2019-03-15 14:33:10,116 - Start-Maja - INFO - => processing date 20160607
Traceback (most recent call last):
  File "./start_maja.py", line 613, in <module>
    start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
  File "./start_maja.py", line 468, in start_maja
    logger.info("Using %s L2 type" % L2type)
UnboundLocalError: local variable 'L2type' referenced before assignment
au173212@D23515:~/MAJA/Maja322/Start-MAJA$ 2019-03-15T14:33:10.413859 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] MAJA Software Version : 3.2.2
2019-03-15T14:33:10.426104 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Overloads the conf value in the command line by '/media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf'.
2019-03-15T14:33:10.453866 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] -------------------------------------------------------------------------
2019-03-15T14:33:10.454058 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Configuration User System parameters:
2019-03-15T14:33:10.463864 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      L2Note                                          : L2 note
2019-03-15T14:33:10.473928 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      L3Note                                          : L3 note
2019-03-15T14:33:10.483434 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      CheckToolsNote                                  : Checktool note
2019-03-15T14:33:10.493402 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      NbThreads                                       : 8
2019-03-15T14:33:10.503721 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      EnableCleaningTemporaryDirectory                : true
2019-03-15T14:33:10.514227 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      EnableCachingConvertReflectanceData             : true
2019-03-15T14:33:10.524279 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      EnableCleaningCachingDirectoryBeforeProcessing  : false
2019-03-15T14:33:10.534569 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      EnableCleaningCachingDirectoryAfterProcessing   : true
2019-03-15T14:33:10.544949 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      EnableL2ResolutionProcessing                    : true
2019-03-15T14:33:10.555407 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      PreProcessingCleanInputZipFiles                 : false
2019-03-15T14:33:10.566046 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      PostProcessingZipFiles                          : false
2019-03-15T14:33:10.577774 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      PostProcessingCleanFiles                        : false
2019-03-15T14:33:10.588672 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      CheckXMLFilesWithSchema                         : false
2019-03-15T14:33:10.599738 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] -------------------------------------------------------------------------
2019-03-15T14:33:10.610517 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Using default Admin configuration system file.
2019-03-15T14:33:10.626151 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] -------------------------------------------------------------------------
2019-03-15T14:33:10.626380 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Configuration Admin System parameters:
2019-03-15T14:33:10.637365 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      System                                          : MAJA
2019-03-15T14:33:10.652594 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      Tolerances / Footprint / Coordinate             : 1e-06
2019-03-15T14:33:10.663734 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *           ''          ''    / Direction              : 1e-06
2019-03-15T14:33:10.674507 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      EnablePerformanceMeasureAlgorithms              : false
2019-03-15T14:33:10.685941 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] -------------------------------------------------------------------------
2019-03-15T14:33:10.697374 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] ******************************************************************************
2019-03-15T14:33:10.711939 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Application information: 
2019-03-15T14:33:10.722985 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      Processor Name               : MAJA_L2_BACKWARD_CHAIN
2019-03-15T14:33:10.735385 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      Test mode                    : false
2019-03-15T14:33:10.747162 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      UserConfigSystemFilename     : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAUserConfigSystem.xml
2019-03-15T14:33:10.760642 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      AdminConfigSystemFileName    : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAAdminConfigSystem.xml
2019-03-15T14:33:10.770913 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]     *      Specifics command line parameters:
2019-03-15T14:33:10.781895 D23515 maja-processing-3.2.2 3.2 [000000028370] [I]           - TileId: 32UMF
2019-03-15T14:33:10.793240 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] ******************************************************************************
2019-03-15T14:33:10.804557 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-15T14:33:10.821243 D23515 maja-processing-3.2.2 3.2 [000000028370] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-15T14:33:10.830535 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>.  [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150]
2019-03-15T14:33:10.845801 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]
2019-03-15T14:33:10.854450 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception]  [vnsMajaMainProcessor.cxx:main:129]
2019-03-15T14:33:10.865996 D23515 maja-processing-3.2.2 3.2 [000000028370] [E]   [vnsMajaMainProcessor.cxx:main:129]
2019-03-15T14:33:10.878191 D23515 maja-processing-3.2.2 3.2 [000000028370] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]

olivierhagolle commented 5 years ago

The L1 files are copied (symbolic linked) to the tmp directory, which is
/media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in

The directories after temp_maja are created on the fly, and allow to run different sites in parallel with the same folders.txt

So there is an issue with finding the L1 data to copy them on the work dir. I don't know why, but,

you should not install MAJA as root and use Maja as root, it often gives issues with permissions.

Greforb commented 5 years ago

I have now tried uninstalling maja and reinstalling as normal user (not root). and then running as normal user also. Exactly the same error. I have checked that L1 files are the correct spot, and I guess that

2019-03-15 14:33:10,115 - Start-Maja - INFO - processing /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE in backward mode

means that it can actually find the L1 files and that my path is indeed correct? All workspace and in and output dirs are owned by the user (me) running maja so there should be no issues with permissions. Also if i run as root (which should allow anything and circumvent permission problems?) it also still gives the same error. Any ideas for debugging?

Greforb commented 5 years ago

Is the the LocalUnboundError in start-maja.py related to this, or could it have something to do with it? It seems like a variable is referenced before it's assignment?

2019-03-15 14:33:10,116 - Start-Maja - INFO - => processing date 20160607
Traceback (most recent call last):
  File "./start_maja.py", line 613, in <module>
    start_maja(folder_file, context, site, tile, orbit, nb_backward, options, debug_mode)
  File "./start_maja.py", line 468, in start_maja
    logger.info("Using %s L2 type" % L2type)
UnboundLocalError: local variable 'L2type' referenced before assignment

olivierhagolle commented 5 years ago

Hi, I have found time to search in the code (well that shouldn't be on saturdays, but these days are quite busy).

This error happens when the backward processing is unsuccessful, no L2 is generated, and, as a result, it is not possible to guess the type of L2 (depends if you use the TM or noTM version of MAJA).

So I updated start_maja to catch this error, and to print the shell command to launch Maja. Could you pull again start_maja, run it again, get the maja command line, and try it on the shell to see what happens ?

Thanks Olivier

Greforb commented 5 years ago

It seems like the exception i start-maja.py is fixed. Now the error is:

~/MAJA/Maja322/Start-MAJA$ python ./start_maja.py -f folders.txt -c GIPP_MAJA_3.1.2_TM -t 32UMF -s Denmark -d 20160101 -e 20170101

2019-03-17 10:32:25,270 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-03-17 10:32:25,271 - Start-Maja - INFO - => processing date 20160604
2019-03-17 10:32:25,271 - Start-Maja - INFO - dates to process in backward mode :
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160604 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160607 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160608T022705_R008_V20160607T104026_20160607T104026.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160610 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160614T210959_R051_V20160610T105030_20160610T105030.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160614 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160615T165512_R108_V20160614T103231_20160614T103231.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160620 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160620T203427_R051_V20160620T105336_20160620T105336.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160624 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160624T173407_R108_V20160624T103023_20160624T103023.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160627 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160627T213517_R008_V20160627T104023_20160627T104023.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160630 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160702T033618_R051_V20160630T105028_20160630T105028.SAFE
2019-03-17 10:32:25,276 - Start-Maja - INFO - #################################
2019-03-17 10:32:25,276 - Start-Maja - INFO - #################################
2019-03-17 10:32:25,276 - Start-Maja - INFO - processing /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE in backward mode
2019-03-17 10:32:25,276 - Start-Maja - INFO - Initialisation mode with backward is longer
2019-03-17 10:32:25,276 - Start-Maja - INFO - MAJA logfile: /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log
2019-03-17 10:32:25,276 - Start-Maja - INFO - #################################
2019-03-17 10:32:25,278 - Start-Maja - INFO - => processing date 20160607
2019-03-17 10:32:25,280 - Start-Maja - INFO - MAJA command failed : /home/au173212/MAJA/Maja322/maja/bin/maja  -i /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in -o /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM/ -m L2BACKWARD -ucs /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf --TileId 32UMF &> /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log
2019-03-17 10:32:25,280 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation
au173212@D23515:~/MAJA/Maja322/Start-MAJA$ 2019-03-17T10:32:28.568218 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] MAJA Software Version : 3.2.2
2019-03-17T10:32:28.589121 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] Overloads the conf value in the command line by '/media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf'.
2019-03-17T10:32:28.671058 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] -------------------------------------------------------------------------
2019-03-17T10:32:28.671232 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] Configuration User System parameters:
2019-03-17T10:32:28.680184 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      L2Note                                          : L2 note
2019-03-17T10:32:28.689095 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      L3Note                                          : L3 note
2019-03-17T10:32:28.695895 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      CheckToolsNote                                  : Checktool note
2019-03-17T10:32:28.696367 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      NbThreads                                       : 8
2019-03-17T10:32:28.706408 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      EnableCleaningTemporaryDirectory                : true
2019-03-17T10:32:28.716161 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      EnableCachingConvertReflectanceData             : true
2019-03-17T10:32:28.730044 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      EnableCleaningCachingDirectoryBeforeProcessing  : false
2019-03-17T10:32:28.740423 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      EnableCleaningCachingDirectoryAfterProcessing   : true
2019-03-17T10:32:28.750809 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      EnableL2ResolutionProcessing                    : true
2019-03-17T10:32:28.759697 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      PreProcessingCleanInputZipFiles                 : false
2019-03-17T10:32:28.769967 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      PostProcessingZipFiles                          : false
2019-03-17T10:32:28.780387 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      PostProcessingCleanFiles                        : false
2019-03-17T10:32:28.788736 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      CheckXMLFilesWithSchema                         : false
2019-03-17T10:32:28.798351 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] -------------------------------------------------------------------------
2019-03-17T10:32:28.809099 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] Using default Admin configuration system file.
2019-03-17T10:32:28.850576 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] -------------------------------------------------------------------------
2019-03-17T10:32:28.850791 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] Configuration Admin System parameters:
2019-03-17T10:32:28.861288 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      System                                          : MAJA
2019-03-17T10:32:28.872141 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      Tolerances / Footprint / Coordinate             : 1e-06
2019-03-17T10:32:28.882840 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *           ''          ''    / Direction              : 1e-06
2019-03-17T10:32:28.893698 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      EnablePerformanceMeasureAlgorithms              : false
2019-03-17T10:32:28.907560 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] -------------------------------------------------------------------------
2019-03-17T10:32:28.922713 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] ******************************************************************************
2019-03-17T10:32:28.933786 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] Application information: 
2019-03-17T10:32:28.944841 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      Processor Name               : MAJA_L2_BACKWARD_CHAIN
2019-03-17T10:32:28.956021 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      Test mode                    : false
2019-03-17T10:32:28.966781 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      UserConfigSystemFilename     : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAUserConfigSystem.xml
2019-03-17T10:32:28.977498 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      AdminConfigSystemFileName    : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAAdminConfigSystem.xml
2019-03-17T10:32:28.988865 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]     *      Specifics command line parameters:
2019-03-17T10:32:28.000013 D23515 maja-processing-3.2.2 3.2 [000000010355] [I]           - TileId: 32UMF
2019-03-17T10:32:29.010824 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] ******************************************************************************
2019-03-17T10:32:29.022020 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-17T10:32:29.032788 D23515 maja-processing-3.2.2 3.2 [000000010355] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-17T10:32:29.044529 D23515 maja-processing-3.2.2 3.2 [000000010355] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>.  [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150]
2019-03-17T10:32:29.056240 D23515 maja-processing-3.2.2 3.2 [000000010355] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]
2019-03-17T10:32:29.064816 D23515 maja-processing-3.2.2 3.2 [000000010355] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception]  [vnsMajaMainProcessor.cxx:main:129]
2019-03-17T10:32:29.075809 D23515 maja-processing-3.2.2 3.2 [000000010355] [E]   [vnsMajaMainProcessor.cxx:main:129]
2019-03-17T10:32:29.087062 D23515 maja-processing-3.2.2 3.2 [000000010355] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]

I then tried running:

~/MAJA/Maja322/Start-MAJA$ /home/au173212/MAJA/Maja322/maja/bin/maja  -i /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in -o /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM/ -m L2BACKWARD -ucs /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf --TileId 32UMF &> /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log

Nothing happens, no errors, no info, no messages and it seems like it doesn't do anything, possibly because there are no files in the in-directory? So the error happens before this, when it tries to copy or what? But there are no errors before this.

olivierhagolle commented 5 years ago

The output is redirected in a log file : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log that's why you see no log. Please have look at this file Olivier

Greforb commented 5 years ago

The content of that log-file seems to be the same error as before:

2019-03-17T10:42:18.149862 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] MAJA Software Version : 3.2.2
2019-03-17T10:42:18.160280 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] Overloads the conf value in the command line by '/media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf'.
2019-03-17T10:42:18.173702 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] -------------------------------------------------------------------------
2019-03-17T10:42:18.173868 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] Configuration User System parameters:
2019-03-17T10:42:18.183105 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      L2Note                                          : L2 note
2019-03-17T10:42:18.192169 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      L3Note                                          : L3 note
2019-03-17T10:42:18.201337 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      CheckToolsNote                                  : Checktool note
2019-03-17T10:42:18.210730 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      NbThreads                                       : 8
2019-03-17T10:42:18.219324 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      EnableCleaningTemporaryDirectory                : true
2019-03-17T10:42:18.228993 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      EnableCachingConvertReflectanceData             : true
2019-03-17T10:42:18.237757 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      EnableCleaningCachingDirectoryBeforeProcessing  : false
2019-03-17T10:42:18.247169 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      EnableCleaningCachingDirectoryAfterProcessing   : true
2019-03-17T10:42:18.256007 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      EnableL2ResolutionProcessing                    : true
2019-03-17T10:42:18.266009 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      PreProcessingCleanInputZipFiles                 : false
2019-03-17T10:42:18.275966 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      PostProcessingZipFiles                          : false
2019-03-17T10:42:18.286181 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      PostProcessingCleanFiles                        : false
2019-03-17T10:42:18.296345 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      CheckXMLFilesWithSchema                         : false
2019-03-17T10:42:18.306688 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] -------------------------------------------------------------------------
2019-03-17T10:42:18.318104 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] Using default Admin configuration system file.
2019-03-17T10:42:18.343540 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] -------------------------------------------------------------------------
2019-03-17T10:42:18.343723 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] Configuration Admin System parameters:
2019-03-17T10:42:18.356946 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      System                                          : MAJA
2019-03-17T10:42:18.368426 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      Tolerances / Footprint / Coordinate             : 1e-06
2019-03-17T10:42:18.381248 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *           ''          ''    / Direction              : 1e-06
2019-03-17T10:42:18.392038 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      EnablePerformanceMeasureAlgorithms              : false
2019-03-17T10:42:18.402920 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] -------------------------------------------------------------------------
2019-03-17T10:42:18.413790 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] ******************************************************************************
2019-03-17T10:42:18.424354 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] Application information: 
2019-03-17T10:42:18.437455 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      Processor Name               : MAJA_L2_BACKWARD_CHAIN
2019-03-17T10:42:18.448610 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      Test mode                    : false
2019-03-17T10:42:18.449150 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      UserConfigSystemFilename     : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAUserConfigSystem.xml
2019-03-17T10:42:18.460992 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      AdminConfigSystemFileName    : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAAdminConfigSystem.xml
2019-03-17T10:42:18.471794 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]     *      Specifics command line parameters:
2019-03-17T10:42:18.482728 D23515 maja-processing-3.2.2 3.2 [000000010571] [I]           - TileId: 32UMF
2019-03-17T10:42:18.493918 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] ******************************************************************************
2019-03-17T10:42:18.504809 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-03-17T10:42:18.515634 D23515 maja-processing-3.2.2 3.2 [000000010571] [I] Starting L2BackwardProcessor::PreProcessing() ...
2019-03-17T10:42:18.526936 D23515 maja-processing-3.2.2 3.2 [000000010571] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>.  [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150]
2019-03-17T10:42:18.538438 D23515 maja-processing-3.2.2 3.2 [000000010571] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]
2019-03-17T10:42:18.549414 D23515 maja-processing-3.2.2 3.2 [000000010571] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception]  [vnsMajaMainProcessor.cxx:main:129]
2019-03-17T10:42:18.560603 D23515 maja-processing-3.2.2 3.2 [000000010571] [E]   [vnsMajaMainProcessor.cxx:main:129]
2019-03-17T10:42:18.572524 D23515 maja-processing-3.2.2 3.2 [000000010571] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:129]

olivierhagolle commented 5 years ago

Sorry, I have reinstalled everything from scratch on another computer, and I am not able to reproduce this error, so there is no way for me to find out where it comes from.

Start-Maja finds L1 files in the input L1 directory:

2019-03-17 10:32:25,271 - Start-Maja - INFO - dates to process in backward mode :
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160604 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160607 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160608T022705_R008_V20160607T104026_20160607T104026.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160610 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160614T210959_R051_V20160610T105030_20160610T105030.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160614 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160615T165512_R108_V20160614T103231_20160614T103231.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160620 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160620T203427_R051_V20160620T105336_20160620T105336.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160624 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160624T173407_R108_V20160624T103023_20160624T103023.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160627 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160627T213517_R008_V20160627T104023_20160627T104023.SAFE
2019-03-17 10:32:25,271 - Start-Maja - INFO - -- 20160630 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160702T033618_R051_V20160630T105028_20160630T105028.SAFE

But it seems they are the symbolic links are not properly made in the temp directory: /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in

You will have to dig in to find out why. As the temp directory is removed and emptied at the beginning of each L1C processing, it is necessary to make a breakpoint to examine its status before it is erased.

Sorry for the issues. Olivier

Greforb commented 5 years ago

Is there a way in which I can see the error that occurs when it tries to make the symbolic links? It must produce some kind of error somewhere that it does not display?

olivierhagolle commented 5 years ago

If there was an error, it would be detected. That's why you should have a look to the temp folder to find what L1C files were copied:

/media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in

Greforb commented 5 years ago

In that folder no files are found, so nothing is copied/symbolically linked. If you get any ideas on how to solve this, please let me know.

olivierhagolle commented 5 years ago

No, things must be copied, but the directory is then erased at the beginning of each new product, that's why you need to add a line in the tool to stop the execution and look into it

Greforb commented 5 years ago

Ahh, then i misunderstood, add a line exactly where? In start-maja?, and then I need to run it from some IDE using a debug-mode?

olivierhagolle commented 5 years ago

Do you know the python language ? For instance, add sys.exit(-3) at line 557 then run and just look into the temp folder

Greforb commented 5 years ago

Yes I know python very well, but probably not all aspect, hadn't thought of exiting :)

Greforb commented 5 years ago

That somehow doesn't seem right? 556 already has a sys.exit, so I probably already exited (because of the error) when reaching line 557? Or do you mean in another scope? The loop scope? Around lines 430ish it looks like it symlinks the L1C files? Could exit after that?

olivierhagolle commented 5 years ago

in your case, start-maja processes two products, an error is caught at the second one, but not the first one (I don't know why yet) 2019-03-17 10:32:25,271 - Start-Maja - INFO - => processing date 20160604 2019-03-17 10:32:25,278 - Start-Maja - INFO - => processing date 20160607

Adding the second sys.exit(-3) would stop at the first product and you could have a look at the content of temp folder, and the maja log file...

        if Error:
                logger.info("#######################################")
                logger.info("Error detected, see: %s" % Maja_logfile)
                logger.info("#######################################")
                sys.exit(-1)
        sys.exit(-3)

Greforb commented 5 years ago

After that it seems to be now running... all products.... I'm really confused now. Holding my breath waiting to see what happens...

Greforb commented 5 years ago

OK, now it finished (I think, Process ok ;-), code return [0], but it doesn't quit, i.e it doesn't give me back the command line). It seems to have processed only one image as expected, it just did a lot of iterations, to do that. It left a lot of folders and files in the "in" directory as you predicted. These L1C products were copied, and are now found in the temporary "in" directory:

S2A_OPER_PRD_MSIL1C_PDMC_20160608T022705_R008_V20160607T104026_20160607T104026.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20160614T210959_R051_V20160610T105030_20160610T105030.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20160615T165512_R108_V20160614T103231_20160614T103231.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20160620T203427_R051_V20160620T105336_20160620T105336.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20160624T173407_R108_V20160624T103023_20160624T103023.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20160627T213517_R008_V20160627T104023_20160627T104023.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20160702T033618_R051_V20160630T105028_20160630T105028.SAFE S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE

olivierhagolle commented 5 years ago

Good news, what is in the logfile ?

Greforb commented 5 years ago

It's empty, but I can give the screen log that was outputted to the screen, but it's pretty long.

Greforb commented 5 years ago

This is what it printed during processing: MAJA_output.txt

olivierhagolle commented 5 years ago

So the first backward processing worked smoothly. As a result a level2 should have appeared in the temp_dir., and also copied in directory defined as repL2 in your folders.txt.

Can you confirm ? Olviier

Greforb commented 5 years ago

Yes, I'm not 100% familiar with the output format yet, it looks a bit different from what version 1.0 outputted, but also looks familiar. I'm not sure I see the L2 resulting tiffs in the temp folder though, but it could be hidden in one of many folders. This is the content of the temp-folder:

convert_CAMS_DBL.py S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR convert_CAMS_DBL.pyc S2A_TEST_GIP_L2TOCR_L_SEASALT50001_00000000_99999999.D convert_to_exo.py S2A_TEST_GIP_L2TOCR_L_SEASALT50001_00000000_99999999.H download_CAMS_daily.py S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR download_CAMS.py S2A_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR ecmwfapi S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR readme.md S2A_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR README.md S2B_TEST_GIP_CKEXTL_S_31TJF__10001_20150703_21000101. S2A_OPER_PRD_MSIL1C_PDMC_20160608T022705_R008_V20160607T104026 S2A_OPER_PRD_MSIL1C_PDMC_20160614T210959_R051_V20160610T105030 S2A_OPER_PRD_MSIL1C_PDMC_20160615T165512_R108_V20160614T103231 S2A_OPER_PRD_MSIL1C_PDMC_20160620T203427_R051_V20160620T105336 S2A_OPER_PRD_MSIL1C_PDMC_20160624T173407_R108_V20160624T103023 S2A_OPER_PRD_MSIL1C_PDMC_20160627T213517_R008_V20160627T104023 S2A_OPER_PRD_MSIL1C_PDMC_20160702T033618_R051_V20160630T105028 S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026 S2A_TEST_GIP_CKEXTL_S_31TJF10001_20150703_21000101.EEF S2A_TEST_GIP_CKQLTL_S_31TJF__10005_20150703_21000101.EEF S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.DBL.DIR S2A_TEST_GIP_L2ALBD_L_CONTINEN_10005_20150703_21000101.HDR S2A_TEST_GIP_L2ALBD_L_DUST_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2ALBD_L_DUST___50001_00000000_99999999.HDR S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR S2A_TEST_GIP_L2ALBD_L_SEASALT50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2ALBD_L_SEASALT50001_00000000_99999999.HDR S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR S2A_TEST_GIP_L2COMM_L_ALLSITES_10009_20150703_21000101.EEF S2A_TEST_GIP_L2COMM_L_ALLSITES_99999_20150703_21000101.EEF S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.DBL.DIR S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.DBL.DIR S2A_TEST_GIP_L2DIFT_L_CONTINEN_10005_20150703_21000101.HDR S2A_TEST_GIP_L2DIFT_L_DUST___50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIFT_L_DUST_50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIFT_L_SEASALT50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIFT_L_SEASALT50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.DBL.DIR S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.DBL.DIR S2A_TEST_GIP_L2DIRT_L_CONTINEN_10005_20150703_21000101.HDR S2A_TEST_GIP_L2DIRT_L_DUST_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIRT_L_DUST___50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIRT_L_SEASALT50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIRT_L_SEASALT50001_00000000_99999999.HDR S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR S2A_TEST_GIP_L2SMAC_L_ALLSITES_10005_20150703_21000101.EEF S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.DBL.DIR S2A_TEST_GIP_L2TOCR_L_CONTINEN_10005_20150703_21000101.HDR S2A_TEST_GIP_L2TOCR_L_DUST___50001_00000000_99999999.DBL.DIR S2A_TEST_GIP_L2TOCR_L_DUST___50001_00000000_99999999.HDR S2A_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR
10005_20150703_21000101.EEF _20160610T105030.SAFE S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.DBL.DIR _20160614T103231.SAFE S2B_TEST_GIP_L2ALBD_L_BLACKCAR_50001_00000000_99999999.HDR _20160620T105336.SAFE S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.DBL.DIR _20160624T103023.SAFE S2B_TEST_GIP_L2ALBD_L_CONTINEN_10003_20150703_21000101.HDR _20160627T104023.SAFE S2B_TEST_GIP_L2ALBD_L_DUST_50001_00000000_99999999.DBL.DIR _20160630T105028.SAFE S2B_TEST_GIP_L2ALBD_L_DUST_____50001_00000000_99999999.HDR _20160604T103026.SAFE S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2ALBD_L_ORGANICM_50001_00000000_99999999.HDR S2B_TEST_GIP_L2ALBD_L_SEASALT__50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2ALBD_L_SEASALT50001_00000000_99999999.HDR S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2ALBD_L_SULPHATE_50001_00000000_99999999.HDR S2B_TEST_GIP_L2COMM_L_ALLSITES_10009_20150703_21000101.EEF S2B_TEST_GIP_L2COMM_L_ALLSITES_99999_20150703_21000101.EEF S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIFT_L_BLACKCAR_50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.DBL.DIR S2B_TEST_GIP_L2DIFT_L_CONTINEN_10002_20150703_21000101.HDR S2B_TEST_GIP_L2DIFT_L_DUST___50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIFT_L_DUST___50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIFT_L_ORGANICM_50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIFT_L_SEASALT50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIFT_L_SEASALT50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIFT_L_SULPHATE_50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIRT_L_BLACKCAR_50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.DBL.DIR S2B_TEST_GIP_L2DIRT_L_CONTINEN_10002_20150703_21000101.HDR S2B_TEST_GIP_L2DIRT_L_DUST___50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIRT_L_DUST_____50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIRT_L_ORGANICM_50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIRT_L_SEASALT__50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIRT_L_SEASALT50001_00000000_99999999.HDR S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2DIRT_L_SULPHATE_50001_00000000_99999999.HDR S2B_TEST_GIP_L2SMAC_L_ALLSITES_10005_20150703_21000101.EEF S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2TOCR_L_BLACKCAR_50001_00000000_99999999.HDR S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.DBL.DIR S2B_TEST_GIP_L2TOCR_L_CONTINEN_10002_20150703_21000101.HDR S2B_TEST_GIP_L2TOCR_L_DUST_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2TOCR_LDUST50001_00000000_99999999.HDR S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2TOCR_L_ORGANICM_50001_00000000_99999999.HDR S2B_TEST_GIP_L2TOCR_L_SEASALT__50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2TOCR_L_SEASALT50001_00000000_99999999.HDR S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.DBL.DIR S2B_TEST_GIP_L2TOCR_L_SULPHATE_50001_00000000_99999999.HDR S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.DBL.DIR S2B_TEST_GIP_L2WATV_L_CONTINEN_10006_20150703_21000101.HDR S2TEST_AUX_REFDE2_T32UMF_0001.DBL.DIR S2TEST_AUX_REFDE2_T32UMF_0001.HDR S2TEST_GIP_L2SITE_S_31TJF__10001_00000000_99999999.EEF />

olivierhagolle commented 5 years ago

The L2A must have been moved to the repL2 folder. Did you find it there ? Olivier

Greforb commented 5 years ago

Yes exactly the L2A products are in the repL2 folder as they should. What I tried to say was that I don't know yet if everything is there. The following is in the repL2 folder:

S2A_OPER_PMC_L2REPT_32UMF____20160604.EEF S2A_TEST_QCK_L2VALD_T32UMF_20160604.DBL.DIR

S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log

S2A_TEST_QCK_L2VALD_T32UMF_20160604.HDR S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0

In the SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0 folder the following is found:

DATA SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B5.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B2.tif MASKS SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B6.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B3.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_ATB_R1.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B7.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B4.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_ATB_R2.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B8A.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B5.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B11.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B8.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B6.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B12.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_MTD_ALL.xml SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B7.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B2.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_QKL_ALL.jpg SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B8A.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B3.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B11.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B8.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_FRE_B4.tif SENTINEL2A_20160604-103026-458_L2A_T32UMF_C_V1-0_SRE_B12.tif

Which I guess is perfectly fine?

olivierhagolle commented 5 years ago

Yes, it looks fine. But we have not found any explanation for what was happening in the first place. Did you change any other thing ? But as everything seems OK, could you try to remove sys.exit() that was added and see what happens Olivier

Greforb commented 5 years ago

Still exactly the same error!

python ./start_maja.py -f folders.txt -c GIPP_MAJA_3.1.2_TM -t 32UMF -s Denmark -d 20160101 -e 20170101 2019-03-19 11:13:25,444 - Start-Maja - INFO - No existing L2 product, we start with backward mode 2019-03-19 11:13:25,444 - Start-Maja - INFO - => processing date 20160604 2019-03-19 11:13:25,444 - Start-Maja - INFO - dates to process in backward mode : 2019-03-19 11:13:25,444 - Start-Maja - INFO - -- 20160604 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE 2019-03-19 11:13:25,444 - Start-Maja - INFO - -- 20160607 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160608T022705_R008_V20160607T104026_20160607T104026.SAFE 2019-03-19 11:13:25,444 - Start-Maja - INFO - -- 20160610 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160614T210959_R051_V20160610T105030_20160610T105030.SAFE 2019-03-19 11:13:25,444 - Start-Maja - INFO - -- 20160614 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160615T165512_R108_V20160614T103231_20160614T103231.SAFE 2019-03-19 11:13:25,444 - Start-Maja - INFO - -- 20160620 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160620T203427_R051_V20160620T105336_20160620T105336.SAFE 2019-03-19 11:13:25,444 - Start-Maja - INFO - -- 20160624 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160624T173407_R108_V20160624T103023_20160624T103023.SAFE 2019-03-19 11:13:25,445 - Start-Maja - INFO - -- 20160627 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160627T213517_R008_V20160627T104023_20160627T104023.SAFE 2019-03-19 11:13:25,445 - Start-Maja - INFO - -- 20160630 : /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160702T033618_R051_V20160630T105028_20160630T105028.SAFE 2019-03-19 11:13:25,451 - Start-Maja - INFO - ################################# 2019-03-19 11:13:25,451 - Start-Maja - INFO - ################################# 2019-03-19 11:13:25,451 - Start-Maja - INFO - processing /media/au173212/bigdrive/Sentinel2raw/Unpacked/Denmark/S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE in backward mode 2019-03-19 11:13:25,451 - Start-Maja - INFO - Initialisation mode with backward is longer 2019-03-19 11:13:25,451 - Start-Maja - INFO - MAJA logfile: /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log 2019-03-19 11:13:25,451 - Start-Maja - INFO - ################################# 2019-03-19 11:13:25,453 - Start-Maja - INFO - => processing date 20160607 2019-03-19 11:13:25,455 - Start-Maja - INFO - MAJA command failed : /home/au173212/MAJA/Maja322/maja/bin/maja -i /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in -o /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM/ -m L2BACKWARD -ucs /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf --TileId 32UMF &> /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log 2019-03-19 11:13:25,455 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation au173212@D23515:~/MAJA/Maja322/Start-MAJA$ 2019-03-19T11:13:25.540066 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] MAJA Software Version : 3.2.2 2019-03-19T11:13:25.550710 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] Overloads the conf value in the command line by '/media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf'. 2019-03-19T11:13:25.564547 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] ------------------------------------------------------------------------- 2019-03-19T11:13:25.564709 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] Configuration User System parameters: 2019-03-19T11:13:25.574103 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * L2Note : L2 note 2019-03-19T11:13:25.583096 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * L3Note : L3 note 2019-03-19T11:13:25.592386 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * CheckToolsNote : Checktool note 2019-03-19T11:13:25.605585 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * NbThreads : 8 2019-03-19T11:13:25.613963 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * EnableCleaningTemporaryDirectory : true 2019-03-19T11:13:25.623736 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * EnableCachingConvertReflectanceData : true 2019-03-19T11:13:25.632172 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * EnableCleaningCachingDirectoryBeforeProcessing : false 2019-03-19T11:13:25.642269 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * EnableCleaningCachingDirectoryAfterProcessing : true 2019-03-19T11:13:25.652479 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * EnableL2ResolutionProcessing : true 2019-03-19T11:13:25.662794 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * PreProcessingCleanInputZipFiles : false 2019-03-19T11:13:25.671614 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * PostProcessingZipFiles : false 2019-03-19T11:13:25.680227 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * PostProcessingCleanFiles : false 2019-03-19T11:13:25.690762 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * CheckXMLFilesWithSchema : false 2019-03-19T11:13:25.703666 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] ------------------------------------------------------------------------- 2019-03-19T11:13:25.714131 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] Using default Admin configuration system file. 2019-03-19T11:13:25.737434 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] ------------------------------------------------------------------------- 2019-03-19T11:13:25.737628 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] Configuration Admin System parameters: 2019-03-19T11:13:25.748117 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * System : MAJA 2019-03-19T11:13:25.759189 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * Tolerances / Footprint / Coordinate : 1e-06 2019-03-19T11:13:25.769768 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * '' '' / Direction : 1e-06 2019-03-19T11:13:25.780531 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * EnablePerformanceMeasureAlgorithms : false 2019-03-19T11:13:25.791161 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] ------------------------------------------------------------------------- 2019-03-19T11:13:25.801816 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] ****************************************************************************** 2019-03-19T11:13:25.813176 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] Application information: 2019-03-19T11:13:25.824039 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * Processor Name : MAJA_L2_BACKWARD_CHAIN 2019-03-19T11:13:25.834824 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * Test mode : false 2019-03-19T11:13:25.845628 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * UserConfigSystemFilename : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAUserConfigSystem.xml 2019-03-19T11:13:25.856721 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * AdminConfigSystemFileName : /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//.maja-working-directory/MAJAAdminConfigSystem.xml 2019-03-19T11:13:25.868386 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] * Specifics command line parameters: 2019-03-19T11:13:25.877060 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] - TileId: 32UMF 2019-03-19T11:13:25.884951 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] ****************************************************************************** 2019-03-19T11:13:25.894668 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads. 2019-03-19T11:13:25.908918 D23515 maja-processing-3.2.2 3.2 [000000020774] [I] Starting L2BackwardProcessor::PreProcessing() ... 2019-03-19T11:13:25.920413 D23515 maja-processing-3.2.2 3.2 [000000020774] [E] vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:GetListOfL1Products:150] 2019-03-19T11:13:25.931551 D23515 maja-processing-3.2.2 3.2 [000000020774] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129] 2019-03-19T11:13:25.941292 D23515 maja-processing-3.2.2 3.2 [000000020774] [E] PreProcessing: vns::Data::ERROR: : Impossible to detect a L1 product in the directory </media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in>. [vnsL1ImageInformationsProviderFactory.cxx:150] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129] 2019-03-19T11:13:25.941768 D23515 maja-processing-3.2.2 3.2 [000000020774] [E] [vnsMajaMainProcessor.cxx:main:129] 2019-03-19T11:13:25.952388 D23515 maja-processing-3.2.2 3.2 [000000020774] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]

I wonder about the maja command that seemingly fails:

/home/au173212/MAJA/Maja322/maja/bin/maja -i /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//in -o /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM/ -m L2BACKWARD -ucs /media/au173212/bigdrive/JEM/Orkideer/temp_maja/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//userconf --TileId 32UMF &> /media/au173212/bigdrive/SentinelProcessed/Denmark/32UMF/GIPP_MAJA_3.1.2_TM//S2A_OPER_PRD_MSIL1C_PDMC_20160718T094807_R108_V20160604T103026_20160604T103026.SAFE.log

The &> after the --TileID 32UMF is that correct? Shouldn't it be a -logfile ... or similar?

olivierhagolle commented 5 years ago

&> redirects the outputs to a file. It has been working for years at many users.

Greforb commented 5 years ago

Right, so not a breakthrough. Any other suggestions to find the problem? Can I run the start-maja in debug mode and get more relevant information?

olivierhagolle commented 5 years ago

Hi, I am sorry, but being not able to reproduce your error, I don't know how to help you.

If I summarize, because I am not always sure what you exactly did :

the install of MAJA works (you have been able to produce a L2A in backward mode)

but this L2A was only obtained adding a sys.exit(-1)

[IMPORTANT] did you obtain it qith stat-maja, of did you get the command line from stat_maja log and paste it on the shell ?

When you launch without sys.exit (-1) (https://github.com/CNES/Start-MAJA/issues/5#issuecomment-474290203)

the L2A is not produced (less than 1s is spent in generating the first L2A product). 2019-03-19 11:13:25,444 - Start-Maja - INFO - => processing date 20160604 2019-03-19 11:13:25,453 - Start-Maja - INFO - => processing date 20160607

So I don't understand how you were able to generate a L2A once but no more. As I said, not being able to reproduce it, i have to let you search in start-maja code.

olivierhagolle commented 5 years ago

"The &> after the --TileID 32UMF is that correct? Shouldn't it be a -logfile ... or similar"

Well ahem, it seems you were right here, I just learned that some types of shells do not accept redirections as I did. I have updated start_maja.py using the portable version of stdout and stderr streams redirections https://stackoverflow.com/questions/6674327/redirect-all-output-to-file

I am not sure it wil solve all issues, but at least, Maja log will get into the logfile Olivier

olivierhagolle commented 5 years ago

It seems the modification solved this issue https://github.com/CNES/Start-MAJA/issues/2 You were right. Did you try it ? Olivier

Greforb commented 5 years ago

I will check it asap. Under high pressure from other tasks currently, but will let you know as soon as I've tried. And sorry for delayed this...

Greforb commented 5 years ago

The first signs shows that this actually fixed my problem. It seems to be running now. Will leave it and see if it finishes :)

Greforb commented 5 years ago

It seems to have fixed the issue. It has now finished two tiles! Amazing. But I wonder why this fix worked?

olivierhagolle commented 5 years ago

Thanks for good news !

I have not tested it, although I could try with anther shell. Start Maja finds information in the log file, that may be a reason why it failed (as the log file was empty). But I am not sure, because the error message should have been different.

© Githubissues.

Githubissues is a development platform for aggregating issues.