search

olivierhagolle / Start_maja

To process a Sentinel-2 time series with MAJA cloud detection and atmospheric correction processor
Other
51 stars 15 forks source link

The input directory contains more than one input data doesn't respect the format #20

Closed bim2010 closed 5 years ago

bim2010 commented 5 years ago

Hi Olivier, Here is command I'm typing in: python start_maja.py -f folders.txt -c MAJA_3_1_S2AS2B_CAMS -t T36UVG -s T36UVG -d 20180503 folders.txt contents:

repCode=/opt/maja/Start_maja
repWork=/opt/maja/work
repL1  =/opt/maja/repL1
repL2  =/opt/maja/repL2
repMaja=/opt/maja/core/1.0/bin/maja
repCAMS  =/opt/maja/dbl

maja output:

2018-11-13T15:27:39.576536 localhost.localdomain maja-processing 01.00 [000000011102] [I] MAJA Software Version : 1.0.0
2018-11-13T15:27:39.578679 localhost.localdomain maja-processing 01.00 [000000011102] [I] Overloads the conf value in the command line by '/opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//userconf'.
/opt/maja/repL2/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//.maja-working-directory/MAJAUserConfigSystem.xml validates
2018-11-13T15:27:39.590974 localhost.localdomain maja-processing 01.00 [000000011102] [I] -------------------------------------------------------------------------
2018-11-13T15:27:39.591392 localhost.localdomain maja-processing 01.00 [000000011102] [I] Configuration User System parameters:
2018-11-13T15:27:39.591806 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      L2Note                                          : L2 note
2018-11-13T15:27:39.592238 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      L3Note                                          : L3 note
2018-11-13T15:27:39.592741 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      CheckToolsNote                                  : Checktool note
2018-11-13T15:27:39.593156 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      NbThreads                                       : 8
2018-11-13T15:27:39.593521 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      EnableCleaningTemporaryDirectory                : true
2018-11-13T15:27:39.593927 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      EnableCachingConvertReflectanceData             : true
2018-11-13T15:27:39.594329 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      EnableCleaningCachingDirectoryBeforeProcessing  : false
2018-11-13T15:27:39.594758 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      EnableCleaningCachingDirectoryAfterProcessing   : true
2018-11-13T15:27:39.595169 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      EnableL2ResolutionProcessing                    : true
2018-11-13T15:27:39.595609 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      PreProcessingCleanInputZipFiles                 : false
2018-11-13T15:27:39.596028 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      PostProcessingZipFiles                          : false
2018-11-13T15:27:39.596475 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      PostProcessingCleanFiles                        : false
2018-11-13T15:27:39.596911 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      CheckXMLFilesWithSchema                         : false
2018-11-13T15:27:39.597371 localhost.localdomain maja-processing 01.00 [000000011102] [I] -------------------------------------------------------------------------
2018-11-13T15:27:39.597914 localhost.localdomain maja-processing 01.00 [000000011102] [I] Using default Admin configuration system file.
/opt/maja/repL2/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//.maja-working-directory/MAJAAdminConfigSystem.xml validates
2018-11-13T15:27:39.608732 localhost.localdomain maja-processing 01.00 [000000011102] [I] -------------------------------------------------------------------------
2018-11-13T15:27:39.609159 localhost.localdomain maja-processing 01.00 [000000011102] [I] Configuration Admin System parameters:
2018-11-13T15:27:39.609713 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      System                                          : MAJA
2018-11-13T15:27:39.610352 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      Tolerances / Footprint / Coordinate             : 1e-06
2018-11-13T15:27:39.610858 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *           ''          ''    / Direction              : 1e-06
2018-11-13T15:27:39.611296 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      EnablePerformanceMeasureAlgorithms              : false
2018-11-13T15:27:39.611760 localhost.localdomain maja-processing 01.00 [000000011102] [I] -------------------------------------------------------------------------
2018-11-13T15:27:39.612333 localhost.localdomain maja-processing 01.00 [000000011102] [I] ******************************************************************************
2018-11-13T15:27:39.612746 localhost.localdomain maja-processing 01.00 [000000011102] [I] Application information: 
2018-11-13T15:27:39.613259 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      Processor Name               : MAJA_L2_BACKWARD_CHAIN
2018-11-13T15:27:39.613798 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      Test mode                    : false
2018-11-13T15:27:39.614365 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      UserConfigSystemFilename     : /opt/maja/repL2/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//.maja-working-directory/MAJAUserConfigSystem.xml
2018-11-13T15:27:39.614861 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      AdminConfigSystemFileName    : /opt/maja/repL2/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//.maja-working-directory/MAJAAdminConfigSystem.xml
2018-11-13T15:27:39.615386 localhost.localdomain maja-processing 01.00 [000000011102] [I]     *      Specifics command line parameters:
2018-11-13T15:27:39.619813 localhost.localdomain maja-processing 01.00 [000000011102] [I]           - TileId: T36UVG
2018-11-13T15:27:39.620483 localhost.localdomain maja-processing 01.00 [000000011102] [I] ******************************************************************************
2018-11-13T15:27:39.620971 localhost.localdomain maja-processing 01.00 [000000011102] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2018-11-13T15:27:39.621583 localhost.localdomain maja-processing 01.00 [000000011102] [I] Starting L2BackwardProcessor::PreProcessing() ...
2018-11-13T15:27:39.907778 localhost.localdomain maja-processing 01.00 [000000011102] [I] There are 8 L1 images products.
2018-11-13T15:27:39.911497 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2B_MSIL1C_20180503T085559_N0206_R007_T36UVG_20180503T115055.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:39.913566 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2A_MSIL1C_20180508T085601_N0206_R007_T36UVG_20180508T141842.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:39.967285 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2A_MSIL1C_20180511T090601_N0206_R050_T36UVG_20180511T103619.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:39.994082 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2B_MSIL1C_20180513T085559_N0206_R007_T36UVG_20180513T110715.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:39.994666 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2A_MSIL1C_20180521T090601_N0206_R050_T36UVG_20180521T104113.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:39.995258 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2B_MSIL1C_20180526T091019_N0206_R050_T36UVG_20180526T125255.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:39.995914 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2A_MSIL1C_20180528T085641_N0206_R007_T36UVG_20180528T103122.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:39.996473 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - /opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//in/S2A_MSIL1C_20180531T091031_N0206_R050_T36UVG_20180531T111240.SAFE/MTD_MSIL1C.xml
2018-11-13T15:27:40.250978 localhost.localdomain maja-processing 01.00 [000000011102] [P] -----------------------------------------------------------------------------------------------
2018-11-13T15:27:40.251501 localhost.localdomain maja-processing 01.00 [000000011102] [P] Starting L2Processor::PreProcessing() ...
2018-11-13T15:27:40.252931 localhost.localdomain maja-processing 01.00 [000000011102] [I] List of the 2 satellite(s) activated :
2018-11-13T15:27:40.253524 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - SENTINEL-2A ('common' satellite SENTINEL-2_ and plugin SENTINEL2)
2018-11-13T15:27:40.254090 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - SENTINEL-2B ('common' satellite SENTINEL-2_ and plugin SENTINEL2)
2018-11-13T15:27:40.254666 localhost.localdomain maja-processing 01.00 [000000011102] [I] Starting reading and checking input data common for each plugin activated...
2018-11-13T15:27:40.255302 localhost.localdomain maja-processing 01.00 [000000011102] [I] According to the input data, there is(are) 1 plugin(s) activated:
2018-11-13T15:27:40.255949 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - SENTINEL2
2018-11-13T15:27:40.256626 localhost.localdomain maja-processing 01.00 [000000011102] [I]  - SENTINEL2
2018-11-13T15:27:40.263676 localhost.localdomain maja-processing 01.00 [000000011102] [I] Starting DEM reading data for the 'unique' (common) satellite 'SENTINEL-2_' ...
2018-11-13T15:27:40.264199 localhost.localdomain maja-processing 01.00 [000000011102] [I] UnCompress the DBL product </opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS/in/S2__TEST_AUX_REFDE2_TT36UVG_0001.DBL> ...
2018-11-13T15:27:40.264936 localhost.localdomain maja-processing 01.00 [000000011102] [I] The DBL product </opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS/in/S2__TEST_AUX_REFDE2_TT36UVG_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2018-11-13T15:27:40.266514 localhost.localdomain maja-processing 01.00 [000000011102] [I] l_NumberOfL2Resolution : 2
2018-11-13T15:27:40.266964 localhost.localdomain maja-processing 01.00 [000000011102] [I] Starting DTM reading ...
2018-11-13T15:27:40.328470 localhost.localdomain maja-processing 01.00 [000000011102] [I] UnCompress the DBL product </opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS/in/S2__TEST_AUX_REFDE2_TT36UVG_0001.DBL> ...
2018-11-13T15:27:40.329150 localhost.localdomain maja-processing 01.00 [000000011102] [I] The DBL product </opt/maja/work/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS/in/S2__TEST_AUX_REFDE2_TT36UVG_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2018-11-13T15:27:40.330516 localhost.localdomain maja-processing 01.00 [000000011102] [I] Starting DTM reading (nb resolutions=2) ...
2018-11-13T15:27:44.014864 localhost.localdomain maja-processing 01.00 [000000011102] [I] In the input data directory, there is(are) 0 meteo filename (EXO_METDTA).
2018-11-13T15:27:44.015555 localhost.localdomain maja-processing 01.00 [000000011102] [I] Starting reading and checking input data for all satellites activated...
2018-11-13T15:27:44.018109 localhost.localdomain maja-processing 01.00 [000000011102] [I] Initialize the Plugin instance detected with the Plugin Name 'SENTINEL2' ...
/opt/maja/repL2/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//.maja-working-directory/MAJAUserConfig_SENTINEL2.xml validates
2018-11-13T15:27:44.031378 localhost.localdomain maja-processing 01.00 [000000011102] [I] The number of streaming lines of buffer for reading/writing images is '800'.
2018-11-13T15:27:44.032011 localhost.localdomain maja-processing 01.00 [000000011102] [I] Using default Admin configuration camera file.
/opt/maja/repL2/T36UVG/T36UVG/MAJA_3_1_S2AS2B_CAMS//.maja-working-directory/MAJAAdminConfig_SENTINEL2.xml validates
2018-11-13T15:27:44.045957 localhost.localdomain maja-processing 01.00 [000000011102] [I] Starting reading data for the satellite 'SENTINEL-2A' and the plugin used is 'SENTINEL2' ...
2018-11-13T15:27:44.046575 localhost.localdomain maja-processing 01.00 [000000011102] [P] Starting GIP_L2SMAC reading ...
2018-11-13T15:27:44.053770 localhost.localdomain maja-processing 01.00 [000000011102] [P] Starting GIP_L2TOCR reading ...
2018-11-13T15:27:44.063078 localhost.localdomain maja-processing 01.00 [000000011102] [E] vns::Data::ERROR: DataApplicationHandler(0x22e2780): DataApplicationHandler::GetGippFilename: Most of one input filenames have been found for the File_Type 'GIP_L2TOCR' and for the satellite 'SENTINEL-2A'.
The input directory contains more than one input data or the JobOrder doesn't respect the format !  [vnsDataApplicationHandler.cxx:GetGippFilename:1415]
2018-11-13T15:27:44.064101 localhost.localdomain maja-processing 01.00 [000000011102] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:127]
2018-11-13T15:27:44.064699 localhost.localdomain maja-processing 01.00 [000000011102] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x22e2780): DataApplicationHandler::GetGippFilename: Most of one input filenames have been found for the File_Type 'GIP_L2TOCR' and for the satellite 'SENTINEL-2A'.
The input directory contains more than one input data or the JobOrder doesn't respect the format ! [vnsDataApplicationHandler.cxx:1415] [MAJA Data Exception]  [vnsMajaMainProcessor.cxx:main:127]
2018-11-13T15:27:44.065379 localhost.localdomain maja-processing 01.00 [000000011102] [E]   [vnsMajaMainProcessor.cxx:main:127]
2018-11-13T15:27:44.066194 localhost.localdomain maja-processing 01.00 [000000011102] [E] ******************************************************************************************************  [vnsMajaMainProcessor.cxx:main:127]

And my directory structure with files: opt https://cloud.mail.ru/public/CK3Q/tYzTBk1zQ For convenience, I archived folders.

Best regards, Igor

S2B_MSIL1C_20180503T085559_N0206_R007_T36UVG_20180503T115055.SAFE.log

olivierhagolle commented 5 years ago

Hi Igor, it seems you are using GIPP parameters for MAJA 3_1, while running maja 1_0. You should use parameters for MAJA 1_0.

For instance, those provided here: http://tully.ups-tlse.fr/olivier/gipp/tree/master/GIPP_MAJA_1_0_S2AS2B_NATIF

MAJA 3_1 still has a few bugs so we do not advise using it yet and only distribute it on demand, bus as I said, it is still bugged. Best regards

bim2010 commented 5 years ago

Thanks a lot, Igor

bim2010 commented 5 years ago

+