Open lefebvre-antoine opened 4 years ago
Here is the log file:
2019-11-07T13:54:47.616138 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] MAJA Software Version : 3.3.0
2019-11-07T13:54:47.618618 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Overloads the conf value in the command line by '/home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///userconf'.
2019-11-07T13:54:47.635475 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] -------------------------------------------------------------------------
2019-11-07T13:54:47.635682 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Configuration User System parameters:
2019-11-07T13:54:47.635815 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * L2Note : L2 note
2019-11-07T13:54:47.635923 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * L3Note : L3 note
2019-11-07T13:54:47.636022 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * CheckToolsNote : Checktool note
2019-11-07T13:54:47.636152 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * NbThreads : 8
2019-11-07T13:54:47.636292 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * EnableCleaningTemporaryDirectory : true
2019-11-07T13:54:47.639620 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * EnableCachingConvertReflectanceData : true
2019-11-07T13:54:47.642853 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * EnableCleaningCachingDirectoryBeforeProcessing : false
2019-11-07T13:54:47.646028 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * EnableCleaningCachingDirectoryAfterProcessing : true
2019-11-07T13:54:47.649149 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * EnableL2ResolutionProcessing : true
2019-11-07T13:54:47.652398 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * PreProcessingCleanInputZipFiles : false
2019-11-07T13:54:47.655560 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * PostProcessingZipFiles : false
2019-11-07T13:54:47.658677 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * PostProcessingCleanFiles : false
2019-11-07T13:54:47.661999 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * CheckXMLFilesWithSchema : false
2019-11-07T13:54:47.665447 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] -------------------------------------------------------------------------
2019-11-07T13:54:47.668656 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Using default Admin configuration system file.
2019-11-07T13:54:47.682139 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] -------------------------------------------------------------------------
2019-11-07T13:54:47.682303 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Configuration Admin System parameters:
2019-11-07T13:54:47.682507 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * System : MAJA
2019-11-07T13:54:47.685863 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * Tolerances / Footprint / Coordinate : 1e-06
2019-11-07T13:54:47.689064 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * '' '' / Direction : 1e-06
2019-11-07T13:54:47.692347 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * EnablePerformanceMeasureAlgorithms : false
2019-11-07T13:54:47.695453 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] -------------------------------------------------------------------------
2019-11-07T13:54:47.698906 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] ******************************************************************************
2019-11-07T13:54:47.701957 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Application information:
2019-11-07T13:54:47.705395 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * Processor Name : MAJA_L2_NOMINAL_CHAIN
2019-11-07T13:54:47.708493 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * Test mode : false
2019-11-07T13:54:47.711825 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * UserConfigSystemFilename : /home/kermap/Documents/tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///.maja-working-directory/MAJAUserConfigSystem.xml
2019-11-07T13:54:47.718499 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * AdminConfigSystemFileName : /home/kermap/Documents/tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///.maja-working-directory/MAJAAdminConfigSystem.xml
2019-11-07T13:54:47.718672 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] * Specifics command line parameters:
2019-11-07T13:54:47.718986 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] - TileId: 36RXV
2019-11-07T13:54:47.719189 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] ******************************************************************************
2019-11-07T13:54:47.719402 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The process uses 8 thread(s). Note: possible to use of maximum 128 threads.
2019-11-07T13:54:47.724409 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting L2NominalProcessor::PreProcessing() ...
2019-11-07T13:54:48.049392 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The L1 product found is </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///in/S2A_MSIL1C_20180526T081601_N0206_R121_T36RXV_20180526T120617.SAFE/MTD_MSIL1C.xml>.
2019-11-07T13:54:48.149186 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Initialize the Plugin instance detected with the Plugin Name 'SENTINEL2_TM' ...
2019-11-07T13:54:48.157590 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The number of streaming lines of buffer for reading/writing images is '800'.
2019-11-07T13:54:48.157753 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Using default Admin configuration camera file.
2019-11-07T13:54:48.169026 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] -----------------------------------------------------------------------------------------------
2019-11-07T13:54:48.169191 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting L2Processor::PreProcessing() ...
2019-11-07T13:54:48.169431 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] List of the 1 satellite(s) activated :
2019-11-07T13:54:48.172757 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] - SENTINEL-2A ('common' satellite SENTINEL-2_ and plugin SENTINEL2_TM)
2019-11-07T13:54:48.175998 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting reading and checking input data common for each plugin activated...
2019-11-07T13:54:48.179441 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] According to the input data, there is(are) 1 plugin(s) activated:
2019-11-07T13:54:48.182661 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] - SENTINEL2_TM
2019-11-07T13:54:48.223973 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting DEM reading data for the 'unique' (common) satellite 'SENTINEL-2_' ...
2019-11-07T13:54:48.224221 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] UnCompress the DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2__TEST_AUX_REFDE2_T36RXV_0001.DBL> ...
2019-11-07T13:54:48.224502 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2__TEST_AUX_REFDE2_T36RXV_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-11-07T13:54:48.231844 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] l_NumberOfL2Resolution : 2
2019-11-07T13:54:48.232060 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting DTM reading ...
2019-11-07T13:54:48.584542 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] UnCompress the DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2__TEST_AUX_REFDE2_T36RXV_0001.DBL> ...
2019-11-07T13:54:48.584925 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2__TEST_AUX_REFDE2_T36RXV_0001.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-11-07T13:54:48.588524 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting DTM reading (nb resolutions=2) ...
2019-11-07T13:54:56.012467 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] In the input data directory, there is(are) 0 meteo filename (EXO_METDTA).
2019-11-07T13:54:56.012987 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] In the input data directory, there is(are) 0 cams filename (EXO_CAMS).
2019-11-07T13:54:56.016971 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting reading and checking input data for all satellites activated...
2019-11-07T13:54:56.020690 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Initialize the Plugin instance detected with the Plugin Name 'SENTINEL2_TM' ...
2019-11-07T13:54:56.027507 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The number of streaming lines of buffer for reading/writing images is '800'.
2019-11-07T13:54:56.027686 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Using default Admin configuration camera file.
2019-11-07T13:54:56.034619 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting reading data for the satellite 'SENTINEL-2A' and the plugin used is 'SENTINEL2_TM' ...
2019-11-07T13:54:56.043862 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2SMAC reading ...
2019-11-07T13:54:56.053755 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2TOCR reading ...
2019-11-07T13:54:56.058723 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] UnCompress the DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL> ...
2019-11-07T13:54:56.058915 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2TOCR_L_CONTINEN_00001_20190626_21000101.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-11-07T13:54:56.071827 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2WATV reading ...
2019-11-07T13:54:56.079697 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] UnCompress the DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL> ...
2019-11-07T13:54:56.079984 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2WATV_L_CONTINEN_00001_20190626_21000101.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-11-07T13:54:56.080448 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Allocating LookUpTable image dimension = 3
2019-11-07T13:54:56.090590 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2DIRT uncompressing ...
2019-11-07T13:54:56.095333 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] UnCompress the DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL> ...
2019-11-07T13:54:56.095541 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2DIRT_L_CONTINEN_00001_20190626_21000101.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-11-07T13:54:56.095793 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2DIFT uncompressing ...
2019-11-07T13:54:56.107068 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] UnCompress the DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL> ...
2019-11-07T13:54:56.107355 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2DIFT_L_CONTINEN_00001_20190626_21000101.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-11-07T13:54:56.107619 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2ALBD uncompressing ...
2019-11-07T13:54:56.118799 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] UnCompress the DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL> ...
2019-11-07T13:54:56.119081 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The DBL product </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx/in/S2A_TEST_GIP_L2ALBD_L_CONTINEN_00001_20190626_21000101.DBL> is already uncompressed (.DBL.DIR directory already exists).
2019-11-07T13:54:56.119342 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2DIRT reading ...
2019-11-07T13:54:56.145330 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2DIFT reading ...
2019-11-07T13:54:56.153439 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2ALBD reading ...
2019-11-07T13:54:56.160807 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting GIP_L2TOCR reading ...
2019-11-07T13:54:56.377247 gagarine maja-processing-3.3.0 3.3 [000000011031] [P] Starting L2Processor::PreProcessing() done.
2019-11-07T13:54:56.377534 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] L2NominalProcessor::PreProcessing() done.
2019-11-07T13:54:56.377866 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Starting L2NominalProcessor::ScientificProcessing() ...
2019-11-07T13:54:56.388351 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] The GIP_L2SITE file detected for the satellite 'SENTINEL-2_' is </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx//in/S2__TEST_GIP_L2SITE_S_ALLSITES_00001_20190626_21000101.EEF>.
2019-11-07T13:54:56.399961 gagarine maja-processing-3.3.0 3.3 [000000011031] [I] Start Sentinel2 L1 ImageFileReader ...
l_NbBandTOA : 13
2019-11-07T13:57:35.536636 gagarine maja-processing-3.3.0 3.3 [000000011031] [E] vns::Data::ERROR: : Impossible to detect a L2 product in the directory </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///in>. [vnsL2ImageFileReaderFactory.txx:GetListOfL2Products:211]
2019-11-07T13:57:35.592794 gagarine maja-processing-3.3.0 3.3 [000000011031] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
2019-11-07T13:57:35.593180 gagarine maja-processing-3.3.0 3.3 [000000011031] [E] ScientificProcessing: vns::Data::ERROR: : Impossible to detect a L2 product in the directory </home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV/36RXV/GIPP_S2AS2B_xxx///in>. [vnsL2ImageFileReaderFactory.txx:211] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:130]
2019-11-07T13:57:35.599231 gagarine maja-processing-3.3.0 3.3 [000000011031] [E] [vnsMajaMainProcessor.cxx:main:130]
2019-11-07T13:57:35.603353 gagarine maja-processing-3.3.0 3.3 [000000011031] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:130]
Hi Antoine, I have had troubles to reproduce this error in general, and was of little help to those who encountered it. I can try to give it a try : could you give me the command line you used ? I am quite slow these days, meetings all the time, so don't expect a quick answer Olivier
Hi Olivier, Thank you for your quick reply.
The command line is very basic :
./start_maja.py -f folders.txt -g GIPP_S2AS2B_xxx/ -l LUT_S2AS2B_xxx/ -t $1 -s $1 -d 20180101 -e 20190101
One thing that I modified in start_maja.py is to locate DTM files on a network drive. I modified the default 'repDtm'
We process 1 year in a row so there is around 70 to 100 images in the L1 folder.
I'm wondering if it may be due to an hardware or software issue...
the folder.txt is organized as follow:
repCode=/home/kermap/Documents/Start-MAJA-master/ <-- SSD local drive
repWork=/home/kermap/Documents/Start-MAJA-master/MAJAtmp **<-- SSD local drive
repL1 =/home/kermap/SENTINEL-2/Antoine/L1C/tmp <-- mounted network drive
repL2 =/home/kermap/Documents/tmp <-- SSD local drive
repMaja=/home/kermap/Documents/Start-MAJA-master/MAJA-3.3.0-TM/bin/maja <-- SSD local drive
repCAMS=/home/kermap/Documents/Start-MAJA-master/CAMS/ <-- SSD local drive_**
Or it may be due to a corrupted jp2 files (but it seems not), a mounted network folder (but it seems not), or the linux distribution (but hard to believe... BTW, are you working with Red Hat at CNES ?)
Thank you very much!
@alefebvre3
Try to delete /home/kermap/Documents/Start-MAJA-master/MAJA_tmp/36RXV
Dear Olivier
It seems that we experienced a kind of "ramdom" error using Start-MAJA. We used to process a full year time series but for some reasons, the process is sometimes stopped with the following message:
This issue makes me think that it could be a similar problem than this post : https://github.com/olivierhagolle/Start_maja/issues/29
Have you any idea ? Our server is running with ubuntu 18.04 and the temporary folder is on the local drive.
Thank you very much! Best regards Antoine
Below the last lines before the error