search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

Error: The MUSCATE product is not valid according to JPI! #15

Closed spatialtrail closed 5 years ago

spatialtrail commented 5 years ago

Hi,

Recently installed and trying to test MAJA but encountering error processing some dates.

My setup: OS = Ubuntu 18.04 CPU = 8 cores RAM = 12 GB maja = 3.3.0 TM start_maja = latest master LUT = LUT_MAJA_S2A_S2B_CAMS_H2ONew_20190411.tgz S2L1C = tile 30UYC with date range 20180601 to 20180701 and 90% cloud threshold

python /data/3.3/git/maja/start_maja.py -f /data/3.3/git/maja/folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t 30UYC -s t30UYC -d 20180601 -e 20180701 -z Error message (after attempting to re-run): 2019-05-21 09:27:36,828 - Start-Maja - INFO - Most recent processed date : 20180606 2019-05-21 09:27:36,851 - Start-Maja - INFO - => processing date 20180608 2019-05-21 09:27:38,017 - Start-Maja - INFO - Using MUSCATE L2 type 2019-05-21 09:27:38,023 - Start-Maja - INFO - previous L2 : /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM/SENTINEL2A_20180606-111256-734_L2A_T30UYC_C_V1-0 2019-05-21 09:28:09,639 - Start-Maja - INFO - ################################# 2019-05-21 09:28:09,639 - Start-Maja - INFO - ################################# 2019-05-21 09:28:09,639 - Start-Maja - INFO - processing /data/SENTINEL2/L1C/t30UYC/S2B_MSIL1C_20180608T105649_N0206_R094_T30UYC_20180608T120643.zip in nominal mode 2019-05-21 09:28:09,639 - Start-Maja - INFO - MAJA logfile: /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//S2B_MSIL1C_20180608T105649_N0206_R094_T30UYC_20180608T120643.zip.log 2019-05-21 09:28:09,639 - Start-Maja - INFO - ################################# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! L2A product /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0 not found !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 2019-05-21 09:41:48,853 - Start-Maja - INFO - => processing date 20180611 2019-05-21 09:41:49,414 - Start-Maja - INFO - Using MUSCATE L2 type 2019-05-21 09:41:49,423 - Start-Maja - INFO - previous L2 : /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0 2019-05-21 09:42:23,512 - Start-Maja - INFO - ################################# 2019-05-21 09:42:23,512 - Start-Maja - INFO - ################################# 2019-05-21 09:42:23,512 - Start-Maja - INFO - processing /data/SENTINEL2/L1C/t30UYC/S2B_MSIL1C_20180611T110619_N0206_R137_T30UYC_20180611T170311.zip in nominal mode 2019-05-21 09:42:23,512 - Start-Maja - INFO - MAJA logfile: /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//S2B_MSIL1C_20180611T110619_N0206_R137_T30UYC_20180611T170311.zip.log 2019-05-21 09:42:23,512 - Start-Maja - INFO - ################################# 2019-05-21T09:54:47.369499 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] vns::Plugin::ERROR: Sentinel2TML2ImageFileReader(0x86dee0): The MUSCATE product </data/SENTINEL2/MAJA/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//in/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0_MTD_ALL.xml> is not valid according to JPI! [vnsMuscateL2ImageFileReader.txx:MuscateDetectL2Products:368]

2019-05-21T09:54:47.408363 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] ** [vnsMajaMainProcessor.cxx:main:130]

2019-05-21T09:54:47.412682 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] ScientificProcessing: vns::Plugin::ERROR: Sentinel2TML2ImageFileReader(0x86dee0): The MUSCATE product </data/SENTINEL2/MAJA/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//in/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0/SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0_MTD_ALL.xml> is not valid according to JPI! [vnsMuscateL2ImageFileReader.txx:368] [MAJA Base Exception] [vnsMajaMainProcessor.cxx:main:130]

2019-05-21T09:54:47.416174 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] [vnsMajaMainProcessor.cxx:main:130]

2019-05-21T09:54:47.419437 proc-06-03 maja-processing-3.3.0 3.3 [000000061661] [E] ** [vnsMajaMainProcessor.cxx:main:130]

2019-05-21 09:54:47,462 - Start-Maja - INFO - ####################################### 2019-05-21 09:54:47,462 - Start-Maja - INFO - Error detected, see: /data/SENTINEL2/L2A_maja/t30UYC/30UYC/GIPP_S2_MAJA_3.3_TM//S2B_MSIL1C_20180611T110619_N0206_R137_T30UYC_20180611T170311.zip.log 2019-05-21 09:54:47,462 - Start-Maja - INFO - #######################################

I have looked at the raw S2L1C images (20180606, 20180608 and 20180611) and they all look fine. Any ideas whats going wrong?

Thanks

olivierhagolle commented 5 years ago

Hi thanks for raising the issue. We have received exactly the same issue on another channel, and we will try to reproduce and understand the error. Sorry for that, Olivier

olivierhagolle commented 5 years ago

Hi, I have not been able to reproduce the error. Re reading your message, I saw that this happened after "attempting to re-run". Is it possible that the L2A product which is used as input is corrupted for some reason (an interrupted processing while writing the product) ? In that case, the best would be to remove SENTINEL2A_20180606-111256-734_L2A_T30UYC_C_V1-0 and SENTINEL2B_20180608-110550-630_L2A_T30UYC_C_V1-0 and start again

lygutas commented 5 years ago

I have the same error with 3.2.2. vns::Plugin::ERROR: Sentinel2TML2ImageFileReader(0xe600f0): The MUSCATE product "/mnt/archive/demmaccs_tmp/tmpRpCmbO/35VMC/SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0/SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_MTD_ALL.xml" is not valid according to JPI! [vnsMuscateL2ImageFileReader.txx:MuscateDetectL2Products:368]

How can I debug what might be the issue?

Product directory : ls -al /mnt/archive/demmaccs_tmp/tmpRpCmbO/35VMC/SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0/ total 2886820 drwxr-xr-x 4 sen2agri-service sen2agri-service 4096 2019-05-29 14:37 . drwxr-xr-x 3 sen2agri-service sen2agri-service 4096 2019-05-29 14:38 .. drwxr-xr-x 3 sen2agri-service sen2agri-service 4096 2019-05-29 14:37 DATA drwxr-xr-x 2 sen2agri-service sen2agri-service 4096 2019-05-29 14:36 MASKS -rw-r--r-- 1 sen2agri-service sen2agri-service 241212436 2019-05-29 14:31 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_ATB_R1.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327916 2019-05-29 14:35 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_ATB_R2.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:35 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B11.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:35 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B12.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:30 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B2.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:30 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B3.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:31 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B4.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:35 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B5.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:35 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B6.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:35 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B7.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:35 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B8A.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:31 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_FRE_B8.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 607057 2019-05-29 14:37 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_MTD_ALL.xml -rw-r--r-- 1 sen2agri-service sen2agri-service 259075 2019-05-29 14:37 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_QKL_ALL.jpg -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:34 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B11.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:34 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B12.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:28 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B2.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:28 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B3.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:28 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B4.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:34 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B5.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:34 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B6.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:34 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B7.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 60327904 2019-05-29 14:34 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B8A.tif -rw-r--r-- 1 sen2agri-service sen2agri-service 241212426 2019-05-29 14:29 SENTINEL2A_20190122-093446-289_L2A_T35VMC_C_V1-0_SRE_B8.tif

olivierhagolle commented 5 years ago

Hi Ligutas, please send this issue to the sen2agri repository: https://github.com/Sen2Agri/Sen2Agri-System @lnicola

It seems there is a little bug in the interface between sen2Agri system and MAJA. I know they have understood it, but I do not know where it stands regarding the correction. Best regards, Olivier

lnicola commented 5 years ago

Hello,

My understanding is that we were using NOTV products (according to the metadata) as previous input for the nominal mode. We fixed this by detecting those and excluding them from any future processing.

I can't tell you a timeline for the next release, but I'll ping you here if I find anything.

olivierhagolle commented 5 years ago

@lnicola, Is this bug explained in the sen2Agri repository and on the download site ? Because, if I understand well, it will probably prevent most users from using MAJA and therefore the whole system. I see a lot of MAJA downloads being done these days, I can't say if it comes from sen2Agri, but probably a large proportion Olivier

olivierhagolle commented 5 years ago

Closed ad the bug concerns Sen2Agri