search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

LANDSAT8 GIPP files #71

Closed lnicola closed 3 years ago

lnicola commented 3 years ago

Hello,

We're trying to run MAJA 4.2.2 and we've ran into a couple of issues:

There might be other problems too, so we'd appreciate it if you have a working version on hand.

petket-5 commented 3 years ago

Hi,

The xsd schemas are not up to date, the GIPPs are tested against a different xsd than what you used. The GIPPs have been tested for Landsat-8 native (natif) and muscate type products and worked in all cases.

Can you please post the error message you received when running maja?

Kind regards, Peter

lnicola commented 3 years ago

Sorry if I wasn't clear, my list of issues was based on actual errors in MAJA 4.2.1. E.g., CLD_Viewing_Direction_Band_Code is indeed missing from the L8 schema:

$ grep -Rn CLD_Viewing_Direction_Band_Code /opt/maja/4.2.1
/opt/maja/4.2.1/share/schemas/SENTINEL2_MUSCATE/GIP_L2COMM_L2Commons.xsd:385:                       <xs:element name="CLD_Viewing_Direction_Band_Code"
/opt/maja/4.2.1/share/schemas/SENTINEL2_TM/GIP_L2COMM_L2Commons.xsd:385:                        <xs:element name="CLD_Viewing_Direction_Band_Code"
/opt/maja/4.2.1/share/schemas/SENTINEL2/GIP_L2COMM_L2Commons.xsd:385:                       <xs:element name="CLD_Viewing_Direction_Band_Code"
/opt/maja/4.2.1/lib/python/orchestrator/common/earth_explorer/gipp_l2_comm_earth_explorer_xml_file_handler.py:149:                                   "CLD_Viewing_Direction_Band_Code/text()",
Binary file /opt/maja/4.2.1/lib/python/orchestrator/common/earth_explorer/__pycache__/gipp_l2_comm_earth_explorer_xml_file_handler.cpython-37.pyc matches
grep: /opt/maja/4.2.1/lib/vnsInstallDataCxx: No such file or directory

, while it's present in the GIPP file: http://osr-cesbio.ups-tlse.fr/gitlab_cesbio/kettigp/maja-gipp/blob/develop/LANDSAT8_NATIF/L8_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF#L187.

Similarly, RCR_HOT is required by the schema:

grep -Rn RCR_HOT /opt/maja/4.2.1                    
/opt/maja/4.2.1/share/schemas/LANDSAT_MUSCATE/L7_GIP_L2COMM_L2Commons.xsd:345:          <xs:element name="RCR_HOT" type="xs:decimal"/>
/opt/maja/4.2.1/share/schemas/LANDSAT_MUSCATE/L5_GIP_L2COMM_L2Commons.xsd:345:          <xs:element name="RCR_HOT" type="xs:decimal"/>
/opt/maja/4.2.1/share/schemas/SENTINEL2_TM/GIP_L2COMM_L2Commons.xsd:425:                        <xs:element name="RCR_HOT" type="xs:decimal"/>
/opt/maja/4.2.1/share/schemas/VENUS/VE-ICD-PRO-437-CNES_0111/schemas/Venus/VE_GIP_L2COMM_L2Commons.xsd:358:         <xs:element name="RCR_HOT" type="xs:decimal"/>
/opt/maja/4.2.1/share/schemas/LANDSAT8/GIP_L2COMM_L2Commons.xsd:357:            <xs:element name="RCR_HOT" type="xs:decimal"/>
grep: /opt/maja/4.2.1/lib/vnsInstallDataCxx: No such file or directory

but it's not present in http://osr-cesbio.ups-tlse.fr/gitlab_cesbio/kettigp/maja-gipp/blob/develop/LANDSAT8_NATIF/L8_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF.

As an example of error (for the first problem above):

2020-10-07 16:42:33 [29624] INFO MAJA 4.2 log_running_task:60 Starting task: xmllint --noout /mnt/archive/demmaccs_tmp/tmpSaZN1I/192026                  /L8_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF --schema /opt/maja/4.2.1/share/schemas/LANDSAT8/GIP_L2SMAC_L2SimplifiedMethodforAtmosphericCorrection.xsd
2020-10-07 16:42:33 [29624] ERROR MAJA 4.2 launch_command:118 b'/mnt/archive/demmaccs_tmp/tmpSaZN1I/192026/L8_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF validates\n'
2020-10-07 16:42:33 [29624] INFO MAJA 4.2 log_running_task:62 Finished task xmllint --noout /mnt/archive/demmaccs_tmp/tmpSaZN1I/192026/L8_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF --schema /opt/maja/4.2.1/share/schemas/LANDSAT8/GIP_L2SMAC_L2SimplifiedMethodforAtmosphericCorrection.xsd
2020-10-07 16:42:33 [29624] INFO MAJA 4.2 __init__:72 Writed new gipp smac to /mnt/archive/demmaccs_tmp/tmpSaZN1I/maccs_192026/.maja-working-directory/L8_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
2020-10-07 16:42:33 [29624] INFO MAJA 4.2 log_running_task:60 Starting task: xmllint --noout /mnt/archive/demmaccs_tmp/tmpSaZN1I/192026/L8_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF --schema /opt/maja/4.2.1/share/schemas/LANDSAT8/GIP_L2COMM_L2Commons.xsd
2020-10-07 16:42:33 [29624] ERROR MAJA 4.2 launch_command:118 b"/mnt/archive/demmaccs_tmp/tmpSaZN1I/192026/L8_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF:187: element CLD_Viewing_Direction_Band_Code: Schemas validity error : Element '{http://eop-cfi.esa.int/CFI}CLD_Viewing_Direction_Band_Code': This element is not expected. Expected is ( {http://eop-cfi.esa.int/CFI}Cloud_Blue_Reflectance_Threshold ).\n"
2020-10-07 16:42:33 [29624] ERROR MAJA 4.2 launch_command:118 b'/mnt/archive/demmaccs_tmp/tmpSaZN1I/192026/L8_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF fails to validate\n'
2020-10-07 16:42:33 [29624] INFO MAJA 4.2 log_running_task:62 Finished task xmllint --noout /mnt/archive/demmaccs_tmp/tmpSaZN1I/192026/L8_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF --schema /opt/maja/4.2.1/share/schemas/LANDSAT8/GIP_L2COMM_L2Commons.xsd
Traceback (most recent call last):
  File "/opt/maja/4.2.1/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
    main()
  File "/opt/maja/4.2.1/lib/python/orchestrator/launcher/maja.py", line 80, in main
    lprocessor.pre_processing()
  File "/opt/maja/4.2.1/lib/python/orchestrator/processor/l2_init_processor.py", line 81, in pre_processing
    super(L2InitProcessor, self).pre_processing()
  File "/opt/maja/4.2.1/lib/python/orchestrator/processor/l2_processor.py", line 195, in pre_processing
    xml_tools.check_xml(gip_l2comm_filename, l_plugin_schema_dir)
  File "/opt/maja/4.2.1/lib/python/orchestrator/common/xml_tools.py", line 581, in check_xml
    "The xml file <" + xml_file + "> is not conform with its schema <" + lFullXsdFilename + "> ! ")
orchestrator.common.maja_exceptions.MajaIOError: The xml file </mnt/archive/demmaccs_tmp/tmpSaZN1I/192026/L8_TEST_GIP_L2COMM_L_ALLSITES_00001_20190626_21000101.EEF> is not conform with its schema </opt/maja/4.2.1/share/schemas/LANDSAT8/GIP_L2COMM_L2Commons.xsd> !
petket-5 commented 3 years ago

As I said the XSD's are yet to be updated whereas the actual GIPPs do work. Which userconf files are you using for your run? The ones provided with Maja 4.2.1 or your own?

Kind regards, Peter

lnicola commented 3 years ago

I'll have to check, we might been using older ones. Can you recommend a certain version? I see that e.g. /opt/maja/4.2.1/etc/conf/user/MAJAUserConfigSystem.xml has CheckXMLFilesWithSchema enabled, while it's disabled in /opt/maja/4.2.1/lib/python/StartMaja/userconf/MAJAUserConfigSystem.xml.

I guess the StartMaja ones would be a better starting point?

petket-5 commented 3 years ago

Yes, when using startmaja the latter contains the whole userconf which conviently deactivates the check :) More importantly you might be missing the "RAM" parameter though, which is used to limit globally the amount of memory used for a single run of maja. Can you please retry with the updated userconf?

Kind regards, Peter

lnicola commented 3 years ago

Thank you, we'll try to compare the three versions and probably use the StartMaja ones. We don't use that script, though.

Memory usage hasn't been a problem as far as I know, though 4.2.1 seems a fair bit slower than 3.2.2. Of course, that's unrelated.

petket-5 commented 3 years ago

Are you running at 120m or 240m? Landsat-8 especially received a big performance boost but of course that depends on your machine. I would be interested to discuss this in a separate issue or by mail given that you run Maja on a fairly advanced level :)

Kind regards, Peter

lnicola commented 3 years ago

240m, IIRC. The difference is about 45 minutes for a Sentinel-2 product vs. about 40 minutes in 3.2.2.

Anyway, thanks for the help. I'll close this issue. I'm not going to file a new one to update the XSDs, but it might still be a good idea.

Regards, Laurențiu

lnicola commented 3 years ago

Just one more thing. We tried again with the native GIPP files and got what looks like a MUSCATE product structure:

DATA
MASKS
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_ATB_XS.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_FRE_B1.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_FRE_B2.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_FRE_B3.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_FRE_B4.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_FRE_B5.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_FRE_B6.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_FRE_B7.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_MTD_ALL.xml
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_QKL_ALL.jpg
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_SRE_B1.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_SRE_B2.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_SRE_B3.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_SRE_B4.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_SRE_B5.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_SRE_B6.tif
LANDSAT8-OLITIRS-XS_20191023-095758-022_L2A_192-026_C_V1-0_SRE_B7.tif

We thought we should also run it with the MUSCATE config and ran again into some of the issues reported above:

4.2 __init__:72 Writed new gipp smac to /mnt/archive/demmaccs_tmp/tmpb15oRi/maccs_192026/.maja-working-directory/L8_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEFTraceback (most recent call last):
  File "/opt/maja/4.2.1/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
    main()
  File "/opt/maja/4.2.1/lib/python/orchestrator/launcher/maja.py", line 80, in main
    lprocessor.pre_processing()
  File "/opt/maja/4.2.1/lib/python/orchestrator/processor/l2_init_processor.py", line 81, in pre_processing
    super(L2InitProcessor, self).pre_processing()
  File "/opt/maja/4.2.1/lib/python/orchestrator/processor/l2_processor.py", line 188, in pre_processing
    "GIP_L2COMM", l_sat)
  File "/opt/maja/4.2.1/lib/python/orchestrator/common/gipp_utils.py", line 51, in get_gipp_filename_with_mission
    raise MajaDataException("No GIPP of type " + file_type + " has been detected for the Mission <" +mission+">  in the input directory!")
orchestrator.common.maja_exceptions.MajaDataException: No GIPP of type GIP_L2COMM has been detected for the Mission <LANDSAT_8>  in the input directory!
4.2 __init__:72 Writed new gipp smac to /mnt/archive/demmaccs_tmp/tmpb15oRi/maccs_192026/.maja-working-directory/L8_TEST_GIP_L2SMAC_L_ALLSITES_00001_20190626_21000101.EEF
2020-10-23 12:10:22 [17538] INFO MAJA 4.2 add_lut_for_model:77 Starting GIP_L2DIRT reading ...
Traceback (most recent call last):
  File "/opt/maja/4.2.1/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
    main()
  File "/opt/maja/4.2.1/lib/python/orchestrator/launcher/maja.py", line 80, in main
    lprocessor.pre_processing()
  File "/opt/maja/4.2.1/lib/python/orchestrator/processor/l2_init_processor.py", line 81, in pre_processing
    super(L2InitProcessor, self).pre_processing()
  File "/opt/maja/4.2.1/lib/python/orchestrator/processor/l2_processor.py", line 225, in pre_processing
    l_plugin_schema_dir)
  File "/opt/maja/4.2.1/lib/python/orchestrator/common/interfaces/maja_athmospheric_lut_handler.py", line 82, in add_lut_for_model
    raise MajaDataException("Missing input L2DIRT gipp for model " + p_model)
orchestrator.common.maja_exceptions.MajaDataException: Missing input L2DIRT gipp for model CONTINEN

Thanks.