search

CNES / MAJA

Level-2A processor used for atmospheric correction and cloud-detection. The active repository is the one below, this one is kept to leave access to the older issues.
https://gitlab.orfeo-toolbox.org/maja/maja
Apache License 2.0
137 stars 25 forks source link

MAJA 4 Errors #78

Closed jprankl closed 3 years ago

jprankl commented 3 years ago

Dear Oliver,
Dear Peter,

thanks for the new Maja release. After some dependency issues (already mentioned in #73 ) I was able to compile it. I can also run startmaja and it produces L2A products, but it always ends with an error. Same behavior of the binary version and my own compiled version. Did I mismatch some parameters or do I have some bad configuration?

I am running Maja on Ubuntu 20.04 in a docker container.

I started from scratch with:

/opt/maja/MAJA-4.2.1/bin/startmaja -f folders.txt -t 33UWP -s test_site -d 2020-01-01 -e 2020-07-01 --type_dem srtm -v --skip_errors --cams

Maja detected the following configuration:

2020-11-22 11:41:10,251 [INFO ] =============This is Start_Maja v4.2.0==============
2020-11-22 11:41:10,251 [DEBUG] Checking config file: folders.txt
2020-11-22 11:41:10,258 [DEBUG] Config file parsed without errors.
2020-11-22 11:41:10,258 [DEBUG] Setting site and tile
2020-11-22 11:41:10,439 [WARNI] 33UWP
2020-11-22 11:41:10,615 [DEBUG] Found site test_site and tile 33UWP
2020-11-22 11:41:10,615 [INFO ] Detecting input products...
2020-11-22 11:41:10,699 [INFO ] 152 L1C product(s) detected for site test_site and tile 33UWP in /data/Sen4Agri/sentinel_products_L1C-test/test_site/T33UWP
2020-11-22 11:41:10,699 [WARNI] No L2A products detected for site test_site and tile 33UWP in /data/Sen4Agri/sentinel_products_L2A_MAJA-test3/test_site/33UWP
Enforce tm. HACK jp, 21.11.2020
2020-11-22 11:41:10,700 [INFO ] Searching for CAMS
2020-11-22 11:41:11,183 [INFO ] ...found 86 CAMS files
2020-11-22 11:41:11,183 [INFO ] Checking GIPP files
2020-11-22 11:41:11,183 [INFO ] Setting up GIPP folder: /data/Sen4Agri/resources_maja4/gipp_maja
2020-11-22 11:41:12,361 [INFO ] Searching for DTM
2020-11-22 11:41:12,364 [INFO ] Found DTM: /data/Sen4Agri/resources_maja4/dtm/S2__TEST_AUX_REFDE2_33UWP_1001.HDR
2020-11-22 11:41:13,668 [INFO ] GIPP Creation succeeded for SENTINEL2_TM_CAMS
2020-11-22 11:41:14,067 [INFO ] 91 workplan(s) successfully created:
               DATE |       TILE |     MODE |                                                             L1-PRODUCT | ADDITIONAL INFO
2020-01-01 10:03:19 |      33UWP | BACKWARD |      S2B_MSIL1C_20200101T100319_N0208_R122_T33UWP_20200101T110654.SAFE | Backward of 8 products with CAMS
2020-01-03 09:54:01 |      33UWP |  NOMINAL |      S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.SAFE | L2 from previous with CAMS
2020-01-06 10:04:01 |      33UWP |  NOMINAL |      S2A_MSIL1C_20200106T100401_N0208_R122_T33UWP_20200106T103423.SAFE | L2 from previous with CAMS
2020-01-08 09:53:09 |      33UWP |  NOMINAL |      S2B_MSIL1C_20200108T095309_N0208_R079_T33UWP_20200108T102330.SAFE | L2 from previous with CAMS
2020-01-09 10:14:01 |      33UWP |  NOMINAL |      S2A_MSIL1C_20200109T101401_N0208_R022_T33UWP_20200109T104556.SAFE | L2 from previous with CAMS
2020-01-11 10:02:59 |      33UWP |  NOMINAL |      S2B_MSIL1C_20200111T100259_N0208_R122_T33UWP_20200111T120153.SAFE | L2 from previous with CAMS
...

The following error comes for the first product:

...
2020-11-22 11:46:51 [4092] INFO MAJA 4.2 process_one_product:830 Starting Scientifics processing done.
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:831 L1 Product : Date 2020-01-14 10:17:04.653261
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:832        *  p_finalize_backward              = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:833        *  l_WriteL2Products                = True
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:834        *  p_WriteL2ProductToL2Resolution   = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:835        *  p_write_temporary_l2_product     = True
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:836        *  p_WriteHeaders                   = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:837        *  l_DealingLTC                     = True
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:838        *  l_IgnoreCurrentLTC               = False
2020-11-22 11:46:51 [4092] DEBUG MAJA 4.2 process_one_product:839   ->  self._productIsValid                     = False
Traceback (most recent call last):
  File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
    main()
  File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 81, in main
    lprocessor.scientific_processing()
  File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/processor/l2_backward_processor.py", line 221, in scientific_processing
    l_GIPPL2COMMHandler)
  File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/processor/l2_backward_processor.py", line 539, in single_product_processing
    checking_conditional_clouds)
  File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/processor/l2_processor.py", line 840, in process_one_product
    LOGGER.debug("  ->  p_checking_conditional_clouds        = " + str(p_checking_conditional_clouds[0]))
TypeError: 'bool' object is not subscriptable
2020-11-22 11:46:52 [4092] DEBUG MAJA 4.2 __del__:75 Destructing : Stats
...

and for the following products:

...
2020-11-22 13:06:21 [14002] DEBUG MAJA 4.2 __del__:79 RAM : 4388 : 4388 : 0 MB
2020-11-22 13:06:21 [14002] INFO MAJA 4.2 post_processing:542 Starting Postprocessing
2020-11-22 13:06:21 [14002] INFO MAJA 4.2 main:85 Cleaning Temporary datas ...
Traceback (most recent call last):
  File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 100, in <module>
    main()
  File "/opt/maja/MAJA-4.2.1/lib/python/orchestrator/launcher/maja.py", line 89, in main
    ["lsof", "+D", str(app_handler.get_working_directory())])
  File "/opt/maja/MAJA-4.2.1/lib/python3.7/subprocess.py", line 323, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/opt/maja/MAJA-4.2.1/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/opt/maja/MAJA-4.2.1/lib/python3.7/subprocess.py", line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'lsof': 'lsof'
2020-11-22 13:06:22 [14002] DEBUG MAJA 4.2 free_otb_app:66 Removing anonymous
...

Some detailed log files: log_verbose_cmd_20201122.txt.tar.gz S2B_MSIL1C_20200101T100319_N0208_R122_T33UWP_20200101T110654.log.tar.gz S2A_MSIL1C_20200103T095401_N0208_R079_T33UWP_20200103T102451.log.tar.gz

Thanks for your help.
Johann

c00kiemon5ter commented 3 years ago

Hello, I bumped into the same error about lsof. It seems lsof is a dependency for MAJA. I just installed it ($ yum install -y lsof) and now restarting the task.

EDIT: seems to be working nicely, now

jprankl commented 3 years ago

Ahh, yes, thanks. I installed several dependencies, but that I missed. The second error (I posted above) is gone, the first one still appears, but maybe that's because of initialization on a totally cloudy image. So I will close it. Thanks again!

c00kiemon5ter commented 3 years ago

[offtopic] @jprankl is it possible to share your Dockerfile?

jprankl commented 3 years ago

Dear @c00kiemon5ter , I did not write a proper Dockerfile, but only used an existing Ubuntu 20.04 container and installed Maja manually. Would be nice to have a Dockerfile:-) For now, I can write down the essential steps I did?

c00kiemon5ter commented 3 years ago

Yes, that would be nice! Given that, I can try to come up with a proper Dockerfile ;)

jprankl commented 3 years ago

@c00kiemon5ter I created a separate issue #80 with the steps I did. I hope it helps.

rohitshetty commented 3 years ago

Hey @c00kiemon5ter in case you are still interested, here is a working Dockerfile: https://github.com/rohitshetty/maja-docker