search

WorldCereal / ewoc_l8

Pre processing of L8 data (SR and TIR)
Other
0 stars 0 forks source link

Production issue related to missing B10 band #79

Closed savmickael closed 1 year ago

savmickael commented 1 year ago

As observed in #77 and #78, in some cases B10 is missing in L2 C2 l2 products. In this case we output the printf command with an unexpected number of bands (8 instead 9 or 1 instead 2). The problem is related to https://github.com/WorldCereal/ewoc_work_plan/issues/122 the input product is too cloudy and USGS cannot generate the B10 L2 band.

The best solution is to filter the wp to avoid these type of products but if it is not possible we should provide a clear exception to the processing module.

agrosu1978 commented 1 year ago

This issue appeared in the l8 re-processing ARD data from 2022-11-04. Please check the attached file. At first processing, the error is cleared defined, but at a second re-try, it seems that some temp files are still there and an exception is raised

36UWV_tile_l8_product_failure.txt

mbattude-cs commented 1 year ago

@agrosu1978 this is related to https://github.com/WorldCereal/ewoc_l8/issues/83 Did you already test the new ewoc_l8 release ?

agrosu1978 commented 1 year ago

@agrosu1978 this is related to #83 Did you already test the new ewoc_l8 release ?

Now, the file can be downloaded. I thought I used ewoc_l8:0.8.2 . This morning I re-tested one of the failed products (for sure with version 0.8.2) and now the second try fails with the same code error as the first try. Example:

[2022-11-07 08:40:20,425] [world-cereal] [DEBUG ] Try number 1 (max 2). Processing product(s): LC08_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1                            
[2022-11-07 08:40:20,425] [world-cereal] [INFO  ] Starting the processor with cmd : ewoc_generate_l8_ard -v --prod-id c728b264-5c97-4f4c-81fe-1500d4c4dfbd_22190_20220929195524 --only-tir prd_ids 36UVA LC$
8_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1                                                                                                                              
[2022-11-07 08:40:20,526] [world-cereal] [INFO  ] The timeout for process end is 1800                                                                                                                       
[2022-11-07 08:40:20,527] [world-cereal] [INFO  ] Starting the watchdog for process messages timeout. Maximum time allowed for not printing any message is 0                                                
[2022-11-07 08:40:20,527] [world-cereal] [INFO  ] Watchdog: the timeout is 0. Watching the SIGTERM signal only                                                                                              
[2022-11-07 08:40:21,361] [world-cereal] [INFO  ] [2022-11-07 08:40:21] INFO:ewoc_l8.l8_process:Following bands will be processed: ['B10', 'QA_PIXEL_TIR']                                                  
[2022-11-07 08:40:23,458] [world-cereal] [INFO  ] [2022-11-07 08:40:23] INFO:ewoc_l8.utils:SRS of 36UVA is EPSG:32636
[2022-11-07 08:40:23,459] [world-cereal] [INFO  ] [2022-11-07 08:40:23] INFO:ewoc_l8.utils:Bounding box of 36UVA is (399960, 5490240, 509760, 5600040)
[2022-11-07 08:40:23,459] [world-cereal] [INFO  ] [2022-11-07 08:40:23] INFO:ewoc_l8.l8_process:Processing B10 band
[2022-11-07 08:40:23,599] [world-cereal] [INFO  ] [2022-11-07 08:40:23] INFO:ewoc_dag.bucket.ewoc:EWoC bucket ewoc-ard is hosted on aws and functional
[2022-11-07 08:40:23,602] [world-cereal] [INFO  ] [2022-11-07 08:40:23] INFO:botocore.credentials:Found credentials in environment variables.
[2022-11-07 08:40:23,612] [world-cereal] [INFO  ] [2022-11-07 08:40:23] INFO:ewoc_l8.l8_process:['/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/025/LC08_L2SR_179025_20220817_2022082$
_02_T1/LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF', '/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/026/LC08_L2SR_179026_20220817_20220824_02_T1/LC08_L2SR_179026_20220817_20$20824_02_T1_ST_B10.TIF']
[2022-11-07 08:40:23,612] [world-cereal] [INFO  ] [2022-11-07 08:40:23] INFO:ewoc_l8.l8_process:Checking needed L8 bands in the product prefix
[2022-11-07 08:40:24,806] [world-cereal] [DEBUG ] [2022-11-07 08:40:24] ERROR:ewoc_l8.l8_process:Band LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF does not exist !
[2022-11-07 08:40:24,806] [world-cereal] [DEBUG ] [2022-11-07 08:40:24] ERROR:ewoc_l8.cli:Error during L8 ARD generation: ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR_179025_20220817_20220824_$
2_T1'] not download !
[2022-11-07 08:40:24,806] [world-cereal] [DEBUG ] [2022-11-07 08:40:24] CRITICAL:ewoc_l8.cli:Error during L8 ARD generation: No L8 ARD on 36UVA for ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR$
179025_20220817_20220824_02_T1'] !

[2022-11-07 08:40:25,636] [world-cereal] [INFO  ] The processor finished in 0:00:05.109887. Output: Return code = 4. Successfully string pattern = pattern not found
[2022-11-07 08:40:25,636] [world-cereal] [DEBUG ] output_splitted_string = ['']
[2022-11-07 08:40:25,636] [world-cereal] [ERROR ] No successfull output string. Retrying number 1
[2022-11-07 08:40:25,637] [world-cereal] [DEBUG ] Trying to connect to database, maximum allowed time to connect is 7200000 ms
[2022-11-07 08:40:25,649] [world-cereal] [DEBUG ] Connection to database succeeded in 0.012284517288208008 seconds (0.0 | 0.012286186218261719)
[2022-11-07 08:40:25,649] [world-cereal] [DEBUG ] Updating status for id 618493 to 3
[2022-11-07 08:40:25,649] [world-cereal] [DEBUG ] Updating also the retries number for id 618493 to 1
[2022-11-07 08:40:25,654] [world-cereal] [DEBUG ] Closing database connection function
[2022-11-07 08:40:25,654] [world-cereal] [DEBUG ] Database connection closed
[2022-11-07 08:40:25,654] [world-cereal] [DEBUG ] Exit from closing the database connection function
[2022-11-07 08:40:25,655] [world-cereal] [INFO  ] Sleeping for 30 seconds before giving another shot...
[2022-11-07 08:40:55,685] [world-cereal] [DEBUG ] Try number 2 (max 2). Processing product(s): LC08_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1
[2022-11-07 08:40:55,685] [world-cereal] [INFO  ] Starting the processor with cmd : ewoc_generate_l8_ard -v --prod-id c728b264-5c97-4f4c-81fe-1500d4c4dfbd_22190_20220929195524 --only-tir prd_ids 36UVA LC0
8_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1
[2022-11-07 08:40:55,758] [world-cereal] [INFO  ] The timeout for process end is 1800
[2022-11-07 08:40:55,758] [world-cereal] [INFO  ] Starting the watchdog for process messages timeout. Maximum time allowed for not printing any message is 0
[2022-11-07 08:40:55,758] [world-cereal] [INFO  ] Watchdog: the timeout is 0. Watching the SIGTERM signal only
[2022-11-07 08:40:56,582] [world-cereal] [INFO  ] [2022-11-07 08:40:56] INFO:ewoc_l8.l8_process:Following bands will be processed: ['B10', 'QA_PIXEL_TIR']
[2022-11-07 08:40:58,643] [world-cereal] [INFO  ] [2022-11-07 08:40:58] INFO:ewoc_l8.utils:SRS of 36UVA is EPSG:32636
[2022-11-07 08:40:58,644] [world-cereal] [INFO  ] [2022-11-07 08:40:58] INFO:ewoc_l8.utils:Bounding box of 36UVA is (399960, 5490240, 509760, 5600040)
[2022-11-07 08:40:58,644] [world-cereal] [INFO  ] [2022-11-07 08:40:58] INFO:ewoc_l8.l8_process:Processing B10 band
[2022-11-07 08:40:58,786] [world-cereal] [INFO  ] [2022-11-07 08:40:58] INFO:ewoc_dag.bucket.ewoc:EWoC bucket ewoc-ard is hosted on aws and functional
[2022-11-07 08:40:58,788] [world-cereal] [INFO  ] [2022-11-07 08:40:58] INFO:botocore.credentials:Found credentials in environment variables.
[2022-11-07 08:40:58,798] [world-cereal] [INFO  ] [2022-11-07 08:40:58] INFO:ewoc_l8.l8_process:['/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/025/LC08_L2SR_179025_20220817_20220824
_02_T1/LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF', '/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/026/LC08_L2SR_179026_20220817_20220824_02_T1/LC08_L2SR_179026_20220817_202
20824_02_T1_ST_B10.TIF']
[2022-11-07 08:40:58,798] [world-cereal] [INFO  ] [2022-11-07 08:40:58] INFO:ewoc_l8.l8_process:Checking needed L8 bands in the product prefix
[2022-11-07 08:40:59,991] [world-cereal] [DEBUG ] [2022-11-07 08:40:59] ERROR:ewoc_l8.l8_process:Band LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF does not exist !
[2022-11-07 08:40:59,991] [world-cereal] [DEBUG ] [2022-11-07 08:40:59] ERROR:ewoc_l8.cli:Error during L8 ARD generation: ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR_179025_20220817_20220824_0
2_T1'] not download !
[2022-11-07 08:40:59,991] [world-cereal] [DEBUG ] [2022-11-07 08:40:59] CRITICAL:ewoc_l8.cli:Error during L8 ARD generation: No L8 ARD on 36UVA for ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR_
179025_20220817_20220824_02_T1'] !

Pausing this, but let's keep an eye on it

mbattude-cs commented 1 year ago

Ok, with this version this is what we expect

agrosu1978 commented 1 year ago

The system was not ran in a nominal mode, a lot of manual starts which led to a mixture between command line launch and crontab launch. The command line used the proper version (0.8.2) while crontab used an older version