Closed savmickael closed 1 year ago
This issue appeared in the l8 re-processing ARD data from 2022-11-04. Please check the attached file. At first processing, the error is cleared defined, but at a second re-try, it seems that some temp files are still there and an exception is raised
@agrosu1978 this is related to https://github.com/WorldCereal/ewoc_l8/issues/83 Did you already test the new ewoc_l8 release ?
@agrosu1978 this is related to #83 Did you already test the new ewoc_l8 release ?
Now, the file can be downloaded. I thought I used ewoc_l8:0.8.2 . This morning I re-tested one of the failed products (for sure with version 0.8.2) and now the second try fails with the same code error as the first try. Example:
[2022-11-07 08:40:20,425] [world-cereal] [DEBUG ] Try number 1 (max 2). Processing product(s): LC08_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1
[2022-11-07 08:40:20,425] [world-cereal] [INFO ] Starting the processor with cmd : ewoc_generate_l8_ard -v --prod-id c728b264-5c97-4f4c-81fe-1500d4c4dfbd_22190_20220929195524 --only-tir prd_ids 36UVA LC$
8_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1
[2022-11-07 08:40:20,526] [world-cereal] [INFO ] The timeout for process end is 1800
[2022-11-07 08:40:20,527] [world-cereal] [INFO ] Starting the watchdog for process messages timeout. Maximum time allowed for not printing any message is 0
[2022-11-07 08:40:20,527] [world-cereal] [INFO ] Watchdog: the timeout is 0. Watching the SIGTERM signal only
[2022-11-07 08:40:21,361] [world-cereal] [INFO ] [2022-11-07 08:40:21] INFO:ewoc_l8.l8_process:Following bands will be processed: ['B10', 'QA_PIXEL_TIR']
[2022-11-07 08:40:23,458] [world-cereal] [INFO ] [2022-11-07 08:40:23] INFO:ewoc_l8.utils:SRS of 36UVA is EPSG:32636
[2022-11-07 08:40:23,459] [world-cereal] [INFO ] [2022-11-07 08:40:23] INFO:ewoc_l8.utils:Bounding box of 36UVA is (399960, 5490240, 509760, 5600040)
[2022-11-07 08:40:23,459] [world-cereal] [INFO ] [2022-11-07 08:40:23] INFO:ewoc_l8.l8_process:Processing B10 band
[2022-11-07 08:40:23,599] [world-cereal] [INFO ] [2022-11-07 08:40:23] INFO:ewoc_dag.bucket.ewoc:EWoC bucket ewoc-ard is hosted on aws and functional
[2022-11-07 08:40:23,602] [world-cereal] [INFO ] [2022-11-07 08:40:23] INFO:botocore.credentials:Found credentials in environment variables.
[2022-11-07 08:40:23,612] [world-cereal] [INFO ] [2022-11-07 08:40:23] INFO:ewoc_l8.l8_process:['/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/025/LC08_L2SR_179025_20220817_2022082$
_02_T1/LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF', '/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/026/LC08_L2SR_179026_20220817_20220824_02_T1/LC08_L2SR_179026_20220817_20$20824_02_T1_ST_B10.TIF']
[2022-11-07 08:40:23,612] [world-cereal] [INFO ] [2022-11-07 08:40:23] INFO:ewoc_l8.l8_process:Checking needed L8 bands in the product prefix
[2022-11-07 08:40:24,806] [world-cereal] [DEBUG ] [2022-11-07 08:40:24] ERROR:ewoc_l8.l8_process:Band LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF does not exist !
[2022-11-07 08:40:24,806] [world-cereal] [DEBUG ] [2022-11-07 08:40:24] ERROR:ewoc_l8.cli:Error during L8 ARD generation: ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR_179025_20220817_20220824_$
2_T1'] not download !
[2022-11-07 08:40:24,806] [world-cereal] [DEBUG ] [2022-11-07 08:40:24] CRITICAL:ewoc_l8.cli:Error during L8 ARD generation: No L8 ARD on 36UVA for ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR$
179025_20220817_20220824_02_T1'] !
[2022-11-07 08:40:25,636] [world-cereal] [INFO ] The processor finished in 0:00:05.109887. Output: Return code = 4. Successfully string pattern = pattern not found
[2022-11-07 08:40:25,636] [world-cereal] [DEBUG ] output_splitted_string = ['']
[2022-11-07 08:40:25,636] [world-cereal] [ERROR ] No successfull output string. Retrying number 1
[2022-11-07 08:40:25,637] [world-cereal] [DEBUG ] Trying to connect to database, maximum allowed time to connect is 7200000 ms
[2022-11-07 08:40:25,649] [world-cereal] [DEBUG ] Connection to database succeeded in 0.012284517288208008 seconds (0.0 | 0.012286186218261719)
[2022-11-07 08:40:25,649] [world-cereal] [DEBUG ] Updating status for id 618493 to 3
[2022-11-07 08:40:25,649] [world-cereal] [DEBUG ] Updating also the retries number for id 618493 to 1
[2022-11-07 08:40:25,654] [world-cereal] [DEBUG ] Closing database connection function
[2022-11-07 08:40:25,654] [world-cereal] [DEBUG ] Database connection closed
[2022-11-07 08:40:25,654] [world-cereal] [DEBUG ] Exit from closing the database connection function
[2022-11-07 08:40:25,655] [world-cereal] [INFO ] Sleeping for 30 seconds before giving another shot...
[2022-11-07 08:40:55,685] [world-cereal] [DEBUG ] Try number 2 (max 2). Processing product(s): LC08_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1
[2022-11-07 08:40:55,685] [world-cereal] [INFO ] Starting the processor with cmd : ewoc_generate_l8_ard -v --prod-id c728b264-5c97-4f4c-81fe-1500d4c4dfbd_22190_20220929195524 --only-tir prd_ids 36UVA LC0
8_L2SR_179026_20220817_20220824_02_T1 LC08_L2SR_179025_20220817_20220824_02_T1
[2022-11-07 08:40:55,758] [world-cereal] [INFO ] The timeout for process end is 1800
[2022-11-07 08:40:55,758] [world-cereal] [INFO ] Starting the watchdog for process messages timeout. Maximum time allowed for not printing any message is 0
[2022-11-07 08:40:55,758] [world-cereal] [INFO ] Watchdog: the timeout is 0. Watching the SIGTERM signal only
[2022-11-07 08:40:56,582] [world-cereal] [INFO ] [2022-11-07 08:40:56] INFO:ewoc_l8.l8_process:Following bands will be processed: ['B10', 'QA_PIXEL_TIR']
[2022-11-07 08:40:58,643] [world-cereal] [INFO ] [2022-11-07 08:40:58] INFO:ewoc_l8.utils:SRS of 36UVA is EPSG:32636
[2022-11-07 08:40:58,644] [world-cereal] [INFO ] [2022-11-07 08:40:58] INFO:ewoc_l8.utils:Bounding box of 36UVA is (399960, 5490240, 509760, 5600040)
[2022-11-07 08:40:58,644] [world-cereal] [INFO ] [2022-11-07 08:40:58] INFO:ewoc_l8.l8_process:Processing B10 band
[2022-11-07 08:40:58,786] [world-cereal] [INFO ] [2022-11-07 08:40:58] INFO:ewoc_dag.bucket.ewoc:EWoC bucket ewoc-ard is hosted on aws and functional
[2022-11-07 08:40:58,788] [world-cereal] [INFO ] [2022-11-07 08:40:58] INFO:botocore.credentials:Found credentials in environment variables.
[2022-11-07 08:40:58,798] [world-cereal] [INFO ] [2022-11-07 08:40:58] INFO:ewoc_l8.l8_process:['/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/025/LC08_L2SR_179025_20220817_20220824
_02_T1/LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF', '/vsis3/usgs-landsat/collection02/level-2/standard/oli-tirs/2022/179/026/LC08_L2SR_179026_20220817_20220824_02_T1/LC08_L2SR_179026_20220817_202
20824_02_T1_ST_B10.TIF']
[2022-11-07 08:40:58,798] [world-cereal] [INFO ] [2022-11-07 08:40:58] INFO:ewoc_l8.l8_process:Checking needed L8 bands in the product prefix
[2022-11-07 08:40:59,991] [world-cereal] [DEBUG ] [2022-11-07 08:40:59] ERROR:ewoc_l8.l8_process:Band LC08_L2SR_179025_20220817_20220824_02_T1_ST_B10.TIF does not exist !
[2022-11-07 08:40:59,991] [world-cereal] [DEBUG ] [2022-11-07 08:40:59] ERROR:ewoc_l8.cli:Error during L8 ARD generation: ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR_179025_20220817_20220824_0
2_T1'] not download !
[2022-11-07 08:40:59,991] [world-cereal] [DEBUG ] [2022-11-07 08:40:59] CRITICAL:ewoc_l8.cli:Error during L8 ARD generation: No L8 ARD on 36UVA for ['LC08_L2SR_179026_20220817_20220824_02_T1', 'LC08_L2SR_
179025_20220817_20220824_02_T1'] !
Pausing this, but let's keep an eye on it
Ok, with this version this is what we expect
The system was not ran in a nominal mode, a lot of manual starts which led to a mixture between command line launch and crontab launch. The command line used the proper version (0.8.2) while crontab used an older version
As observed in #77 and #78, in some cases B10 is missing in L2 C2 l2 products. In this case we output the printf command with an unexpected number of bands (8 instead 9 or 1 instead 2). The problem is related to https://github.com/WorldCereal/ewoc_work_plan/issues/122 the input product is too cloudy and USGS cannot generate the B10 L2 band.
The best solution is to filter the wp to avoid these type of products but if it is not possible we should provide a clear exception to the processing module.