search

WorldCereal / ewoc_l8

Pre processing of L8 data (SR and TIR)
Other
0 stars 0 forks source link

tmp files not always removed in case of failure #83

Closed mbattude-cs closed 1 year ago

mbattude-cs commented 1 year ago

First run:

[2022-11-03 13:01:31,933] [world-cereal] [DEBUG ] [2022-11-03 13:01:31] CRITICAL:ewoc_l8.cli:Error during L8 ARD generation: No L8 ARD on 60KXG for ['LC08_L2SR_074072_20200827_20200923_02_T1', 'LC08_L2SR_
074071_20200827_20200923_02_T2'] !
[2022-11-03 13:01:32,164] [world-cereal] [ERROR ] Errors found in the output from command: ewoc_generate_l8_ard -v --prod-id c728b264-5c97-4f4c-81fe-1500d4c4dfbd_12036_20220926141537 --only-tir prd_ids 60
KXG LC08_L2SR_074072_20200827_20200923_02_T1 LC08_L2SR_074071_20200827_20200923_02_T2

Second run:

[2022-11-03 13:02:03,450] [world-cereal] [DEBUG ] Try number 2 (max 2). Processing product(s): LC08_L2SR_074072_20200827_20200923_02_T1 LC08_L2SR_074071_20200827_20200923_02_T2
[2022-11-03 13:02:03,450] [world-cereal] [INFO  ] Starting the processor with cmd : ewoc_generate_l8_ard -v --prod-id c728b264-5c97-4f4c-81fe-1500d4c4dfbd_12036_20220926141537 --only-tir prd_ids 60KXG LC0
8_L2SR_074072_20200827_20200923_02_T1 LC08_L2SR_074071_20200827_20200923_02_T2
[2022-11-03 13:02:03,545] [world-cereal] [INFO  ] The timeout for process end is 1800
[2022-11-03 13:02:03,546] [world-cereal] [INFO  ] Starting the watchdog for process messages timeout. Maximum time allowed for not printing any message is 0
[2022-11-03 13:02:03,546] [world-cereal] [INFO  ] Watchdog: the timeout is 0. Watching the SIGTERM signal only
[2022-11-03 13:02:05,130] [world-cereal] [INFO  ] [2022-11-03 13:02:05] INFO:ewoc_l8.l8_process:Following bands will be processed: ['B10', 'QA_PIXEL_TIR']
[2022-11-03 13:02:10,580] [world-cereal] [INFO  ] [2022-11-03 13:02:10] INFO:ewoc_l8.utils:SRS of 60KXG is EPSG:32760
[2022-11-03 13:02:10,581] [world-cereal] [INFO  ] [2022-11-03 13:02:10] INFO:ewoc_l8.utils:Bounding box of 60KXG is (600000, 8090200, 709800, 8200000)
[2022-11-03 13:02:10,582] [world-cereal] [INFO  ] [2022-11-03 13:02:10] INFO:ewoc_l8.l8_process:Processing B10 band
[2022-11-03 13:02:10,809] [world-cereal] [INFO  ] [2022-11-03 13:02:10] INFO:ewoc_dag.bucket.ewoc:EWoC bucket ewoc-ard is hosted on aws and functional
[2022-11-03 13:02:11,179] [world-cereal] [ERROR ] Errors found in the output from command: ewoc_generate_l8_ard -v --prod-id c728b264-5c97-4f4c-81fe-1500d4c4dfbd_12036_20220926141537 --only-tir prd_ids 60
KXG LC08_L2SR_074072_20200827_20200923_02_T1 LC08_L2SR_074071_20200827_20200923_02_T2
[2022-11-03 13:02:11,180] [world-cereal] [ERROR ] {{{
[2022-11-03 13:02:11,180] [world-cereal] [ERROR ] /opt/ewoc_l8_venv/lib/python3.8/site-packages/eotile/eotiles/get_bb_from_tile_id.py:87: FutureWarning: The frame.append method is deprecated and will be r
emoved from pandas in a future version. Use pandas.concat instead.
  output = output.append(tile)
Traceback (most recent call last):
  File "/opt/ewoc_l8_venv/bin/ewoc_generate_l8_ard", line 8, in <module>
    sys.exit(run())
  File "/opt/ewoc_l8_venv/lib/python3.8/site-packages/ewoc_l8/cli.py", line 273, in run
    main(sys.argv[1:])
  File "/opt/ewoc_l8_venv/lib/python3.8/site-packages/ewoc_l8/cli.py", line 237, in main
    run_id(args.l8c2l2_prd_ids,
  File "/opt/ewoc_l8_venv/lib/python3.8/site-packages/ewoc_l8/cli.py", line 106, in run_id
    process_group(
  File "/opt/ewoc_l8_venv/lib/python3.8/site-packages/ewoc_l8/l8_process.py", line 259, in process_group
    up_count, up_size, upload_path, bucket_name = process_group_band(
  File "/opt/ewoc_l8_venv/lib/python3.8/site-packages/ewoc_l8/l8_process.py", line 95, in process_group_band
    tmp_folder.mkdir(parents=True, exist_ok=False)
  File "/usr/lib/python3.8/pathlib.py", line 1288, in mkdir
    self._accessor.mkdir(self, mode)
FileExistsError: [Errno 17] File exists: '/tmp/tmp/2020-08-27/B10'

[2022-11-03 13:02:11,180] [world-cereal] [ERROR ] }}}