search

ACCESS-Cloud-Based-InSAR / DockerizedTopsApp

Apache License 2.0
21 stars 2 forks source link

TopoStep Failed - ⚠️ - Instance Termination on Spot Market #158

Closed cmarshak closed 8 months ago

cmarshak commented 8 months ago

There are small fraction of jobs that have previously worked (on the spot market) and now do not.

The error that I saw is:

 Actual DEM bounds used: 
 Dimensions:         5202        2333
 Top Left:   -118.21083190200000        34.870098296000002     
 Spacing:    2.7777800000000001E-004  -2.7777800000000001E-004
 Lon:   -118.21083190200000       -116.76610852399999     
 Lat:    34.222320000000003        34.870098296000002     
 Lines:         7669       10001
 Pixels:         6442       11643
 Max DEM height:    3034.91162    
 Primary iterations:           25
 Secondary iterations:           10
 Distance threshold :    5.0000000000000003E-002
 Processing line:            1   7594.9157404310417     
 Dopplers:    0.0000000000000000        0.0000000000000000        0.0000000000000000     

TopsApp Steps:   0%|          | 0/24 [00:00<?, ?it/s]
TopsApp Steps:   4%|▍         | 1/24 [00:00<00:06,  3.68it/s]
TopsApp Steps:   8%|â–Š         | 2/24 [01:35<20:39, 56.36s/it]
TopsApp Steps:  12%|█▎        | 3/24 [01:37<10:58, 31.37s/it]
TopsApp Steps:  17%|█▋        | 4/24 [01:37<06:21, 19.10s/it]
TopsApp Steps:  17%|█▋        | 4/24 [15:18<1:16:33, 229.66s/it]
Traceback (most recent call last):
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/topsapp_env/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 452, in <module>
    main()
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 448, in main
    sys.exit(process_entry_point.load()())
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/__main__.py", line 252, in gunw_slc
    topsapp_processing(
  File "/opt/conda/envs/topsapp_env/lib/python3.9/site-packages/isce2_topsapp/topsapp_proc.py", line 113, in topsapp_processing
    raise ValueError(f'TopsApp failed at step: {step}')
ValueError: TopsApp failed at step: topo

The job spec is:

{'job_id': '08f0be28-5821-4ece-9cb2-29263eb9a032',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2024-01-08T20:06:29+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRRR_0108',
  'job_parameters': {'frame_id': 9849,
   'granules': ['S1A_IW_SLC__1SDV_20220206T015033_20220206T015101_041786_04F91C_DC3C',
    'S1A_IW_SLC__1SDV_20220206T015059_20220206T015126_041786_04F91C_5AC7'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T015034_20220125T015102_041611_04F317_A805',
    'S1A_IW_SLC__1SDV_20220125T015059_20220125T015126_041611_04F317_8459'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/08f0be28-5821-4ece-9cb2-29263eb9a032/08f0be28-5821-4ece-9cb2-29263eb9a032.log'],
  'expiration_time': '2024-07-07T00:00:00+00:00',
  'processing_times': [1196.083]}]

However, upon closer inspection - the issue is related to termination of the instance.

image