Closed cmarshak closed 1 year ago
cmarshak
commented
1 year ago Also worth noting in the piece about latency is 20 minutes.
2023-06-15 01:49:54 Request is queued
2023-06-15 02:07:03 Request is active
2023-06-15 02:07:42 Calling 'nice mars /tmp/20230615-0200/4e/tmp-_marsl3W7RZ.req'
bbuzz31
commented
1 year ago Closed with #551. @cmarshak feel free to start another stack!
cmarshak
commented
1 year ago I don't think this was fixed. I am seeing the same errors in job submissions after I submitted them last week. I requested 18 jobs over Tibet and all failed with the same error.
[{'job_id': '5800dbe7-2a43-4ba5-b54f-5f2ef5a6b27f',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01',
'S1A_IW_SLC__1SDV_20220123T235953_20220124T000020_041595_04F294_C407'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220111T235928_20220111T235955_041420_04ECCD_F95A',
'S1A_IW_SLC__1SDV_20220111T235953_20220112T000020_041420_04ECCD_58DE'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/5800dbe7-2a43-4ba5-b54f-5f2ef5a6b27f/5800dbe7-2a43-4ba5-b54f-5f2ef5a6b27f.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7435.677, 609.719]},
{'job_id': '346b4324-b7f3-4417-bc27-37b2f2545614',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220111T235903_20220111T235931_041420_04ECCD_BDCE',
'S1A_IW_SLC__1SDV_20220111T235928_20220111T235955_041420_04ECCD_F95A'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20211230T235903_20211230T235931_041245_04E6D7_5206',
'S1A_IW_SLC__1SDV_20211230T235929_20211230T235956_041245_04E6D7_CFFB'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/346b4324-b7f3-4417-bc27-37b2f2545614/346b4324-b7f3-4417-bc27-37b2f2545614.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7444.288, 13.351]},
{'job_id': '956c0281-2aa3-4d68-b559-9d6cf20ca618',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01',
'S1A_IW_SLC__1SDV_20220123T235953_20220124T000020_041595_04F294_C407'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20211230T235929_20211230T235956_041245_04E6D7_CFFB',
'S1A_IW_SLC__1SDV_20211230T235954_20211231T000021_041245_04E6D7_0AF9'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/956c0281-2aa3-4d68-b559-9d6cf20ca618/956c0281-2aa3-4d68-b559-9d6cf20ca618.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7636.384, 656.997]},
{'job_id': 'fffbc463-414c-4f69-9366-3a1c8c319982',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220204T235901_20220204T235929_041770_04F88B_F6CE',
'S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220111T235903_20220111T235931_041420_04ECCD_BDCE',
'S1A_IW_SLC__1SDV_20220111T235928_20220111T235955_041420_04ECCD_F95A'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/fffbc463-414c-4f69-9366-3a1c8c319982/fffbc463-414c-4f69-9366-3a1c8c319982.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7460.419, 592.041]},
{'job_id': 'd4840935-ffed-42b8-85fe-9a118f8367be',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220216T235901_20220216T235929_041945_04FEB2_F35F',
'S1A_IW_SLC__1SDV_20220216T235927_20220216T235954_041945_04FEB2_5555'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220123T235902_20220123T235930_041595_04F294_D65F',
'S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/d4840935-ffed-42b8-85fe-9a118f8367be/d4840935-ffed-42b8-85fe-9a118f8367be.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7684.618, 15.061]},
{'job_id': '54859bde-9c62-42ec-88df-2602ad739f1b',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220111T235928_20220111T235955_041420_04ECCD_F95A',
'S1A_IW_SLC__1SDV_20220111T235953_20220112T000020_041420_04ECCD_58DE'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20211230T235929_20211230T235956_041245_04E6D7_CFFB',
'S1A_IW_SLC__1SDV_20211230T235954_20211231T000021_041245_04E6D7_0AF9'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/54859bde-9c62-42ec-88df-2602ad739f1b/54859bde-9c62-42ec-88df-2602ad739f1b.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7536.151, 13.158]},
{'job_id': 'ca24cc5f-5811-4b9b-965e-56406d72ac36',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220216T235927_20220216T235954_041945_04FEB2_5555',
'S1A_IW_SLC__1SDV_20220216T235952_20220217T000019_041945_04FEB2_BE03'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D',
'S1A_IW_SLC__1SDV_20220204T235952_20220205T000019_041770_04F88B_3725'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/ca24cc5f-5811-4b9b-965e-56406d72ac36/ca24cc5f-5811-4b9b-965e-56406d72ac36.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7391.936, 558.664]},
{'job_id': '00c7b139-585f-4979-b54f-b16dcac88ff2',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220204T235901_20220204T235929_041770_04F88B_F6CE',
'S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220123T235902_20220123T235930_041595_04F294_D65F',
'S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/00c7b139-585f-4979-b54f-b16dcac88ff2/00c7b139-585f-4979-b54f-b16dcac88ff2.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7510.162, 651.218]},
{'job_id': 'd98076fa-ec44-417b-82fd-46882f6f4bcc',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220228T235927_20220228T235954_042120_0504AF_0D7B',
'S1A_IW_SLC__1SDV_20220228T235952_20220301T000019_042120_0504AF_D95E'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D',
'S1A_IW_SLC__1SDV_20220204T235952_20220205T000019_041770_04F88B_3725'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/d98076fa-ec44-417b-82fd-46882f6f4bcc/d98076fa-ec44-417b-82fd-46882f6f4bcc.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7430.911, 559.278]},
{'job_id': '9f172e8b-837e-4fde-91a4-7873cfdb55fa',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220228T235927_20220228T235954_042120_0504AF_0D7B',
'S1A_IW_SLC__1SDV_20220228T235952_20220301T000019_042120_0504AF_D95E'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220216T235927_20220216T235954_041945_04FEB2_5555',
'S1A_IW_SLC__1SDV_20220216T235952_20220217T000019_041945_04FEB2_BE03'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/9f172e8b-837e-4fde-91a4-7873cfdb55fa/9f172e8b-837e-4fde-91a4-7873cfdb55fa.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7399.677, 604.661]},
{'job_id': '34368337-fa58-4f6d-8dbf-fd8073fb0d2f',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220216T235901_20220216T235929_041945_04FEB2_F35F',
'S1A_IW_SLC__1SDV_20220216T235927_20220216T235954_041945_04FEB2_5555'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220204T235901_20220204T235929_041770_04F88B_F6CE',
'S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/34368337-fa58-4f6d-8dbf-fd8073fb0d2f/34368337-fa58-4f6d-8dbf-fd8073fb0d2f.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7627.852, 13.747]},
{'job_id': '2cf027ba-03e5-4e94-a11d-728a8263d396',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220228T235901_20220228T235929_042120_0504AF_6899',
'S1A_IW_SLC__1SDV_20220228T235927_20220228T235954_042120_0504AF_0D7B'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220204T235901_20220204T235929_041770_04F88B_F6CE',
'S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/2cf027ba-03e5-4e94-a11d-728a8263d396/2cf027ba-03e5-4e94-a11d-728a8263d396.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7450.047, 13.444]},
{'job_id': 'b3d5e3b9-30ad-402b-ba32-29d169502809',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220216T235927_20220216T235954_041945_04FEB2_5555',
'S1A_IW_SLC__1SDV_20220216T235952_20220217T000019_041945_04FEB2_BE03'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01',
'S1A_IW_SLC__1SDV_20220123T235953_20220124T000020_041595_04F294_C407'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/b3d5e3b9-30ad-402b-ba32-29d169502809/b3d5e3b9-30ad-402b-ba32-29d169502809.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7607.973, 13.85]},
{'job_id': '25a0b732-9090-433f-b3c7-62f128bb4fb3',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220123T235902_20220123T235930_041595_04F294_D65F',
'S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220111T235903_20220111T235931_041420_04ECCD_BDCE',
'S1A_IW_SLC__1SDV_20220111T235928_20220111T235955_041420_04ECCD_F95A'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/25a0b732-9090-433f-b3c7-62f128bb4fb3/25a0b732-9090-433f-b3c7-62f128bb4fb3.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7409.09, 14.036]},
{'job_id': 'e66bc1af-df57-407f-bc38-3e73e441c8fb',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220228T235901_20220228T235929_042120_0504AF_6899',
'S1A_IW_SLC__1SDV_20220228T235927_20220228T235954_042120_0504AF_0D7B'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220216T235901_20220216T235929_041945_04FEB2_F35F',
'S1A_IW_SLC__1SDV_20220216T235927_20220216T235954_041945_04FEB2_5555'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/e66bc1af-df57-407f-bc38-3e73e441c8fb/e66bc1af-df57-407f-bc38-3e73e441c8fb.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7437.423, 13.711]},
{'job_id': '73c38992-f6d2-444f-8489-c8faab8f38b6',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7419,
'granules': ['S1A_IW_SLC__1SDV_20220123T235902_20220123T235930_041595_04F294_D65F',
'S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20211230T235903_20211230T235931_041245_04E6D7_5206',
'S1A_IW_SLC__1SDV_20211230T235929_20211230T235956_041245_04E6D7_CFFB'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/73c38992-f6d2-444f-8489-c8faab8f38b6/73c38992-f6d2-444f-8489-c8faab8f38b6.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7712.356, 14.297]},
{'job_id': '1ecae3ba-442a-4b7d-bd22-ab2d75c91bf3',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D',
'S1A_IW_SLC__1SDV_20220204T235952_20220205T000019_041770_04F88B_3725'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220123T235928_20220123T235955_041595_04F294_3E01',
'S1A_IW_SLC__1SDV_20220123T235953_20220124T000020_041595_04F294_C407'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/1ecae3ba-442a-4b7d-bd22-ab2d75c91bf3/1ecae3ba-442a-4b7d-bd22-ab2d75c91bf3.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7437.696, 646.279]},
{'job_id': 'cf930bdf-31cc-4ac1-8cbf-f06eae532e5b',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:00:47+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Tibet_48_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 7420,
'granules': ['S1A_IW_SLC__1SDV_20220204T235927_20220204T235954_041770_04F88B_BD1D',
'S1A_IW_SLC__1SDV_20220204T235952_20220205T000019_041770_04F88B_3725'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220111T235928_20220111T235955_041420_04ECCD_F95A',
'S1A_IW_SLC__1SDV_20220111T235953_20220112T000020_041420_04ECCD_58DE'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/cf930bdf-31cc-4ac1-8cbf-f06eae532e5b/cf930bdf-31cc-4ac1-8cbf-f06eae532e5b.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7452.941, 640.543]}]I also submitted jobs over LA using a different track than above (below 71).
[{'job_id': 'ca1f3a92-fa4b-4453-8928-7b348b2e774d',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:08:23+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Los-Angeles-1_71_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 11041,
'granules': ['S1A_IW_SLC__1SDV_20220218T135216_20220218T135244_041968_04FF7A_81E6',
'S1A_IW_SLC__1SDV_20220218T135242_20220218T135309_041968_04FF7A_0A50'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220206T135216_20220206T135244_041793_04F965_0E4C',
'S1A_IW_SLC__1SDV_20220206T135242_20220206T135309_041793_04F965_DD6F'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/ca1f3a92-fa4b-4453-8928-7b348b2e774d/ca1f3a92-fa4b-4453-8928-7b348b2e774d.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [8603.787, 1126.958]},
{'job_id': '2e53cb8f-185c-4bba-ab3b-63da069aad7d',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:08:23+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Los-Angeles-1_71_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 11041,
'granules': ['S1A_IW_SLC__1SDV_20220218T135216_20220218T135244_041968_04FF7A_81E6',
'S1A_IW_SLC__1SDV_20220218T135242_20220218T135309_041968_04FF7A_0A50'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T135217_20220125T135245_041618_04F361_0830',
'S1A_IW_SLC__1SDV_20220125T135243_20220125T135310_041618_04F361_72DA'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/2e53cb8f-185c-4bba-ab3b-63da069aad7d/2e53cb8f-185c-4bba-ab3b-63da069aad7d.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [8325.12, 18.771]},
{'job_id': '6d3eea4b-d876-4d4b-8d5a-8c8f187e022a',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:08:23+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Los-Angeles-1_71_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 11041,
'granules': ['S1A_IW_SLC__1SDV_20220206T135216_20220206T135244_041793_04F965_0E4C',
'S1A_IW_SLC__1SDV_20220206T135242_20220206T135309_041793_04F965_DD6F'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T135217_20220125T135245_041618_04F361_0830',
'S1A_IW_SLC__1SDV_20220125T135243_20220125T135310_041618_04F361_72DA'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/6d3eea4b-d876-4d4b-8d5a-8c8f187e022a/6d3eea4b-d876-4d4b-8d5a-8c8f187e022a.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [8190.843, 1201.334]},
{'job_id': 'b104358e-d38f-44b3-b21f-7021a397fb85',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:08:23+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Los-Angeles-1_71_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 11042,
'granules': ['S1A_IW_SLC__1SDV_20220218T135242_20220218T135309_041968_04FF7A_0A50',
'S1A_IW_SLC__1SDV_20220218T135307_20220218T135334_041968_04FF7A_8B41'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T135243_20220125T135310_041618_04F361_72DA',
'S1A_IW_SLC__1SDV_20220125T135308_20220125T135334_041618_04F361_8C45'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/b104358e-d38f-44b3-b21f-7021a397fb85/b104358e-d38f-44b3-b21f-7021a397fb85.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [7941.767, 18.937]},
{'job_id': '39fb0105-d27e-4f24-82da-19ebee252739',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:08:23+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Los-Angeles-1_71_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 11041,
'granules': ['S1A_IW_SLC__1SDV_20220125T135217_20220125T135245_041618_04F361_0830',
'S1A_IW_SLC__1SDV_20220125T135243_20220125T135310_041618_04F361_72DA'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220113T135218_20220113T135245_041443_04ED72_18C1',
'S1A_IW_SLC__1SDV_20220113T135243_20220113T135310_041443_04ED72_52D3'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/39fb0105-d27e-4f24-82da-19ebee252739/39fb0105-d27e-4f24-82da-19ebee252739.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [8188.074, 18.67]},
{'job_id': '0a426a44-27b3-4834-a982-6b57eef77eca',
'job_type': 'INSAR_ISCE_TEST',
'request_time': '2023-06-16T01:08:23+00:00',
'status_code': 'FAILED',
'user_id': 'cmarshak',
'name': 'Los-Angeles-1_71_HRES_0615',
'job_parameters': {'compute_solid_earth_tide': True,
'estimate_ionosphere_delay': True,
'frame_id': 11041,
'granules': ['S1A_IW_SLC__1SDV_20220206T135216_20220206T135244_041793_04F965_0E4C',
'S1A_IW_SLC__1SDV_20220206T135242_20220206T135309_041793_04F965_DD6F'],
'secondary_granules': ['S1A_IW_SLC__1SDV_20220113T135218_20220113T135245_041443_04ED72_18C1',
'S1A_IW_SLC__1SDV_20220113T135243_20220113T135310_041443_04ED72_52D3'],
'weather_model': 'HRES'},
'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/0a426a44-27b3-4834-a982-6b57eef77eca/0a426a44-27b3-4834-a982-6b57eef77eca.log'],
'expiration_time': '2023-12-14T00:00:00+00:00',
'processing_times': [8246.258, 1280.766]}]
sssangha
commented
1 year ago Hello @jlmaurer I receive a similar error when attempting to download HRRR:
Extent of the weather model is (xmin, ymin, xmax, ymax):-133.96, 21.14, -61.09, 47.63
Extent of the input is (xmin, ymin, xmax, ymax): -125.56, 21.44, -61.44, 47.56
/u/leffe-data/ssangha/conda_installation/stable_july30_2020/envs/RAiDER/lib/python3.10/site-packages/rasterio/__init__.py:304: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix will be returned.
dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
Traceback (most recent call last):
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/envs/RAiDER/bin/raider.py", line 8, in <module>
sys.exit(main())
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/RAiDER/tools/RAiDER/cli/__main__.py", line 42, in main
process_entry_point.load()()
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/RAiDER/tools/RAiDER/cli/raider.py", line 316, in calcDelays
wet_delay, hydro_delay = tropo_delay(
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/RAiDER/tools/RAiDER/delay.py", line 78, in tropo_delay
ds = _get_delays_on_cube(dt, weather_model_file, wm_proj, aoi, height_levels,
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/RAiDER/tools/RAiDER/delay.py", line 167, in _get_delays_on_cube
raise Exception('There are missing delay values. Check your inputs. Not writing to disk.')
Exception: There are missing delay values. Check your inputs. Not writing to disk.Also, not sure if this is related (i.e. associated with the specified area of interest) but the program fails when I pass a list of stations as an argument for station_field as it spans beyond the available range of HRRR data. Looks like we will have to adjust the padding behavior here but am not sure how best to proceed. @bbuzz31 perhaps you can advise?
Extent of the weather model is (xmin, ymin, xmax, ymax):-134.08, 21.14, -60.97, 47.83
Extent of the input is (xmin, ymin, xmax, ymax): -125.60, 20.96, -59.96, 49.06
ERROR: The weather model passed does not cover all of the input points; you may need to download a larger area.
ERROR: No weather model data was successfully obtained.
Traceback (most recent call last):
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/envs/RAiDER/bin/raider.py", line 8, in <module>
sys.exit(main())
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/RAiDER/tools/RAiDER/cli/__main__.py", line 42, in main
process_entry_point.load()()
File "/u/leffe-data/ssangha/conda_installation/stable_july30_2020/RAiDER/tools/RAiDER/cli/raider.py", line 276, in calcDelays
raise RuntimeError
RuntimeErrorHi @sssangha this issue ticket is specifically for HRES over LA. Please open two new issue tickets for each of these, with the config file you used so that I can debug. These bounding issues are always trickier than they seem
Describe the bug There were some issues with #550 because GUNWs were strangely not available. I am creating this separate bug ticket for transparency and clarity.
Here is a GUNW link for the job below. Please download @bbuzz31.
Here is the log:
Here is the job dict: