search

dbekaert / RAiDER

Raytracing Atmospheric Delay Estimation for RADAR
Apache License 2.0
70 stars 39 forks source link

[BUG] HRES over Los Angeles has "Missing Delay Values" - Could be error in submission #550

Closed cmarshak closed 1 year ago

cmarshak commented 1 year ago

Describe the bug Had a conversation with @jlmaurer about HRES availability. He mentioned it has far fewer challenges than ERA5. I submitted some Hyp3 jobs to take it for a proverbial test drive.

9/15 failed. As can be see below, all of the jobs entered the Raider step (see processing_times - Raider is the second step).

The job parameters and logs of those that failed are at the end of this ticket. For a few I inspected, here is the log:

Downloading s3://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b/214b416b-d9a7-46e8-b1bd-c1d30e95749f/S1-GUNW-A-R-064-tops-20220206_20220125-015033-00119W_00033N-PP-8438-v3_0_0.nc to S1-GUNW-A-R-064-tops-20220206_20220125-015033-00119W_00033N-PP-8438-v3_0_0.nc
Downloading s3://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b/214b416b-d9a7-46e8-b1bd-c1d30e95749f/S1-GUNW-A-R-064-tops-20220206_20220125-015033-00119W_00033N-PP-8438-v3_0_0.json to S1-GUNW-A-R-064-tops-20220206_20220125-015033-00119W_00033N-PP-8438-v3_0_0.json
Writing /home/raider/.ecmwfapirc locally!

Downloading products:   0%|          | 0/1 [00:00<?, ?product/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220226T081547_V20220205T225942_20220207T005942.EOF:   0%|          | 0.00/4.41M [00:00<?, ?B/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220226T081547_V20220205T225942_20220207T005942.EOF:  24%|██▍       | 1.05M/4.41M [00:00<00:02, 1.42MB/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220226T081547_V20220205T225942_20220207T005942.EOF:  48%|████▊     | 2.10M/4.41M [00:00<00:00, 2.68MB/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220226T081547_V20220205T225942_20220207T005942.EOF:  95%|█████████▌| 4.19M/4.41M [00:01<00:00, 5.43MB/s]
Downloading S1A_OPER_AUX_POEORB_OPOD_20220226T081547_V20220205T225942_20220207T005942.EOF: 100%|██████████| 4.41M/4.41M [00:01<00:00, 4.25MB/s]

MD5 checksumming:   0%|          | 0.00/4.41M [00:00<?, ?B/s]

                                                             
Downloading products: 100%|██████████| 1/1 [00:01<00:00,  1.94s/product]
Downloading products: 100%|██████████| 1/1 [00:01<00:00,  1.94s/product]
Wrote new cfg file: GUNW_20220206-20220125_015033.yaml
WARNING: Weather model only extends to the surface topography; height levels below the topography will be interpolated from the surface and may be inaccurate.
Invalid extension GTiff for cube. Defaulting to .nc
Invalid extension GTiff for cube. Defaulting to .nc
Output cube spacing: 0.1 degrees
Output SNWE: [32.9, 35.0, -119.2, -115.7]
Starting to run the weather model calculation
Date: 20220206
Beginning weather model pre-processing
Weather model HRES is available from 1983-04-20 to Present
2023-06-08 21:52:57 ECMWF API python library 1.6.3
2023-06-08 21:52:57 ECMWF API at https://api.ecmwf.int/v1
2023-06-08 21:52:57 Welcome David Bekaert
2023-06-08 21:52:59 In case of problems, please check https://confluence.ecmwf.int/display/WEBAPI/Web+API+FAQ or contact servicedesk@ecmwf.int
2023-06-08 21:53:00 Request submitted
2023-06-08 21:53:00 Request id: 64824dbcc9093e435b42abee
2023-06-08 21:53:00 Request is submitted
2023-06-08 21:53:02 Request is queued
2023-06-08 22:01:47 Request is active
2023-06-08 22:02:57 Calling 'nice mars /tmp/20230608-2200/63/tmp-_marsgBGyhP.req'
2023-06-08 22:02:57 mars - WARN -
2023-06-08 22:02:57 mars - WARN -
2023-06-08 22:02:57 MIR environment variables:
2023-06-08 22:02:57 MIR_CACHE_PATH=/data/ec_coeff
2023-06-08 22:02:57 Using MARS binary: /usr/local/apps/mars/versions/6.33.15.2/bin/mars.bin
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Welcome to MARS
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - MARS Client build stamp: 20230328082615
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - MARS Client bundle version: 6.33.15.2
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - package mars-client version: 6.33.15
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - package mir version: 1.16.2
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - package odc version: 1.4.6
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - package fdb version: 5.11.7
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - package metkit version: 1.10.4
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - package eckit version: 1.22.0
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - package eccodes version: 2.28.1
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Maximum retrieval size is 75.00 G
2023-06-08 22:02:57 retrieve,levelist=all,stream=oper,area=35.16216216216216/-120.0/32.7/-115.53783783783784,levtype=ml,expver=1,dataset=hres,padding=0,step=0,grid=0.08108108108108109/0.08108108108108109,param=129/130/133/152,time=00:00,date=2022-02-06,resol=av,type=an,class=odmars - INFO   - 20230608.220145 - Automatic split on dates is on
2023-06-08 22:02:57 
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Processing request 1
2023-06-08 22:02:57 
2023-06-08 22:02:57 RETRIEVE,
2023-06-08 22:02:57     DATASET    = hres,
2023-06-08 22:02:57     CLASS      = OD,
2023-06-08 22:02:57     TYPE       = AN,
2023-06-08 22:02:57     STREAM     = OPER,
2023-06-08 22:02:57     EXPVER     = 0001,
2023-06-08 22:02:57     REPRES     = SH,
2023-06-08 22:02:57     LEVTYPE    = ML,
2023-06-08 22:02:57     LEVELIST   = ALL,
2023-06-08 22:02:57     PARAM      = 129/130/133/152,
2023-06-08 22:02:57     TIME       = 0000,
2023-06-08 22:02:57     STEP       = 0,
2023-06-08 22:02:57     DOMAIN     = G,
2023-06-08 22:02:57     RESOL      = AV,
2023-06-08 22:02:57     AREA       = 35.16216216216216/-120.0/32.7/-115.53783783783784,
2023-06-08 22:02:57     GRID       = 0.08108108108108109/0.08108108108108109,
2023-06-08 22:02:57     PADDING    = 0,
2023-06-08 22:02:57     DATE       = 20220206
2023-06-08 22:02:57 
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Web API request id: 64824dbcc9093e435b42abee
2023-06-08 22:02:57 mars - WARN   - 20230608.220145 - Cannot compute number of fields from request.
2023-06-08 22:02:57 mars - WARN   - 20230608.220145 - Try to avoid the use of the value 'ALL'
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Setting SO_SNDBUF to 33554432 (32.00 M)
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Current value is 8192 (8.00 K)
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Setting SO_RCVBUF to 33554432 (32.00 M)
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Current value is 43690 (42.67 K)
2023-06-08 22:02:57 mars - INFO   - 20230608.220145 - Calling mars on 'fdbprod', local port is 53533
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Server task is 312 [ATOS FDB]
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Retrieving from FDB [ATOS FDB]
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Looking up FDB indexes: 0.002226 second elapsed, 0.002216 second cpu [ATOS FDB]
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Calling mars on 'fdbbc', local port is 33987
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Server task is 97 [ATOS FDB BC]
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Retrieving from FDB [ATOS FDB BC]
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Looking up FDB indexes: 0.000561 second elapsed, 0.000561 second cpu [ATOS FDB BC]
2023-06-08 22:02:57 mars - INFO   - 20230608.220146 - Calling mars on 'marsod-core', local port is 48972
2023-06-08 22:02:57 mars - INFO   - 20230608.220201 - Server task is 318 [marsod]
2023-06-08 22:02:57 mars - INFO   - 20230608.220201 - Request cost: 276 fields, 2.11128 Gbytes online, nodes: mvr007 mvr008 [marsod]
2023-06-08 22:02:57 mars - INFO   - 20230608.220201 - The efficiency of your requests in the last 12 hours is 100% [marsod]
2023-06-08 22:02:57 mars - INFO   - 20230608.220201 - Transfering 2266967084 bytes
2023-06-08 22:02:57 mars - INFO   - 20230608.220202 - ShToGridded: loading Legendre coefficients '/data/ec_coeff/mir/legendre/4/local-T1279-GaussianN1280-OPT4189816c2e.leg'
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 - 276 fields retrieved from 'marsod'
2023-06-08 22:02:57 mars - WARN   - 20230608.220255 - Visiting database marsod : expected 0, got 276
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 - 276 fields have been interpolated
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 - Request time:  wall: 1 min 10 sec  cpu: 42 sec
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 -   Processing in marsod: wall: 16 sec
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 -   Visiting marsod: wall: 1 min 9 sec
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 -   Read from network: 2.11 Gbyte(s) in 4 sec  [599.46 Mbyte/sec]
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 -   Post-processing: wall: 49 sec cpu: 40 sec
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 -   Writing to target file: 1.26 Mbyte(s) in < 1 sec [129.46 Mbyte/sec]
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 - Memory used: 375.55 Mbyte(s)
2023-06-08 22:02:57 mars - INFO   - 20230608.220255 - No errors reported
2023-06-08 22:02:57 Process '['nice', 'mars', '/tmp/20230608-2200/63/tmp-_marsgBGyhP.req']' finished
2023-06-08 22:02:57 Calling 'nice grib_to_netcdf /data/scratch/private/blue/04/20230608-2200/c1/_mars-bol-webmars-private-svc-blue-006-7a527896970b09a4fc90fa37bf98d3ff-gnZBLG.grib -o /data/scratch/private/blue/01/20230608-2200/6a/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-3D6NSz.nc -utime'
2023-06-08 22:02:57 grib_to_netcdf: Version 2.26.0
2023-06-08 22:02:57 grib_to_netcdf: Processing input file '/data/scratch/private/blue/04/20230608-2200/c1/_mars-bol-webmars-private-svc-blue-006-7a527896970b09a4fc90fa37bf98d3ff-gnZBLG.grib'.
2023-06-08 22:02:57 grib_to_netcdf: Found 276 GRIB fields in 1 file.
2023-06-08 22:02:57 grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
2023-06-08 22:02:57 grib_to_netcdf: Creating netCDF file '/data/scratch/private/blue/01/20230608-2200/6a/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-3D6NSz.nc'
2023-06-08 22:02:57 grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
2023-06-08 22:02:57 grib_to_netcdf: Creating large (64 bit) file format.
2023-06-08 22:02:57 grib_to_netcdf: Defining variable 'z'.
2023-06-08 22:02:57 grib_to_netcdf: Defining variable 't'.
2023-06-08 22:02:57 grib_to_netcdf: Defining variable 'q'.
2023-06-08 22:02:57 grib_to_netcdf: Defining variable 'lnsp'.
2023-06-08 22:02:57 grib_to_netcdf: Done.
2023-06-08 22:02:57 Process '['nice', 'grib_to_netcdf', '/data/scratch/private/blue/04/20230608-2200/c1/_mars-bol-webmars-private-svc-blue-006-7a527896970b09a4fc90fa37bf98d3ff-gnZBLG.grib', '-o', '/data/scratch/private/blue/01/20230608-2200/6a/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-3D6NSz.nc', '-utime']' finished
2023-06-08 22:02:57 Request is complete
2023-06-08 22:02:57 Transfering 1.81736 Mbytes into /home/raider/weather_files/HRES_2022_02_06_T00_00_00.nc
2023-06-08 22:02:57 From https://apps.ecmwf.int/api/streaming/private/blue/01/20230608-2200/6a/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-3D6NSz.nc
2023-06-08 22:03:31 Transfer rate 54.8345 Kbytes/s
2023-06-08 22:03:31 Done
Number of weather model nodes: 239250
Shape of weather model: (30, 55, 145)
Bounds of the weather model: 32.78/35.13/-119.92/-115.54 (SNWE)
Weather model: HRES
Mean value of the wet refractivity: 3.706192
Mean value of the hydrostatic refractivity: 104.281334
======Weather Model class object=====
Weather model time: 2022-02-06 00:00:00
Latitude resolution: 0.08108108108108109
Longitude resolution: 0.08108108108108109
Native projection: EPSG:4326
ZMIN: -100.0
ZMAX: 15000.0
k1 = 0.776
k2 = 0.233
k3 = 3750.0
Humidity type = q
=====================================
Class name: od
Dataset: hres
=====================================
A: [0.0, 2.00036502, 3.10224104, 4.66608381, 6.82797718, 9.74696636, 13.6054239, 18.6089306, 24.9857178, 32.9857101, 42.8792419, 54.9554634, 69.5205765, 86.8958817, 107.415741, 131.425507, 159.279404, 191.338562, 227.968948, 269.539581, 316.420746, 368.982361, 427.592499, 492.616028, 564.413452, 643.339905, 729.744141, 823.967834, 926.34491, 1037.20117, 1156.85364, 1285.61035, 1423.77014, 1571.62292, 1729.44897, 1897.51929, 2076.09595, 2265.43164, 2465.77051, 2677.34814, 2900.39136, 3135.11938, 3381.74365, 3640.46826, 3911.49048, 4194.93066, 4490.81738, 4799.14941, 5119.89502, 5452.99072, 5798.34473, 6156.07422, 6526.94678, 6911.87061, 7311.86914, 7727.41211, 8159.354, 8608.52539, 9076.40039, 9562.68262, 10065.9785, 10584.6318, 11116.6621, 11660.0674, 12211.5479, 12766.873, 13324.6689, 13881.3311, 14432.1396, 14975.6152, 15508.2568, 16026.1152, 16527.3223, 17008.7891, 17467.6133, 17901.6211, 18308.4336, 18685.7188, 19031.2891, 19343.5117, 19620.043, 19859.3906, 20059.9316, 20219.6641, 20337.8633, 20412.3086, 20442.0781, 20425.7188, 20361.8164, 20249.5117, 20087.0859, 19874.0254, 19608.5723, 19290.2266, 18917.4609, 18489.707, 18006.9258, 17471.8398, 16888.6875, 16262.0469, 15596.6953, 14898.4531, 14173.3242, 13427.7695, 12668.2578, 11901.3398, 11133.3047, 10370.1758, 9617.51562, 8880.45312, 8163.375, 7470.34375, 6804.42188, 6168.53125, 5564.38281, 4993.79688, 4457.375, 3955.96094, 3489.23438, 3057.26562, 2659.14062, 2294.24219, 1961.5, 1659.47656, 1387.54688, 1143.25, 926.507812, 734.992188, 568.0625, 424.414062, 302.476562, 202.484375, 122.101562, 62.78125, 22.8359375, 3.75781298, 0.0, 0.0]
B: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.81999996e-08, 6.76070022e-06, 2.43480008e-05, 5.89219999e-05, 0.000111914298, 0.0001985774, 0.000340379687, 0.000561555324, 0.000889697927, 0.00135280553, 0.00199183798, 0.0028571242, 0.00397095364, 0.00537781464, 0.00713337678, 0.00926146004, 0.0118060224, 0.0148156285, 0.0183184519, 0.022354845, 0.0269635208, 0.032176096, 0.0380263999, 0.0445479602, 0.0517730154, 0.0597284138, 0.068448253, 0.0779583082, 0.0882857367, 0.0994616672, 0.111504652, 0.124448128, 0.138312891, 0.153125033, 0.168910414, 0.185689449, 0.203491211, 0.222332865, 0.242244005, 0.263241887, 0.285354018, 0.308598459, 0.332939088, 0.358254194, 0.384363323, 0.411124766, 0.438391209, 0.466003299, 0.493800312, 0.521619201, 0.549301147, 0.576692164, 0.603648067, 0.630035818, 0.65573597, 0.680643022, 0.704668999, 0.727738738, 0.749796569, 0.770797551, 0.790716767, 0.80953604, 0.827256083, 0.84388113, 0.859431803, 0.873929262, 0.887407541, 0.899900496, 0.911448181, 0.922095656, 0.931880772, 0.940859556, 0.949064434, 0.956549525, 0.963351727, 0.969513416, 0.975078404, 0.980071604, 0.984541893, 0.988499522, 0.99198401, 0.995002508, 0.997630119, 1.0]
Number of points in Lon/Lat = 30/55
Total number of grid points (3D): 239250
=====================================
Weather model HRES is available from 1983-04-20 to Present
2023-06-08 22:03:34 ECMWF API python library 1.6.3
2023-06-08 22:03:34 ECMWF API at https://api.ecmwf.int/v1
2023-06-08 22:03:34 Welcome David Bekaert
2023-06-08 22:03:36 In case of problems, please check https://confluence.ecmwf.int/display/WEBAPI/Web+API+FAQ or contact servicedesk@ecmwf.int
2023-06-08 22:03:37 Request submitted
2023-06-08 22:03:37 Request id: 64825039c9093e435b42abfe
2023-06-08 22:03:37 Request is submitted
2023-06-08 22:03:39 Request is queued
2023-06-08 22:10:55 Request is active
2023-06-08 22:11:41 Calling 'nice mars /tmp/20230608-2210/c6/tmp-_marsqK2mZj.req'
2023-06-08 22:11:41 mars - WARN -
2023-06-08 22:11:41 mars - WARN -
2023-06-08 22:11:41 MIR environment variables:
2023-06-08 22:11:41 MIR_CACHE_PATH=/data/ec_coeff
2023-06-08 22:11:41 Using MARS binary: /usr/local/apps/mars/versions/6.33.15.2/bin/mars.bin
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Welcome to MARS
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - MARS Client build stamp: 20230328082615
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - MARS Client bundle version: 6.33.15.2
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - package mars-client version: 6.33.15
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - package mir version: 1.16.2
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - package odc version: 1.4.6
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - package fdb version: 5.11.7
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - package metkit version: 1.10.4
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - package eckit version: 1.22.0
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - package eccodes version: 2.28.1
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Maximum retrieval size is 75.00 G
2023-06-08 22:11:41 retrieve,levelist=all,stream=oper,area=35.16216216216216/-120.0/32.7/-115.53783783783784,levtype=ml,expver=1,dataset=hres,padding=0,step=0,grid=0.08108108108108109/0.08108108108108109,param=129/130/133/152,time=06:00,date=2022-02-06,resol=av,type=an,class=odmars - INFO   - 20230608.221051 - Automatic split on dates is on
2023-06-08 22:11:41 
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Processing request 1
2023-06-08 22:11:41 
2023-06-08 22:11:41 RETRIEVE,
2023-06-08 22:11:41     DATASET    = hres,
2023-06-08 22:11:41     CLASS      = OD,
2023-06-08 22:11:41     TYPE       = AN,
2023-06-08 22:11:41     STREAM     = OPER,
2023-06-08 22:11:41     EXPVER     = 0001,
2023-06-08 22:11:41     REPRES     = SH,
2023-06-08 22:11:41     LEVTYPE    = ML,
2023-06-08 22:11:41     LEVELIST   = ALL,
2023-06-08 22:11:41     PARAM      = 129/130/133/152,
2023-06-08 22:11:41     TIME       = 0600,
2023-06-08 22:11:41     STEP       = 0,
2023-06-08 22:11:41     DOMAIN     = G,
2023-06-08 22:11:41     RESOL      = AV,
2023-06-08 22:11:41     AREA       = 35.16216216216216/-120.0/32.7/-115.53783783783784,
2023-06-08 22:11:41     GRID       = 0.08108108108108109/0.08108108108108109,
2023-06-08 22:11:41     PADDING    = 0,
2023-06-08 22:11:41     DATE       = 20220206
2023-06-08 22:11:41 
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Web API request id: 64825039c9093e435b42abfe
2023-06-08 22:11:41 mars - WARN   - 20230608.221051 - Cannot compute number of fields from request.
2023-06-08 22:11:41 mars - WARN   - 20230608.221051 - Try to avoid the use of the value 'ALL'
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Setting SO_SNDBUF to 33554432 (32.00 M)
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Current value is 8192 (8.00 K)
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Setting SO_RCVBUF to 33554432 (32.00 M)
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Current value is 43690 (42.67 K)
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Calling mars on 'fdbprod', local port is 33945
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Server task is 364 [ATOS FDB]
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Retrieving from FDB [ATOS FDB]
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Looking up FDB indexes: 0.004434 second elapsed, 0.003935 second cpu [ATOS FDB]
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Calling mars on 'fdbbc', local port is 49564
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Server task is 143 [ATOS FDB BC]
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Retrieving from FDB [ATOS FDB BC]
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Looking up FDB indexes: 0.00263 second elapsed, 0.002006 second cpu [ATOS FDB BC]
2023-06-08 22:11:41 mars - INFO   - 20230608.221051 - Calling mars on 'marsod-core', local port is 60571
2023-06-08 22:11:41 mars - INFO   - 20230608.221052 - Server task is 444 [marsod]
2023-06-08 22:11:41 mars - INFO   - 20230608.221052 - Request cost: 276 fields, 2.11128 Gbytes online, nodes: mvr007 [marsod]
2023-06-08 22:11:41 mars - INFO   - 20230608.221052 - The efficiency of your requests in the last 12 hours is 100% [marsod]
2023-06-08 22:11:41 mars - INFO   - 20230608.221053 - Transfering 2266967084 bytes
2023-06-08 22:11:41 mars - INFO   - 20230608.221053 - ShToGridded: loading Legendre coefficients '/data/ec_coeff/mir/legendre/4/local-T1279-GaussianN1280-OPT4189816c2e.leg'
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 - 276 fields retrieved from 'marsod'
2023-06-08 22:11:41 mars - WARN   - 20230608.221134 - Visiting database marsod : expected 0, got 276
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 - 276 fields have been interpolated
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 - Request time:  wall: 44 sec  cpu: 40 sec
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 -   Processing in marsod: wall: 2 sec
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 -   Visiting marsod: wall: 43 sec
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 -   Read from network: 2.11 Gbyte(s) in 3 sec  [767.54 Mbyte/sec]
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 -   Post-processing: wall: 38 sec cpu: 38 sec
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 -   Writing to target file: 1.26 Mbyte(s) in < 1 sec [127.08 Mbyte/sec]
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 - Memory used: 375.55 Mbyte(s)
2023-06-08 22:11:41 mars - INFO   - 20230608.221134 - No errors reported
2023-06-08 22:11:41 Process '['nice', 'mars', '/tmp/20230608-2210/c6/tmp-_marsqK2mZj.req']' finished
2023-06-08 22:11:41 Calling 'nice grib_to_netcdf /data/scratch/private/blue/04/20230608-2210/96/_mars-bol-webmars-private-svc-blue-001-7a527896970b09a4fc90fa37bf98d3ff-MFAlcO.grib -o /data/scratch/private/blue/01/20230608-2210/2e/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-44Wpih.nc -utime'
2023-06-08 22:11:41 grib_to_netcdf: Version 2.26.0
2023-06-08 22:11:41 grib_to_netcdf: Processing input file '/data/scratch/private/blue/04/20230608-2210/96/_mars-bol-webmars-private-svc-blue-001-7a527896970b09a4fc90fa37bf98d3ff-MFAlcO.grib'.
2023-06-08 22:11:41 grib_to_netcdf: Found 276 GRIB fields in 1 file.
2023-06-08 22:11:41 grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
2023-06-08 22:11:41 grib_to_netcdf: Creating netCDF file '/data/scratch/private/blue/01/20230608-2210/2e/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-44Wpih.nc'
2023-06-08 22:11:41 grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
2023-06-08 22:11:41 grib_to_netcdf: Creating large (64 bit) file format.
2023-06-08 22:11:41 grib_to_netcdf: Defining variable 'z'.
2023-06-08 22:11:41 grib_to_netcdf: Defining variable 't'.
2023-06-08 22:11:41 grib_to_netcdf: Defining variable 'q'.
2023-06-08 22:11:41 grib_to_netcdf: Defining variable 'lnsp'.
2023-06-08 22:11:41 grib_to_netcdf: Done.
2023-06-08 22:11:41 Process '['nice', 'grib_to_netcdf', '/data/scratch/private/blue/04/20230608-2210/96/_mars-bol-webmars-private-svc-blue-001-7a527896970b09a4fc90fa37bf98d3ff-MFAlcO.grib', '-o', '/data/scratch/private/blue/01/20230608-2210/2e/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-44Wpih.nc', '-utime']' finished
2023-06-08 22:11:41 Request is complete
2023-06-08 22:11:41 Transfering 1.81736 Mbytes into /home/raider/weather_files/HRES_2022_02_06_T06_00_00.nc
2023-06-08 22:11:41 From https://apps.ecmwf.int/api/streaming/private/blue/01/20230608-2210/2e/_grib2netcdf-bol-webmars-private-svc-blue-005-0d255c279e874ad03d66f5487aef1007-44Wpih.nc
2023-06-08 22:12:44 Transfer rate 29.2098 Kbytes/s
2023-06-08 22:12:45 Done
Number of weather model nodes: 239250
Shape of weather model: (30, 55, 145)
Bounds of the weather model: 32.78/35.13/-119.92/-115.54 (SNWE)
Weather model: HRES
Mean value of the wet refractivity: 3.950427
Mean value of the hydrostatic refractivity: 104.572166
======Weather Model class object=====
Weather model time: 2022-02-06 06:00:00
Latitude resolution: 0.08108108108108109
Longitude resolution: 0.08108108108108109
Native projection: EPSG:4326
ZMIN: -100.0
ZMAX: 15000.0
k1 = 0.776
k2 = 0.233
k3 = 3750.0
Humidity type = q
=====================================
Class name: od
Dataset: hres
=====================================
A: [0.0, 2.00036502, 3.10224104, 4.66608381, 6.82797718, 9.74696636, 13.6054239, 18.6089306, 24.9857178, 32.9857101, 42.8792419, 54.9554634, 69.5205765, 86.8958817, 107.415741, 131.425507, 159.279404, 191.338562, 227.968948, 269.539581, 316.420746, 368.982361, 427.592499, 492.616028, 564.413452, 643.339905, 729.744141, 823.967834, 926.34491, 1037.20117, 1156.85364, 1285.61035, 1423.77014, 1571.62292, 1729.44897, 1897.51929, 2076.09595, 2265.43164, 2465.77051, 2677.34814, 2900.39136, 3135.11938, 3381.74365, 3640.46826, 3911.49048, 4194.93066, 4490.81738, 4799.14941, 5119.89502, 5452.99072, 5798.34473, 6156.07422, 6526.94678, 6911.87061, 7311.86914, 7727.41211, 8159.354, 8608.52539, 9076.40039, 9562.68262, 10065.9785, 10584.6318, 11116.6621, 11660.0674, 12211.5479, 12766.873, 13324.6689, 13881.3311, 14432.1396, 14975.6152, 15508.2568, 16026.1152, 16527.3223, 17008.7891, 17467.6133, 17901.6211, 18308.4336, 18685.7188, 19031.2891, 19343.5117, 19620.043, 19859.3906, 20059.9316, 20219.6641, 20337.8633, 20412.3086, 20442.0781, 20425.7188, 20361.8164, 20249.5117, 20087.0859, 19874.0254, 19608.5723, 19290.2266, 18917.4609, 18489.707, 18006.9258, 17471.8398, 16888.6875, 16262.0469, 15596.6953, 14898.4531, 14173.3242, 13427.7695, 12668.2578, 11901.3398, 11133.3047, 10370.1758, 9617.51562, 8880.45312, 8163.375, 7470.34375, 6804.42188, 6168.53125, 5564.38281, 4993.79688, 4457.375, 3955.96094, 3489.23438, 3057.26562, 2659.14062, 2294.24219, 1961.5, 1659.47656, 1387.54688, 1143.25, 926.507812, 734.992188, 568.0625, 424.414062, 302.476562, 202.484375, 122.101562, 62.78125, 22.8359375, 3.75781298, 0.0, 0.0]
B: [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 3.81999996e-08, 6.76070022e-06, 2.43480008e-05, 5.89219999e-05, 0.000111914298, 0.0001985774, 0.000340379687, 0.000561555324, 0.000889697927, 0.00135280553, 0.00199183798, 0.0028571242, 0.00397095364, 0.00537781464, 0.00713337678, 0.00926146004, 0.0118060224, 0.0148156285, 0.0183184519, 0.022354845, 0.0269635208, 0.032176096, 0.0380263999, 0.0445479602, 0.0517730154, 0.0597284138, 0.068448253, 0.0779583082, 0.0882857367, 0.0994616672, 0.111504652, 0.124448128, 0.138312891, 0.153125033, 0.168910414, 0.185689449, 0.203491211, 0.222332865, 0.242244005, 0.263241887, 0.285354018, 0.308598459, 0.332939088, 0.358254194, 0.384363323, 0.411124766, 0.438391209, 0.466003299, 0.493800312, 0.521619201, 0.549301147, 0.576692164, 0.603648067, 0.630035818, 0.65573597, 0.680643022, 0.704668999, 0.727738738, 0.749796569, 0.770797551, 0.790716767, 0.80953604, 0.827256083, 0.84388113, 0.859431803, 0.873929262, 0.887407541, 0.899900496, 0.911448181, 0.922095656, 0.931880772, 0.940859556, 0.949064434, 0.956549525, 0.963351727, 0.969513416, 0.975078404, 0.980071604, 0.984541893, 0.988499522, 0.99198401, 0.995002508, 0.997630119, 1.0]
Number of points in Lon/Lat = 30/55
Total number of grid points (3D): 239250
=====================================
/opt/conda/envs/RAiDER/lib/python3.10/site-packages/rasterio/__init__.py:304: NotGeoreferencedWarning: Dataset has no geotransform, gcps, or rpcs. The identity matrix will be returned.
  dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
Processing slice 1 / 20: -500.0
Processing slice 2 / 20: 0.0
Processing slice 3 / 20: 500.0
Processing slice 4 / 20: 1000.0
Processing slice 5 / 20: 1500.0
Processing slice 6 / 20: 2000.0
Processing slice 7 / 20: 2500.0
Processing slice 8 / 20: 3000.0
Processing slice 9 / 20: 3500.0
Processing slice 10 / 20: 4000.0
Processing slice 11 / 20: 4500.0
Processing slice 12 / 20: 5000.0
Processing slice 13 / 20: 5500.0
Processing slice 14 / 20: 6000.0
Processing slice 15 / 20: 6500.0
Processing slice 16 / 20: 7000.0
Processing slice 17 / 20: 7500.0
Processing slice 18 / 20: 8000.0
Processing slice 19 / 20: 8500.0
Processing slice 20 / 20: 9000.0
Traceback (most recent call last):
  File "/opt/conda/envs/RAiDER/bin/raider.py", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/__main__.py", line 42, in main
    process_entry_point.load()()
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/raider.py", line 524, in calcDelaysGUNW
    cube_filenames = calcDelays([path_cfg])
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/raider.py", line 317, in calcDelays
    wet_delay, hydro_delay = tropo_delay(
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/delay.py", line 78, in tropo_delay
    ds = _get_delays_on_cube(dt, weather_model_file, wm_proj, aoi, height_levels,
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/delay.py", line 167, in _get_delays_on_cube
    raise Exception('There are missing delay values. Check your inputs. Not writing to disk.')
Exception: There are missing delay values. Check your inputs. Not writing to disk.
[{'job_id': '214b416b-d9a7-46e8-b1bd-c1d30e95749f',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9848,
   'granules': ['S1A_IW_SLC__1SDV_20220206T015006_20220206T015035_041786_04F91C_FABC',
    'S1A_IW_SLC__1SDV_20220206T015033_20220206T015101_041786_04F91C_DC3C'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T015006_20220125T015036_041611_04F317_D09A',
    'S1A_IW_SLC__1SDV_20220125T015034_20220125T015102_041611_04F317_A805'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/214b416b-d9a7-46e8-b1bd-c1d30e95749f/214b416b-d9a7-46e8-b1bd-c1d30e95749f.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8600.719, 1216.898]},
 {'job_id': 'bc961ccf-e28e-4621-a7ff-dfb7a36692ca',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9849,
   'granules': ['S1A_IW_SLC__1SDV_20220125T015034_20220125T015102_041611_04F317_A805',
    'S1A_IW_SLC__1SDV_20220125T015059_20220125T015126_041611_04F317_8459'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220113T015034_20220113T015102_041436_04ED43_9F80',
    'S1A_IW_SLC__1SDV_20220113T015100_20220113T015127_041436_04ED43_14E4'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/bc961ccf-e28e-4621-a7ff-dfb7a36692ca/bc961ccf-e28e-4621-a7ff-dfb7a36692ca.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8232.271, 1330.327]},
 {'job_id': '56ffdc94-739a-4513-8f15-53a6c84274e1',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9848,
   'granules': ['S1A_IW_SLC__1SDV_20220206T015006_20220206T015035_041786_04F91C_FABC',
    'S1A_IW_SLC__1SDV_20220206T015033_20220206T015101_041786_04F91C_DC3C'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220113T015007_20220113T015037_041436_04ED43_1441',
    'S1A_IW_SLC__1SDV_20220113T015034_20220113T015102_041436_04ED43_9F80'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/56ffdc94-739a-4513-8f15-53a6c84274e1/56ffdc94-739a-4513-8f15-53a6c84274e1.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8506.019, 1972.04]},
 {'job_id': '7d38a818-aab8-4764-b714-bb24888951c4',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9849,
   'granules': ['S1A_IW_SLC__1SDV_20220218T015033_20220218T015101_041961_04FF34_E691',
    'S1A_IW_SLC__1SDV_20220218T015059_20220218T015126_041961_04FF34_DD48'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T015034_20220125T015102_041611_04F317_A805',
    'S1A_IW_SLC__1SDV_20220125T015059_20220125T015126_041611_04F317_8459'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/7d38a818-aab8-4764-b714-bb24888951c4/7d38a818-aab8-4764-b714-bb24888951c4.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8304.889, 1527.815]},
 {'job_id': '2b3849ec-8eb3-4652-99cc-ef5295865243',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9849,
   'granules': ['S1A_IW_SLC__1SDV_20220206T015033_20220206T015101_041786_04F91C_DC3C',
    'S1A_IW_SLC__1SDV_20220206T015059_20220206T015126_041786_04F91C_5AC7'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T015034_20220125T015102_041611_04F317_A805',
    'S1A_IW_SLC__1SDV_20220125T015059_20220125T015126_041611_04F317_8459'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/2b3849ec-8eb3-4652-99cc-ef5295865243/2b3849ec-8eb3-4652-99cc-ef5295865243.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8224.992, 1453.423]},
 {'job_id': '8c167d60-c969-4b5e-b677-3299daaff770',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9849,
   'granules': ['S1A_IW_SLC__1SDV_20220218T015033_20220218T015101_041961_04FF34_E691',
    'S1A_IW_SLC__1SDV_20220218T015059_20220218T015126_041961_04FF34_DD48'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220206T015033_20220206T015101_041786_04F91C_DC3C',
    'S1A_IW_SLC__1SDV_20220206T015059_20220206T015126_041786_04F91C_5AC7'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/8c167d60-c969-4b5e-b677-3299daaff770/8c167d60-c969-4b5e-b677-3299daaff770.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8351.944, 1500.971]},
 {'job_id': '13c12e98-ba02-4014-8b97-a315ab82db80',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9849,
   'granules': ['S1A_IW_SLC__1SDV_20220206T015033_20220206T015101_041786_04F91C_DC3C',
    'S1A_IW_SLC__1SDV_20220206T015059_20220206T015126_041786_04F91C_5AC7'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220113T015034_20220113T015102_041436_04ED43_9F80',
    'S1A_IW_SLC__1SDV_20220113T015100_20220113T015127_041436_04ED43_14E4'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/13c12e98-ba02-4014-8b97-a315ab82db80/13c12e98-ba02-4014-8b97-a315ab82db80.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8231.542, 1551.787]},
 {'job_id': 'ead00014-aafc-4068-b111-62ea3c2a0cd7',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9848,
   'granules': ['S1A_IW_SLC__1SDV_20220218T015005_20220218T015035_041961_04FF34_65A4',
    'S1A_IW_SLC__1SDV_20220218T015033_20220218T015101_041961_04FF34_E691'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220206T015006_20220206T015035_041786_04F91C_FABC',
    'S1A_IW_SLC__1SDV_20220206T015033_20220206T015101_041786_04F91C_DC3C'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/ead00014-aafc-4068-b111-62ea3c2a0cd7/ead00014-aafc-4068-b111-62ea3c2a0cd7.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8747.584, 1773.649]},
 {'job_id': '592f21a8-d4b7-442b-9cbc-83f078440f8f',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'FAILED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9848,
   'granules': ['S1A_IW_SLC__1SDV_20220125T015006_20220125T015036_041611_04F317_D09A',
    'S1A_IW_SLC__1SDV_20220125T015034_20220125T015102_041611_04F317_A805'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220113T015007_20220113T015037_041436_04ED43_1441',
    'S1A_IW_SLC__1SDV_20220113T015034_20220113T015102_041436_04ED43_9F80'],
   'weather_model': 'HRES'},
  'logs': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/592f21a8-d4b7-442b-9cbc-83f078440f8f/592f21a8-d4b7-442b-9cbc-83f078440f8f.log'],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [8591.454, 1267.847]}]
cmarshak commented 1 year ago

Here are the 5/15 jobs that completed successfully:

https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/53fa596a-f299-435e-b9f7-d1c685cd2a90/S1-GUNW-A-R-064-tops-20220206_20220125-015020-00119W_00032N-PP-b0e2-v3_0_0.nc
https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/3b38ff13-78b8-4523-b1c9-0308907ee471/S1-GUNW-A-R-064-tops-20220218_20220125-015020-00119W_00032N-PP-0670-v3_0_0.nc
https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/4a8b6892-8ad1-40f3-a50c-e28c8b2edebc/S1-GUNW-A-R-064-tops-20220125_20220113-015021-00119W_00032N-PP-1fc8-v3_0_0.nc
https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/40e00903-9a89-4ece-bfc9-55e17fef7912/S1-GUNW-A-R-064-tops-20220218_20220206-015020-00119W_00032N-PP-c915-v3_0_0.nc
https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/b7803fb8-2969-4a6f-9491-01369af89423/S1-GUNW-A-R-064-tops-20220206_20220113-015020-00119W_00032N-PP-a1b7-v3_0_0.nc

(1 / 15 jobs is still running)

Here is one of the job dicts for a successful job:

{'job_id': '53fa596a-f299-435e-b9f7-d1c685cd2a90',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-06-08T17:59:33+00:00',
  'status_code': 'SUCCEEDED',
  'user_id': 'cmarshak',
  'name': 'Los-Angeles-0_64_HRES_0608',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 9847,
   'granules': ['S1A_IW_SLC__1SDV_20220206T015006_20220206T015035_041786_04F91C_FABC'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20220125T015006_20220125T015036_041611_04F317_D09A'],
   'weather_model': 'HRES'},
  'files': [{'filename': 'S1-GUNW-A-R-064-tops-20220206_20220125-015020-00119W_00032N-PP-b0e2-v3_0_0.nc',
    's3': {'bucket': 'hyp3-a19-jpl-contentbucket-1wfnatpznlg8b',
     'key': '53fa596a-f299-435e-b9f7-d1c685cd2a90/S1-GUNW-A-R-064-tops-20220206_20220125-015020-00119W_00032N-PP-b0e2-v3_0_0.nc'},
    'size': 46763184,
    'url': 'https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/53fa596a-f299-435e-b9f7-d1c685cd2a90/S1-GUNW-A-R-064-tops-20220206_20220125-015020-00119W_00032N-PP-b0e2-v3_0_0.nc'}],
  'logs': [],
  'browse_images': ['https://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b.s3.us-west-2.amazonaws.com/53fa596a-f299-435e-b9f7-d1c685cd2a90/S1-GUNW-A-R-064-tops-20220206_20220125-015020-00119W_00032N-PP-b0e2-v3_0_0.png'],
  'thumbnail_images': [],
  'expiration_time': '2023-12-06T00:00:00+00:00',
  'processing_times': [6576.498, 3373.337]}
cmarshak commented 1 year ago

@jlmaurer - looking at the log files from raider - is there any idea of how long the data took before the script started downloading from ECMWF? In the case below - it looks like a 10 minute lag time? Or is that the time for the complete download over https?

Using this log

2023-06-08 22:01:49 In case of problems, please check https://confluence.ecmwf.int/display/WEBAPI/Web+API+FAQ or contact servicedesk@ecmwf.int
2023-06-08 22:01:50 Request submitted
2023-06-08 22:01:50 Request id: 64824fcee4fb1eb8b842b7a3
2023-06-08 22:01:50 Request is submitted
2023-06-08 22:01:52 Request is queued
2023-06-08 22:09:53 Request is active
2023-06-08 22:10:21 Calling 'nice mars /tmp/20230608-2200/b8/tmp-_marsI1oAF1.req'
2023-06-08 22:10:21 mars - WARN -
2023-06-08 22:10:21 mars - WARN -
2023-06-08 22:10:21 MIR environment variables:
2023-06-08 22:10:21 MIR_CACHE_PATH=/data/ec_coeff
2023-06-08 22:10:21 Using MARS binary: /usr/local/apps/mars/versions/6.33.15.2/bin/mars.bin
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Welcome to MARS
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - MARS Client build stamp: 20230328082615
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - MARS Client bundle version: 6.33.15.2
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - package mars-client version: 6.33.15
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - package mir version: 1.16.2
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - package odc version: 1.4.6
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - package fdb version: 5.11.7
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - package metkit version: 1.10.4
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - package eckit version: 1.22.0
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - package eccodes version: 2.28.1
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Maximum retrieval size is 75.00 G
2023-06-08 22:10:21 retrieve,levelist=all,stream=oper,area=35.16216216216216/-120.0/32.7/-115.53783783783784,levtype=ml,expver=1,dataset=hres,padding=0,step=0,grid=0.08108108108108109/0.08108108108108109,param=129/130/133/152,time=06:00,date=2022-01-25,resol=av,type=an,class=odmars - INFO   - 20230608.220924 - Automatic split on dates is on
2023-06-08 22:10:21 
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Processing request 1
2023-06-08 22:10:21 
2023-06-08 22:10:21 RETRIEVE,
2023-06-08 22:10:21     DATASET    = hres,
2023-06-08 22:10:21     CLASS      = OD,
2023-06-08 22:10:21     TYPE       = AN,
2023-06-08 22:10:21     STREAM     = OPER,
2023-06-08 22:10:21     EXPVER     = 0001,
2023-06-08 22:10:21     REPRES     = SH,
2023-06-08 22:10:21     LEVTYPE    = ML,
2023-06-08 22:10:21     LEVELIST   = ALL,
2023-06-08 22:10:21     PARAM      = 129/130/133/152,
2023-06-08 22:10:21     TIME       = 0600,
2023-06-08 22:10:21     STEP       = 0,
2023-06-08 22:10:21     DOMAIN     = G,
2023-06-08 22:10:21     RESOL      = AV,
2023-06-08 22:10:21     AREA       = 35.16216216216216/-120.0/32.7/-115.53783783783784,
2023-06-08 22:10:21     GRID       = 0.08108108108108109/0.08108108108108109,
2023-06-08 22:10:21     PADDING    = 0,
2023-06-08 22:10:21     DATE       = 20220125
2023-06-08 22:10:21 
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Web API request id: 64824fcee4fb1eb8b842b7a3
2023-06-08 22:10:21 mars - WARN   - 20230608.220924 - Cannot compute number of fields from request.
2023-06-08 22:10:21 mars - WARN   - 20230608.220924 - Try to avoid the use of the value 'ALL'
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Setting SO_SNDBUF to 33554432 (32.00 M)
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Current value is 8192 (8.00 K)
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Setting SO_RCVBUF to 33554432 (32.00 M)
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Current value is 43690 (42.67 K)
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Calling mars on 'fdbprod', local port is 59743
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Server task is 420 [ATOS FDB]
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Retrieving from FDB [ATOS FDB]
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Looking up FDB indexes: 0.004183 second elapsed, 0.003815 second cpu [ATOS FDB]
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Calling mars on 'fdbbc', local port is 51629
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Server task is 499 [ATOS FDB BC]
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Retrieving from FDB [ATOS FDB BC]
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Looking up FDB indexes: 0.000566 second elapsed, 0.000566 second cpu [ATOS FDB BC]
2023-06-08 22:10:21 mars - INFO   - 20230608.220924 - Calling mars on 'marsod-core', local port is 53305
2023-06-08 22:10:21 mars - INFO   - 20230608.220927 - Server task is 713 [marsod]
2023-06-08 22:10:21 mars - INFO   - 20230608.220927 - Request cost: 276 fields, 2.11128 Gbytes online, nodes: mvr003 [marsod]
2023-06-08 22:10:21 mars - INFO   - 20230608.220927 - The efficiency of your requests in the last 12 hours is 100% [marsod]
2023-06-08 22:10:21 mars - INFO   - 20230608.220927 - Transfering 2266967084 bytes
2023-06-08 22:10:21 mars - INFO   - 20230608.220927 - ShToGridded: loading Legendre coefficients '/data/ec_coeff/mir/legendre/4/local-T1279-GaussianN1280-OPT4189816c2e.leg'
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 - 276 fields retrieved from 'marsod'
2023-06-08 22:10:21 mars - WARN   - 20230608.221014 - Visiting database marsod : expected 0, got 276
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 - 276 fields have been interpolated
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 - Request time:  wall: 50 sec  cpu: 42 sec
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 -   Processing in marsod: wall: 3 sec
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 -   Visiting marsod: wall: 50 sec
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 -   Read from network: 2.11 Gbyte(s) in 7 sec  [313.03 Mbyte/sec]
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 -   Post-processing: wall: 39 sec cpu: 39 sec
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 -   Writing to target file: 1.26 Mbyte(s) in < 1 sec [115.74 Mbyte/sec]
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 - Memory used: 375.55 Mbyte(s)
2023-06-08 22:10:21 mars - INFO   - 20230608.221014 - No errors reported
2023-06-08 22:10:21 Process '['nice', 'mars', '/tmp/20230608-2200/b8/tmp-_marsI1oAF1.req']' finished
2023-06-08 22:10:21 Calling 'nice grib_to_netcdf /data/scratch/private/blue/02/20230608-2200/5d/_mars-bol-webmars-private-svc-blue-008-4a73a881a8d5eead47db9eff2f9935a4-bSsvPl.grib -o /data/scratch/private/blue/02/20230608-2210/0b/_grib2netcdf-bol-webmars-private-svc-blue-003-4a73a881a8d5eead47db9eff2f9935a4-BbQMRA.nc -utime'
2023-06-08 22:10:21 grib_to_netcdf: Version 2.26.0
2023-06-08 22:10:21 grib_to_netcdf: Processing input file '/data/scratch/private/blue/02/20230608-2200/5d/_mars-bol-webmars-private-svc-blue-008-4a73a881a8d5eead47db9eff2f9935a4-bSsvPl.grib'.
2023-06-08 22:10:21 grib_to_netcdf: Found 276 GRIB fields in 1 file.
2023-06-08 22:10:21 grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
2023-06-08 22:10:21 grib_to_netcdf: Creating netCDF file '/data/scratch/private/blue/02/20230608-2210/0b/_grib2netcdf-bol-webmars-private-svc-blue-003-4a73a881a8d5eead47db9eff2f9935a4-BbQMRA.nc'
2023-06-08 22:10:21 grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
2023-06-08 22:10:21 grib_to_netcdf: Creating large (64 bit) file format.
2023-06-08 22:10:21 grib_to_netcdf: Defining variable 'z'.
2023-06-08 22:10:21 grib_to_netcdf: Defining variable 't'.
2023-06-08 22:10:21 grib_to_netcdf: Defining variable 'q'.
2023-06-08 22:10:21 grib_to_netcdf: Defining variable 'lnsp'.
2023-06-08 22:10:21 grib_to_netcdf: Done.
2023-06-08 22:10:21 Process '['nice', 'grib_to_netcdf', '/data/scratch/private/blue/02/20230608-2200/5d/_mars-bol-webmars-private-svc-blue-008-4a73a881a8d5eead47db9eff2f9935a4-bSsvPl.grib', '-o', '/data/scratch/private/blue/02/20230608-2210/0b/_grib2netcdf-bol-webmars-private-svc-blue-003-4a73a881a8d5eead47db9eff2f9935a4-BbQMRA.nc', '-utime']' finished
2023-06-08 22:10:21 Request is complete
2023-06-08 22:10:21 Transfering 1.81736 Mbytes into /home/raider/weather_files/HRES_2022_01_25_T06_00_00.nc
2023-06-08 22:10:21 From https://apps.ecmwf.int/api/streaming/private/blue/02/20230608-2210/0b/_grib2netcdf-bol-webmars-private-svc-blue-003-4a73a881a8d5eead47db9eff2f9935a4-BbQMRA.nc
2023-06-08 22:10:26 Transfer rate 413.797 Kbytes/s
2023-06-08 22:10:26 Done
cmarshak commented 1 year ago

This is also an interesting issue. We should follow up about this:

Downloading s3://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b/3454a837-7f14-47c6-a663-2ea960e2cce5/S1-GUNW-D-R-048-tops-20220216_20220204-235953-00090E_00040N-PP-ff95-v3_0_0.nc to S1-GUNW-D-R-048-tops-20220216_20220204-235953-00090E_00040N-PP-ff95-v3_0_0.nc
Downloading s3://hyp3-a19-jpl-contentbucket-1wfnatpznlg8b/3454a837-7f14-47c6-a663-2ea960e2cce5/S1-GUNW-D-R-048-tops-20220216_20220204-235953-00090E_00040N-PP-ff95-v3_0_0.json to S1-GUNW-D-R-048-tops-20220216_20220204-235953-00090E_00040N-PP-ff95-v3_0_0.json
Writing /home/raider/.ecmwfapirc locally!

Downloading products:   0%|          | 0/1 [00:00<?, ?product/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220309T081630_V20220216T225942_20220218T005942.EOF:   0%|          | 0.00/4.41M [00:00<?, ?B/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220309T081630_V20220216T225942_20220218T005942.EOF:  24%|██▍       | 1.05M/4.41M [00:00<00:03, 1.11MB/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220309T081630_V20220216T225942_20220218T005942.EOF:  48%|████▊     | 2.10M/4.41M [00:01<00:01, 2.19MB/s]

Downloading S1A_OPER_AUX_POEORB_OPOD_20220309T081630_V20220216T225942_20220218T005942.EOF:  95%|█████████▌| 4.19M/4.41M [00:01<00:00, 4.60MB/s]
Downloading S1A_OPER_AUX_POEORB_OPOD_20220309T081630_V20220216T225942_20220218T005942.EOF: 100%|██████████| 4.41M/4.41M [00:01<00:00, 3.51MB/s]

MD5 checksumming:   0%|          | 0.00/4.41M [00:00<?, ?B/s]

                                                             
Downloading products: 100%|██████████| 1/1 [00:01<00:00,  1.91s/product]
Downloading products: 100%|██████████| 1/1 [00:01<00:00,  1.91s/product]
Wrote new cfg file: GUNW_20220216-20220204_235953.yaml
WARNING: Weather model only extends to the surface topography; height levels below the topography will be interpolated from the surface and may be inaccurate.
Invalid extension GTiff for cube. Defaulting to .nc
Invalid extension GTiff for cube. Defaulting to .nc
Output cube spacing: 0.1 degrees
Output SNWE: [39.8, 42.0, 89.7, 93.5]
Starting to run the weather model calculation
Date: 20220217
Beginning weather model pre-processing
Weather model HRES is available from 1983-04-20 to Present
2023-06-08 20:27:58 ECMWF API python library 1.6.3
2023-06-08 20:27:58 ECMWF API at https://api.ecmwf.int/v1
2023-06-08 20:27:59 Welcome David Bekaert
2023-06-08 20:28:01 In case of problems, please check https://confluence.ecmwf.int/display/WEBAPI/Web+API+FAQ or contact servicedesk@ecmwf.int
2023-06-08 20:28:02 Request submitted
2023-06-08 20:28:02 Request id: 648239d202015829529e8117
2023-06-08 20:28:02 Request is submitted
ERROR: 'ecmwf.API error 1: ERROR 102 (USER_QUEUED_LIMIT_EXCEEDED): Too many queued requests. Max allowed queued requests per user is 20.'
Weather model point bounds are 39.80/42.00/89.70/94.12
Query datetime: 2022-02-17 00:00:00
ERROR: [Errno 2] No such file or directory: '/home/raider/weather_files/HRES_2022_02_17_T00_00_00.nc'
ERROR: Downloading and/or preparation of HRES failed.
ERROR: No weather model data was successfully obtained.
Traceback (most recent call last):
  File "/opt/conda/envs/RAiDER/bin/raider.py", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/__main__.py", line 42, in main
    process_entry_point.load()()
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/raider.py", line 524, in calcDelaysGUNW
    cube_filenames = calcDelays([path_cfg])
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/raider.py", line 277, in calcDelays
    raise RuntimeError
RuntimeError