Google is throwing HTTP 429 errors when opening Sentinel-2 scenes

jhkennedy commented 1 year ago

I've attached a list of ~1500 HTTP links to Sentinel-2 jobs which failed due to Google throwing HTTP 429s when trying to read the data.

From the logs it looks like GDAL is trying to open a pile of side-car files that may contain metadata, which we have no interest in and that may be causing the rate-limit issues. We are setting export GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR' to prevent GDAL From trying to list the entire cloud object store, but it's possible there's another GDAL environment variable we can set to prevent side-car files from being loaded.

To reproduce locally, you can use either the autorift environment (linux only) or docker container, which looks like:

hyp3_autorft <S2_REF> <S2_SEC>

Essentially, what's happening in the code is just a gdal.Open on the data, so we likely can reproduce by simply writing a python script that opens and reads two scenes. Psudo code:

ds = gdal.Open('/vsicurl/http...')
ds.readAsArray()

making sure the GDAL environment variables are set accordingly.

important parts of the hyp3-autorift:

the processing workflow starts here: https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L317 (it's a mess, sorry)
We first make HTTP requests to google when we loaded the SAFE manifest file here:
- https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L363-L364 which calls
- https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L94-L99
from the metadata, we determine the path to the actual data file (*.jp2) and then pass it to Geogrid https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L448-L454 which just ends up opening the files with GDAL https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/vend/testGeogridOptical.py#L95

jhkennedy commented 1 year ago

Okay, here's the pair list for real:

import json
from pathlib import Path

pair_dict = json.loads(Path('update_dict_zones_01_60_1581_pairs_20230215.json').read_text())
print(pair_dict['pair_list'])

update_dict_zones_01_60_1581_pairs_20230215.json.zip

jhkennedy commented 1 year ago

And here's a list of HTTP links to the ~1500 jobs that failed via the 429s

import json
from pathlib import Path

job_links = json.loads(Path('its-live-s2-429s.json').read_text())
print(job_links)

its-live-s2-429s.json.zip

jtherrmann commented 1 year ago

I'm not seeing the 429 error when I run autorift locally, so I re-submitted https://hyp3-its-live.asf.alaska.edu/jobs/a299d2b7-4b42-4ac7-82b6-e85abfd52730 as https://hyp3-its-live.asf.alaska.edu/jobs/ac70b6f8-efca-493c-b655-0b33c5beb174 to see if we're still seeing those errors for AWS jobs.

jtherrmann commented 1 year ago

Here's the output that I get for running the above job locally, with the following command:

hyp3_autorift S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520 S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514

Output:

This is the Open Source version of ISCE.
Some of the workflows depend on a separate licensed package.
To obtain the licensed package, please make a request for ISCE
through the website: https://download.jpl.nasa.gov/ops/request/index.cfm.
Alternatively, if you are a member, or can become a member of WinSAR
you may be able to obtain access to a version of the licensed sofware at
https://winsar.unavco.org/software/isce
2023-03-09 14:49:57,402 - matplotlib - DEBUG - matplotlib data path: /home/jth/jth-apps/miniconda3/envs/hyp3-autorift/lib/python3.9/site-packages/matplotlib/mpl-data
2023-03-09 14:49:57,407 - matplotlib - DEBUG - CONFIGDIR=/home/jth/.config/matplotlib
2023-03-09 14:49:57,408 - matplotlib - DEBUG - interactive is False
2023-03-09 14:49:57,408 - matplotlib - DEBUG - platform is linux
2023-03-09 14:49:57,454 - matplotlib - DEBUG - CACHEDIR=/home/jth/.cache/matplotlib
2023-03-09 14:49:57,455 - matplotlib.font_manager - DEBUG - Using fontManager instance from /home/jth/.cache/matplotlib/fontlist-v330.json
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.aux.xml: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.AUX: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.AUX: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.ovr: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.OVR: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.msk: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.MSK: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.aux.xml: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.AUX: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.AUX: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.ovr: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.OVR: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.msk: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.MSK: 403
2023-03-09 14:50:07,716 - hyp3_autorift.process - INFO - Reference scene path: /vsicurl/https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2
2023-03-09 14:50:07,717 - hyp3_autorift.process - INFO - Secondary scene path: /vsicurl/https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2

Optical Image parameters: 
X-direction coordinate: 399960  10
Y-direction coordinate: 1.40002e+06  -10
Dimensions: 10980 10980

Map inputs: 
EPSG: 3031
Smallest Allowable Chip Size in m: 240
Grid spacing in m: 120
Repeat Time: 9.504e+06
XLimits: -81075.1  32383.9
YLimits: -1.3691e+06  -1.25565e+06
Extent in km: 113.459  113.456
DEM: /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_h.tif
Slopes: /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_dhdx.tif  /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_dhdy.tif
Velocities: /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_vx0.tif  /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_vy0.tif
Search Range: /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_vxSearchRange.tif  /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_vySearchRange.tif
Chip Size Min: /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_xMinChipSize.tif  /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_yMinChipSize.tif
Chip Size Max: /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_xMaxChipSize.tif  /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_yMaxChipSize.tif
Stable Surface Mask: /vsicurl/https://its-live-data.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_StableSurface.tif

Outputs: 
Window locations: window_location.tif
Window offsets: window_offset.tif
Window rdr_off2vel_x vector: window_rdr_off2vel_x_vec.tif
Window rdr_off2vel_y vector: window_rdr_off2vel_y_vec.tif
Window scale factor: window_scale_factor.tif
Window search range: window_search_range.tif
Window chip size min: window_chip_size_min.tif
Window chip size max: window_chip_size_max.tif
Window stable surface mask: window_stable_surface_mask.tif
Output Nodata Value: -32767

Starting processing .... 
Xlimits : -81127.5  32392.5
Ylimits : -1.36919e+06  -1.25555e+06
Origin index (in DEM) of geogrid: 32668   43807
Dimensions of geogrid: 946 x 947
X-direction pixel size: 10
Y-direction pixel size: 10
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.aux.xml: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.AUX: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20161114T194522_N0204_R013_T01CDP_20161114T194520.SAFE/./GRANULE/L1C_T01CDP_A007303_20161114T194520/IMG_DATA/T01CDP_20161114T194522_B08.jp2.AUX: 403
Scene-center lat/lon: -77.95275473083345  1.1384257630735704
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.aux.xml: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.AUX: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.aux: 403
Warning 1: HTTP response code on https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/01/C/DP/S2A_MSIL1C_20170304T194511_N0204_R013_T01CDP_20170304T194514.SAFE/./GRANULE/L1C_T01CDP_A008876_20170304T194514/IMG_DATA/T01CDP_20170304T194511_B08.jp2.AUX: 403
Preprocessing filter width 5
Using preprocessing methods ['hps', 'hps']
Setting Wallis Filter Width to 5
Pre-process Start!!!
Using Wallis Filter Width: 5
Pre-process Done!!!
1.5302999019622803
Uniform Data Type Done!!!
5.025902032852173
AutoRIFT Start!!!
/home/jth/jth-apps/miniconda3/envs/hyp3-autorift/lib/python3.9/site-packages/autoRIFT/autoRIFT.py:1824: RuntimeWarning: divide by zero encountered in divide
  DxMadmin = np.ones(Dx.shape) / OverSampleRatio / SearchLimitX * 2
/home/jth/jth-apps/miniconda3/envs/hyp3-autorift/lib/python3.9/site-packages/autoRIFT/autoRIFT.py:1825: RuntimeWarning: divide by zero encountered in divide
  DyMadmin = np.ones(Dy.shape) / OverSampleRatio / SearchLimitY * 2
/home/jth/jth-apps/miniconda3/envs/hyp3-autorift/lib/python3.9/site-packages/numpy/lib/nanfunctions.py:1217: RuntimeWarning: All-NaN slice encountered
  r, k = function_base._ureduce(a, func=_nanmedian, axis=axis, out=out,
/home/jth/jth-apps/miniconda3/envs/hyp3-autorift/lib/python3.9/site-packages/autoRIFT/autoRIFT.py:1758: RuntimeWarning: Mean of empty slice
  C = np.nanmean(B, axis=0)
AutoRIFT Done!!!
149.4101424217224
Write Outputs Start!!!
/home/jth/jth-apps/miniconda3/envs/hyp3-autorift/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3432: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/jth/jth-apps/miniconda3/envs/hyp3-autorift/lib/python3.9/site-packages/numpy/core/_methods.py:190: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Write Outputs Done!!!
5.654061555862427

Note the several warnings about 403 errors.

jhkennedy commented 1 year ago

@jtherrmann the 403's are GDAL trying to load the "side-car" (typically metadata) files that might exist for a JP2000 image.

maybe google's resolved itself! :crossed_fingers:

jtherrmann commented 1 year ago

The job re-run (above) has succeeded, so perhaps the 429 errors resolved themselves.

jhkennedy commented 1 year ago

@jtherrmann thanks for checking on that! I'll rerun the ~1500 to see what happens

jhkennedy commented 1 year ago

Closing yet again as Google seems to have resolved itself

ASFHyP3 / hyp3-autorift

Google is throwing HTTP 429 errors when opening Sentinel-2 scenes #207