National Map Server for 3DEP offline?

ericbarefoot commented 10 months ago

What happened?

I am trying to programmatically download 1 m lidar tiles from the usgs3dep database for a number of sites across the united states. I have written a software package that wraps around by three dab to accomplish this in a randomized way, however the package started failing approximately a month ago. It serves the following error:

Service is currently not available, try again later: https://elevation.nationalmap.gov/arcgis/services/3DEPElevation/ImageServer/WMSServer

What did you expect to happen?

I was expecting to get a tile of lidar data downloaded from the usgs.

Minimal Complete Verifiable Example

import py3dep

bbox = (-78.027678, 39.054019, -77.913653, 39.123205)

dem = py3dep.get_map("DEM", bbox, 1)

MVCE confirmation

[X] Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue.
[X] Complete example — the example is self-contained, including all data and the text of any traceback.
[X] New issue — a search of GitHub Issues suggests this is not a duplicate.

Relevant log output

---------------------------------------------------------------------------
CPLE_OpenFailedError                      Traceback (most recent call last)
File rasterio/_base.pyx:308, in rasterio._base.DatasetBase.__init__()

File rasterio/_base.pyx:219, in rasterio._base.open_dataset()

File rasterio/_err.pyx:221, in rasterio._err.exc_wrap_pointer()

CPLE_OpenFailedError: '/vsimem/8c5edfaa-95e4-4fed-9782-d12c00c6c682/8c5edfaa-95e4-4fed-9782-d12c00c6c682.tif' not recognized as a supported file format.

During handling of the above exception, another exception occurred:

RasterioIOError                           Traceback (most recent call last)
File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/py3dep/py3dep.py:161, in get_map(layers, geometry, resolution, geo_crs, crs)
    160 try:
--> 161     ds = geoutils.gtiff2xarray(r_dict, _geometry, crs)
    162 except RasterioIOError as ex:

File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/pygeoutils/pygeoutils.py:243, in gtiff2xarray(r_dict, geometry, geo_crs, ds_dims, driver, all_touched, nodata, drop)
    241     var_name = {lyr: "_".join(lyr.split("_")[:-3]) for lyr in r_dict.keys()}
--> 243 attrs = utils.get_gtiff_attrs(r_dict[key1], ds_dims, driver, nodata)
    244 dtypes: dict[str, type] = {}

File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/pygeoutils/_utils.py:374, in get_gtiff_attrs(resp, ds_dims, driver, nodata)
    373 memfile.write(resp)
--> 374 with memfile.open(driver=driver) as src:
    375     r_crs = pyproj.CRS(src.crs)

File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/rasterio/env.py:401, in ensure_env.<locals>.wrapper(*args, **kwds)
    400 with Env.from_defaults():
--> 401     return f(*args, **kwds)

File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/rasterio/io.py:138, in MemoryFile.open(self, driver, width, height, count, crs, transform, dtype, nodata, sharing, **kwargs)
    137     log.debug("VSI path: {}".format(mempath.path))
--> 138     return DatasetReader(mempath, driver=driver, sharing=sharing, **kwargs)
    139 else:

File rasterio/_base.pyx:310, in rasterio._base.DatasetBase.__init__()

RasterioIOError: '/vsimem/8c5edfaa-95e4-4fed-9782-d12c00c6c682/8c5edfaa-95e4-4fed-9782-d12c00c6c682.tif' not recognized as a supported file format.

The above exception was the direct cause of the following exception:

ServiceUnavailableError                   Traceback (most recent call last)
Cell In[5], line 2
      1 bbox = (-78.027678, 39.054019, -77.913653, 39.123205)
----> 2 dem = py3dep.get_map("DEM", bbox, 1)

File ~/mambaforge/envs/levees_usa/lib/python3.11/site-packages/py3dep/py3dep.py:163, in get_map(layers, geometry, resolution, geo_crs, crs)
    161     ds = geoutils.gtiff2xarray(r_dict, _geometry, crs)
    162 except RasterioIOError as ex:
--> 163     raise ServiceUnavailableError(wms_url) from ex
    164 valid_layers = wms.get_validlayers()
    165 return utils.rename_layers(ds, list(valid_layers))

ServiceUnavailableError: Service is currently not available, try again later:
https://elevation.nationalmap.gov/arcgis/services/3DEPElevation/ImageServer/WMSServer

Anything else we need to know?

No response

Environment

SYS INFO -------- commit: None python: 3.11.1 | packaged by conda-forge | (main, Mar 30 2023, 16:51:07) [GCC 11.3.0] python-bits: 64 OS: Linux OS-release: 6.2.0-39-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: en_US.UTF-8 libhdf5: 1.12.2 libnetcdf: 4.9.1 PACKAGE VERSION ------------------------------- aiodns N/A aiohttp 3.8.4 aiohttp-client-cache 0.8.1 aiosqlite 0.18.0 async-retriever 0.15.0 bottleneck 1.3.7 brotli N/A click 8.1.3 cytoolz 0.12.0 dask 2023.3.2 defusedxml 0.7.1 folium 0.14.0 geopandas 0.12.2 h5netcdf N/A hydrosignatures N/A lxml 4.9.2 matplotlib 3.7.1 netCDF4 1.6.3 networkx 3.0 numba N/A numpy 1.24.2 owslib 0.29.2 pandas 1.5.3 py3dep 0.15.0 pyarrow N/A pydaymet N/A pygeohydro N/A pygeoogc 0.15.0 pygeos N/A pygeoutils 0.15.0 pynhd N/A pynldas2 N/A pyproj 3.5.0 pytest N/A pytest-cov N/A rasterio 1.3.6 requests 2.28.2 requests-cache 1.1.0 richdem N/A rioxarray 0.14.0 scipy 1.10.1 shapely 2.0.1 tables N/A ujson 5.7.0 urllib3 1.26.15 xarray 2023.3.0 xdist N/A yaml N/A -------------------------------

ericbarefoot commented 10 months ago

Oh I should say that I have already successfully downloaded this particular tile in the past.

cheginit commented 10 months ago

Thanks for reporting the issue. It seems that since this is a large request the 3DEP service was being pushed to its limit so it failed. It appears that the 3DEP service capacity is either not as good as before or the number of requests has increased. So, I had to make some modifications to reduce the load of each request. This solved the issue:

I will push this fix. I am planning to release new versions for all HyRiver packages soon. So you can either wait for the official release or install them from git:

pip install git+https://github.com/hyriver/async-retriever.git git+https://github.com/hyriver/pygeoogc.git git+https://github.com/hyriver/pygeoutils.git git+https://github.com/hyriver/py3dep.git

cheginit commented 10 months ago

BTW, you can use py3dep.get_dem(bbox, 1) if you only want to get DEM and don't want slope or other layers. This offers the benefit of automatically using the static 3DEP service instead of dynamic for 10, 30, and 60 m resolutions, which can be much faster than the dynamic service. Also, in version 0.16.0 (next release), I added a new function for large requests called py3dep.get_dem_vrt that will create a VRT file instead of loading everything to the memory, so it's much more memory efficient.

ericbarefoot commented 10 months ago

Excellent, thanks, I'll work with the new fix in the short term and look for the new stack when you release it.

Our issue is that we only want 1m lidar. Any lower resolution does not satisfy our requirements. So we just discard tiles that lack 1m-resolution coverage in our region of interest.

cheginit commented 10 months ago

The new version is already out. Please give it a try and let me know if it works.

ericbarefoot commented 10 months ago

I updated py3dep via conda-forge, and it had the same failure. However, I created a fresh environment, with only py3dep as the install, and it works fine. I now suspect there's an issue in my environment preventing the update from working correctly. Instead of trouble-shoot it, I'll just build a fresh environment.

I'm going to close this for now, but if I run into the issue again while re-building the computing environment, I'll let you know where the issue arises.

Thanks for the help!

cheginit commented 10 months ago

Great, thanks!

hyriver / py3dep