National Flood Hazard Layer

[fernando-aristizabal]

The following is a draft/WIP PR that adds support for FEMA's National Flood Hazard Layer via their ArcGISRestful APIs. I selected the ArcGIS services in favor of the WFS or WMS options due to greater optionality and reliability. Please see some of the issues with WFS and WMS observed below.

Example Usage

from pygeohydro import NFHL
nfhl = NFHL("NFHL", "cross-sections")
gdf_xs = nfhl.bygeom((-73.42, 43.28, -72.9, 43.52), geo_crs="epsg:4269")
gdf_xs.explore()

Issues

WFS

PyGeoOGC: Retrieve Data from RESTful, WMS, and WFS Services Notebook

basin_geom = NLDI().get_basins("01031500").geometry[0]

wfs = WFS(
    ServiceURL().wfs.fema,
    layer="public_NFHL:Base_Flood_Elevations",
    outformat="esrigeojson",
    crs="epsg:4269",
)
r = wfs.getfeature_bybox(basin_geom.bounds, box_crs="epsg:4326")
flood = geoutils.json2geodf(r.json(), "epsg:4269", "epsg:4326")

yields the following error:

555 if self.layer:
    556     if self.layer not in self.available_layer:
--> 557         raise InputValueError("layers", self.available_layer)
    558     layers = [self.layer]
    559 else:

InputValueError: Given layers is invalid. Valid options are:
NFHL:NFHL_Availability
NFHL:FIRM_Panels
NFHL:LOMRs
NFHL:LOMAs
NFHL:Political_Jurisdictions
NFHL:Profile_Baselines
NFHL:Water_Lines
NFHL:Cross-Sections
NFHL:Base_Flood_Elevations
...

changing layer parameter to "NFHL:Base_Flood_Elevations" yields:

    535 payload = {
    536     "service": "wfs",
    537     "version": self.version,
   (...)
    543     "resultType": "hits",
    544 }
    545 resp = ar.retrieve_text([self.url], [{"params": payload}])
--> 546 nfeatures = int(resp[0].split(self.nfeat_key)[-1].split(" ")[0].strip('"'))
    548 payloads = [
    549     {
    550         "service": "wfs",
   (...)
    559     for i in range(0, nfeatures, self.max_nrecords)
    560 ]
    562 return self.retrieve([self.url] * len(payloads), [{"params": p} for p in payloads])

ValueError: invalid literal for int() with base 10: '\n<ExceptionReport\n'

PyGeoUtils: Utilities for (Geo)JSON and (Geo)TIFF Conversion Notebook

geometry = Polygon(
    [
        [-118.72, 34.118],
        [-118.31, 34.118],
        [-118.31, 34.518],
        [-118.72, 34.518],
        [-118.72, 34.118],
    ]
)
crs = "epsg:4326"

url_wfs = "https://hazards.fema.gov/gis/nfhl/services/public/NFHL/MapServer/WFSServer"
wfs = WFS(
    url_wfs,
    #layer="public_NFHL:Base_Flood_Elevations",
    layer="NFHL:Base_Flood_Elevations",
    outformat="esrigeojson",
    crs="epsg:4269",
)

r = wfs.getfeature_bybox(geometry.bounds, box_crs=crs)
flood = geoutils.json2geodf(r.json(), "epsg:4269", crs)

yields:

    545 resp = ar.retrieve_text([self.url], [{"params": payload}])
--> 546 nfeatures = int(resp[0].split(self.nfeat_key)[-1].split(" ")[0].strip('"'))
    548 payloads = [
    549     {
    550         "service": "wfs",
...
    559     for i in range(0, nfeatures, self.max_nrecords)
    560 ]
    562 return self.retrieve([self.url] * len(payloads), [{"params": p} for p in payloads])

ValueError: invalid literal for int() with base 10: '\n<ExceptionReport\n'

WMS

Trying WMS gives this issue:

wms = WMS(
    "https://hazards.fema.gov/gis/nfhl/rest/services/public/NFHLWMS/MapServer/WMSServer",
    layers="0",
    outformat="image/png",
    crs="epsg:4269",
)

fema_dict = wms.getmap_bybox((-73.42, 43.28, -72.9, 43.52), 1e3, box_crs="epsg:4269")

      File src/lxml/etree.pyx:3257 in lxml.etree.fromstring

  File src/lxml/parser.pxi:1916 in lxml.etree._parseMemoryDocument

  File src/lxml/parser.pxi:1803 in lxml.etree._parseDoc

  File src/lxml/parser.pxi:1144 in lxml.etree._BaseParser._parseDoc

  File src/lxml/parser.pxi:618 in lxml.etree._ParserContext._handleParseResultDoc

  File src/lxml/parser.pxi:728 in lxml.etree._handleParseResult

  File src/lxml/parser.pxi:657 in lxml.etree._raiseParseError

  File <string>:42
XMLSyntaxError: EntityRef: expecting ';', line 42, column 122

Note

Connecting to both the WFS and WMS via QGIS both resolve in issues as well so this may not be HyRiver specific.

To-Do

• Update pygeoogc with additional ServiceURLs.

  • [X] Tests added and nox passes.
  • [X] Passes pre-commit run --all-files
  • [X] Changes and the contributor name are documented in HISTORY.rst.

[cheginit]

Thanks for the PR! Overall, looks good to me. I added a few minor comments.

[fernando-aristizabal]

Hi @cheginit - Thanks for the quick review! I'm just wondering if your comments are tagged as pending on your end?

Here is a discussion on the matter that might be relevant.

[cheginit]

I sent the previous comment before finishing my comments on the code. I was trying to figure out the issues with WFS and WMS. Can you see them now?

The WFS issue seems to be related to an issue with the web service. In the dev version of pygeoogc I made a few changes to catch some errors, so you'd have to install it from git to test it. The NFHL service returns this error: <ExceptionText><![CDATA[OutputFormat 'text/xml' not supported.]]></ExceptionText>. So it seems to be some kind of misconfiguration on their part.

For WMS, the service requires providing style, so this seems to work:

wms = WMS(
    "https://hazards.fema.gov/nfhl/services/public/NFHLWMS/MapServer/WMSServer",
    layers=0,
    outformat="image/tiff",
    crs=4269,
    validation=False,
)
fema_dict = wms.getmap_bybox((-73.42, 43.28, -72.9, 43.52), 1e3, box_crs=4269, kwargs={"styles": "default"})

[fernando-aristizabal]

The WFS issue seems to be related to an issue with the web service. In the dev version of pygeoogc I made a few changes to catch some errors, so you'd have to install it from git to test it. The NFHL service returns this error: <ExceptionText><![CDATA[OutputFormat 'text/xml' not supported.]]></ExceptionText>. So it seems to be some kind of misconfiguration on their part.

For WMS, the service requires providing style, so this seems to work:

wms = WMS(
    "https://hazards.fema.gov/nfhl/services/public/NFHLWMS/MapServer/WMSServer",
    layers=0,
    outformat="image/tiff",
    crs=4269,
    validation=False,
)
fema_dict = wms.getmap_bybox((-73.42, 43.28, -72.9, 43.52), 1e3, box_crs=4269, kwargs={"styles": "default"})

Thanks for looking into this! The WMS now works for me given the style argument.

[cheginit]

I think the only that's left is removing the commented lines, right?

[fernando-aristizabal]

Yes that's right. I've pushed up the latest with the lines commented out and some exceptions renamed. Unfortunately, the tests aren't currently passing due a possibly broader issue.

I'm currently getting 62 failing tests within pygeohydro in this branch and the main branch as well. Almost all of them appear to be related to:

FAILED tests/test_pygeohydro.py::test_sensorthings - async_retriever.exceptions.InputValueError: Given request_kwds (params) is invalid. Valid options are:

Detailing the error on a specific test reveals:

        not_found = [p for kwds in request_kwds for p in kwds if p not in session_kwds]
        if not_found:
            invalids = ", ".join(not_found)
E           raise InputValueError(f"request_kwds ({invalids})", list(session_kwds))
E           async_retriever.exceptions.InputValueError: Given request_kwds (params) is invalid. Valid options are:
E           self
E           method
E           str_or_url
E           expire_after
E           kwargs

.nox/tests-3-8/lib/python3.8/site-packages/async_retriever/_utils.py:208: InputValueError

[cheginit]

The issue was related to a breaking change in one of the dependencies of async-retriever. I just pushed a fix. Can you please try running nox again?

[fernando-aristizabal]

Thanks for getting that addressed. My latest commit passes all tests. It should be ready to merge!

[cheginit]

Awesome, thanks for taking the time to address the issue and your contribution!