[general] PWS collection items server error

[devincowan]

PWS returning 500 https://reference.geoconnex.us/collections/pws/items

[ksonda]

It's too big and the /items endpoint will probably not work anytime soon. I can provide a geopackage if desired, but imho you can leave this one off for now. Sorry it's a weird exception.

[devincowan]

No problem @ksonda, thanks for the info

[ksonda]

I need to figure out a solution to this sooner rather than later. gpkg of this size is too big maybe. One option is to use the esri feature service https://lincolninstitute.maps.arcgis.com/home/item.html?id=f3d10b6e8ad942ea8857410ddd4b0737

Cons in CI/CD terms is the update process at https://github.com/cgs-earth/ref_pws would have to be more manual.

An alternative is to have state-based pws collections to reduce individual file sizes, but that makes reference.geoconnex.us/collections unwieldly I think

Another alternative is to set up a separate pygeoapi server dedicated to pws that divides by state or something to use smaller gpkg, and then change the PID redirects

cons there is "reference.geoconnex.us" no longer includes "pws", but IDK how big of a deal that is

[dblodgett-usgs]

Could we setup a postgis loader for some of the bigger layers?

[ksonda]

Yeah we're exploring that

[mhweber]

@ksonda look forward to seeing where you land with this, was trying to pull in pws for work today and realized issue. Keep up the great work!

[ksonda]

@mhweber if you just want the data it's at this hydroshare resource item https://github.com/internetofwater/geoconnex.us/blob/d51a384f6c0cbde29d2c27bafed69de611e92859/pygeoapi/Dockerfile#L32

[mhweber]

Thanks @ksonda - I pulled it via hydroshare wfs - assume gets me to same data?

wfs_pws <- "https://geoserver.hydroshare.org/geoserver/HS-20b908d73a784fc1a097a3b3f2b58bfb/wfs?"

wfs_request <- "request=GetFeature&service=WFS&version=2.0.0&typeName=HS-20b908d73a784fc1a097a3b3f2b58bfb:temm&outputFormat=application/json"

pws_wfs <- paste0(wfs_pws,wfs_request)
pws <- st_read(pws_wfs)

[ksonda]

It would be the same data but I’ve never tried hydroshare wfs on a geopackage. I would just http GET the geopackage to a local file.

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: Marc Weber @.> Sent: Tuesday, August 30, 2022 2:40:45 PM To: internetofwater/geoconnex.us @.> Cc: Kyle Onda @.>; Mention @.> Subject: Re: [internetofwater/geoconnex.us] [general] PWS collection items server error (Issue #165)

Thanks @ksondahttps://github.com/ksonda - I pulled it via hydroshare wfs - assume gets me to same data?

wfs_pws <- "https://geoserver.hydroshare.org/geoserver/HS-20b908d73a784fc1a097a3b3f2b58bfb/wfs?"

wfs_request <- "request=GetFeature&service=WFS&version=2.0.0&typeName=HS-20b908d73a784fc1a097a3b3f2b58bfb:temm&outputFormat=application/json"

pws_wfs <- paste0(wfs_pws,wfs_request) pws <- st_read(pws_wfs)

— Reply to this email directly, view it on GitHubhttps://github.com/internetofwater/geoconnex.us/issues/165#issuecomment-1232027900, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AKQHTNWQ4MUTZBS6ZYXAP2DV3ZIS3ANCNFSM56BYNT4A. You are receiving this because you were mentioned.Message ID: @.***>

[ksonda]

temm should have the same geometry but not necessarily all the descriptive attributes. ALso, in general ref_pws.gpkg is the most up-to-date file, not temm.shp which is what the wfs is pulling in.

[martinseul]

currently HS does not have the ability to serve geopackages via it's connected Geoserver automatically. Only public geotiffs and shapefiles are accessible via WMS,WFS. This is something I'm considering as Geoserver can read geopackages now.

[ksonda]

We have migrated to postgis so this should no longer be a problem