`./background_layers.sh` bails on fetch from `hillcrestgeo` server

[NewGraphEnvironment]

Hitting you up with the @ just in case you don't get notifications for this repo anymore (I would understand why).

@smnorris - did the flatgeobuf used to dump out to the server and no longer does or is it just not visible or alternatively the server is down perhaps? Should this be an issue logged in bcfishpass

$ ./background_layers.sh "'KOTL'"
+ '[' 1 -eq 0 ']'
+ BCGW_SOURCES='whse_fish.fiss_fish_obsrvtn_pnt_sp     whse_fish.fiss_obstacles_pnt_sp     whse_fish.fiss_stream_sample_sites_sp     whse_imagery_and_base_maps.mot_culverts_sp     whse_fish.pscis_assessment_svw     whse_fish.pscis_design_proposal_svw     whse_fish.pscis_habitat_confirmation_svw     whse_fish.pscis_remediation_svw     whse_basemapping.gba_railway_tracks_sp     whse_forest_tenure.ften_road_section_lines_svw     whse_basemapping.gba_transmission_lines_sp'
+ rm -f background_layers.gpkg
+ bcdata dump WHSE_BASEMAPPING.FWA_WATERSHED_GROUPS_POLY --query 'WATERSHED_GROUP_CODE in ('\''KOTL'\'')'
+ ogr2ogr -f GPKG background_layers.gpkg -t_srs EPSG:3005 -nln fwa_watershed_groups_poly /vsistdin/
++ fio info background_layers.gpkg --layer fwa_watershed_groups_poly --bounds
+ BOUNDS='1606085.66832271 485306.943079432 1765145.64277952 623300.204096635'
++ echo '[1606085.66832271 485306.943079432 1765145.64277952 623300.204096635]'
++ tr ' ,' ,
++ rio transform --src_crs EPSG:3005 --dst_crs EPSG:4326
++ tr -d '[] '
+ BOUNDS_LL=-117.69723916889333,49.06383071608184,-115.26278483671035,50.102292320083414
+ ogr2ogr -f GPKG background_layers.gpkg -update -nln crossings -t_srs EPSG:3005 -dim XY -spat 1606085.66832271 485306.943079432 1765145.64277952 623300.204096635 -spat_srs EPSG:3005 /vsicurl/https://www.hillcrestgeo.ca/outgoing/fishpassage/data/bcfishpass/outputs/flatgeobuf/crossings.fgb
FAILURE:
Unable to open datasource `/vsicurl/https://www.hillcrestgeo.ca/outgoing/fishpassage/data/bcfishpass/outputs/flatgeobuf/crossings.fgb' with the following drivers.
  -> `FITS'
  -> `PCIDSK'
  -> `netCDF'
  -> `PDS4'
  -> `VICAR'
  -> `JP2OpenJPEG'
  -> `PDF'
  -> `MBTiles'
  -> `BAG'
  -> `EEDA'
  -> `OGCAPI'
  -> `ESRI Shapefile'
  -> `MapInfo File'
  -> `UK .NTF'
  -> `LVBAG'
  -> `OGR_SDTS'
  -> `S57'
  -> `DGN'
  -> `OGR_VRT'
  -> `Memory'
  -> `CSV'
  -> `NAS'
  -> `GML'
  -> `GPX'
  -> `LIBKML'
  -> `KML'
  -> `GeoJSON'
  -> `GeoJSONSeq'
  -> `ESRIJSON'
  -> `TopoJSON'
  -> `Interlis 1'
  -> `Interlis 2'
  -> `OGR_GMT'
  -> `GPKG'
  -> `SQLite'
  -> `WAsP'
  -> `PostgreSQL'
  -> `OpenFileGDB'
  -> `DXF'
  -> `CAD'
  -> `FlatGeobuf'
  -> `Geoconcept'
  -> `GeoRSS'
  -> `VFK'
  -> `PGDUMP'
  -> `OSM'
  -> `GPSBabel'
  -> `OGR_PDS'
  -> `WFS'
  -> `OAPIF'
  -> `EDIGEO'
  -> `SVG'
  -> `Idrisi'
  -> `XLS'
  -> `ODS'
  -> `XLSX'
  -> `Elasticsearch'
  -> `Carto'
  -> `AmigoCloud'
  -> `SXF'
  -> `Selafin'
  -> `JML'
  -> `PLSCENES'
  -> `CSW'
  -> `VDV'
  -> `GMLAS'
  -> `MVT'
  -> `NGW'
  -> `MapML'
  -> `TIGER'
  -> `AVCBin'
  -> `AVCE00'
  -> `HTTP'

[NewGraphEnvironment]

thinking we should just grab these layers from featureserve

ex. https://features.hillcrestgeo.ca/bcfishpass/collections.html

[NewGraphEnvironment]

# ---------------
# get streams from bcfishpass
# ---------------
ogr2ogr -f GPKG background_layers.gpkg \
    -update \
    -t_srs EPSG:3005 \
    -nln streams \
    "https://features.hillcrestgeo.ca/bcfishpass/collections/bcfishpass.streams/items.json?bbox=$BOUNDS_LL"

[smnorris]

The challenge with that is the 10k feature request limit.

[smnorris]

Great for reducing data redundancy but trickier to set up.

[NewGraphEnvironment]

That seems like an understatement. I think we will need another solution. Dang

https://github.com/poissonconsulting/fwapgr/issues/49

https://github.com/CrunchyData/pg_featureserv/issues/129

[smnorris]

dumping to flatgeobuf/gpkg/whatever on S3 is no big deal. And a few GB of S3 storage is only pennies... compared to monthly $$ for additional VM memory and CPU.