search

smnorris / bcfishpass

Model and monitor aquatic habitat connectivity in BC. Tools to plan and prioritize the assessment and remediation of barriers.
https://smnorris.github.io/bcfishpass
Apache License 2.0
7 stars 13 forks source link

DRA load failing #496

Open smnorris opened 1 month ago

smnorris commented 1 month ago

Scheduled loads are failing.

+ ogr2ogr -f PostgreSQL 'PG:***' --config PG_USE_COPY YES -lco GEOMETRY_NAME=geom -lco FID=transport_line_id -nln bcdata.transport_line /vsizip/tmp/dgtl_road_atlas.gdb.zip TRANSPORT_LINE
ERROR 1: Couldn't fetch requested layer 'TRANSPORT_LINE'!
Error: Process completed with exit code 1.

New data structure:

$ ogrinfo /vsizip/tmp/dgtl_road_atlas.gdb.zip
INFO: Open of `/vsizip/tmp/dgtl_road_atlas.gdb.zip'
      using driver `OpenFileGDB' successful.
Layer: VERSION (None)
Layer: DGTL_ROAD_ATLAS_MPAR_SP (Multi Line String)
Layer: DGTL_ROAD_ATLAS_DPAR_SP (Multi Line String)
smnorris commented 1 month ago

New layers have shortened, shapefile compatible names :(

smnorris commented 1 month ago

scheduled load has failed: https://github.com/smnorris/bcfishpass/actions/runs/8945871747/job/24575590978

smnorris commented 1 month ago

GeoBC confirms that what is currently posted is what we get:

the public DRA data model will not be changing any time soon. We will be updating the public data dictionary

This is unfortunate, given the shortened names in the posted data (that do not match the short names provided by the catalogue api) it is easier to just download from WFS using bcdata.

Download is straightforward but I'm not certain if it is simpler to update all DRA queries to use the new data model or add a script that maps WFS/BCGW column names to the existing column names based on the TRANSPORT_LINE layer.

smnorris commented 1 week ago

Current workaround is to use a cached DRA download from April, allowing modelled crossing rebuild with latest FTEN.