updated background_layers.sh, with clipping

NewGraphEnvironment / dff-2022

Building digital field forms and processing data collected using R, postgresql, QGIS and other tools

Creative Commons Zero v1.0 Universal

0 stars 3 forks source link

updated background_layers.sh, with clipping #89

Closed smnorris closed 1 year ago

smnorris commented 1 year ago

pulls bcfishpass related datasets from pre-processed flatgeobuf files on S3, using attribute query to clip to AOI
uses attribute query to clip FWA data coming from bcgw/pgfs
for all other BCGW WFS data, clipping to AOI is done by ogr2ogr (slow but reliable)

smnorris commented 1 year ago

Datasets that seem reasonable to add to the cached set of fgb files:

fiss_fish_obsrvtn_events_vw (from bcfishobs)
mot_culverts_sp
gba_railway_tracks_sp
ften_road_section_lines_svw

My proposal is for these files to be refreshed weekly. The weekly replication to postgres is already mostly set up, this is not a lot of additional work.

In theory, if all data sources noted in bcfishpass/qgis were synced to this spot, these files could also be used by QGIS for general bcfishpass mapping (in place of a connection to a bcfishpass postgres database).

smnorris commented 1 year ago

for instructions on how to run scripts, see https://github.com/smnorris/dff-2022/blob/preclip_fgb/scripts/background_layers.md

smnorris commented 1 year ago

Only 3min to download all WFS data for entire Skeena, thats pretty good.

$ time ./bcdata.sh "'BABL','BABR','BULK','KISP','KLUM','LKEL','LSKE','MORR','MSKE','SPAT','SUST','TAKL','USKE','ZYMO'"
real    2m45.054s
user    0m30.070s
sys 0m6.408s

The clipping is still really slow though. I don't have a time for a complete Skeena run yet. Some other tool could probably improve performance if that is critical. I tried clipping in geopandas but that was similarly slow. QGIS does seem to have optimized clipping algorithms.