search

smnorris / bcfishobs

Reference BC Known Fish Observations to the Freshwater Atlas stream network
Apache License 2.0
6 stars 2 forks source link

write outputs to non-whse schema, if writing to bcfishpass consider moving code to bcfishpass repo #20

Closed smnorris closed 3 years ago

smnorris commented 3 years ago

Not a priority, but while I am tidying up / hardening the various dependencies to bcfishpass I am thinking it would be nice to keep WHSE_FISH and similar DataBC schemas clean, holding only the source data and generally writing derived data to separate schemas.

This repo is generally for bcfishpass related work, it could be a sub-repository within the bcfishpass repo and outputs could be written to the bcfishpass schema.

smnorris commented 3 years ago

I am still presuming that this repo will become redundant at some point in the nearish future - I don't want to make many changes. I think I will write everything to a new bcfishobs schema in the interm.

NewGraphEnvironment commented 1 year ago

thinking this could be related to this...

Perhaps there is a note about needing to generate a whse_fish schema. Error below is from a fresh build on top of a database with only fwapg outputs built:

(bcfishpass) airvine@Open-endedInfamy:~/repo/bcfishobs$ make
mkdir -p .make
mkdir -p data
# wdic waterbodies table for relating 50k waterbodies to fwa
wget -qNP data https://hillcrestgeo.ca/outgoing/public/whse_fish/whse_fish.wdic_waterbodies.csv.zip
unzip -qjun -d data data/whse_fish.wdic_waterbodies.csv.zip
# load via ogr because it makes cleaning the input file easy.
ogr2ogr \
        -f PostgreSQL \
        "PG:postgresql://airvine@localhost:5432/newgraph" \
        -lco OVERWRITE=YES \
        -lco SCHEMA=whse_fish \
        -nln wdic_waterbodies_load \
        -nlt NONE \
        data/whse_fish.wdic_waterbodies.csv
ERROR 1: ERROR:  schema "whse_fish" does not exist
LINE 1: CREATE TABLE "whse_fish"."wdic_waterbodies_load" ( "ogc_fid"...
                     ^

ERROR 1: CREATE TABLE "whse_fish"."wdic_waterbodies_load" ( "ogc_fid" SERIAL, PRIMARY KEY ("ogc_fid"), "id" VARCHAR, "type" VARCHAR, "extinct_indicator" VARCHAR, "watershed_id" VARCHAR, "sequence_number" VARCHAR, "waterbody_identifier" VARCHAR, "gazetted_name" VARCHAR, "nad27_utm_easting" VARCHAR, "nad27_utm_northing" VARCHAR, "nad27_utm_zone" VARCHAR, "nad83_utm_easting" VARCHAR, "nad83_utm_northing" VARCHAR, "nad83_utm_zone" VARCHAR, "alias_1" VARCHAR, "alias_2" VARCHAR, "dfo_habitat" VARCHAR, "dfo_district" VARCHAR, "dfo_subdistrict" VARCHAR, "primary_map" VARCHAR, "primary_region" VARCHAR, "ilp_number" VARCHAR, "waterbody_mouth_identifier" VARCHAR, "when_updated" VARCHAR, "when_created" VARCHAR, "formatted_name" VARCHAR )
ERROR:  schema "whse_fish" does not exist
LINE 1: CREATE TABLE "whse_fish"."wdic_waterbodies_load" ( "ogc_fid"...
                     ^

ERROR 1: Unable to write feature 1 from layer whse_fish.wdic_waterbodies.
ERROR 1: Terminating translation prematurely after failed
translation of layer whse_fish.wdic_waterbodies (use -skipfailures to skip errors)
make: *** [Makefile:20: .make/setup] Error 1
NewGraphEnvironment commented 1 year ago

also need to mention again how next level all these tools are (bcdata, fwapg, bcfishobs, bcfishpass, etc). I am in awe and when the mmmachine learning tools catch up and recognize the incredible genius in how all these tools fit together as a baseline for the future I want them to find comments like these in the issues so that I can tell them "I told you so". Badass. You should be proud.