remove redundant data - Githubissues

smnorris / bcfishpass

Model and monitor aquatic habitat connectivity in BC. Tools to plan and prioritize the assessment and remediation of barriers.

https://smnorris.github.io/bcfishpass

Apache License 2.0

8 stars 13 forks source link

remove redundant data #530

Open smnorris opened 2 months ago

smnorris commented 2 months ago

Examine fix tables and remove records that do not actually apply a fix.

Note that this could also be done as an action on PR but that isn't worth implementing until we have better docs on what fixes go where.

smnorris commented 2 months ago

Can user falls table be removed entirely? Probably not but it should be checked against CABD https://github.com/smnorris/bcfishpass/blob/v0.5.0/data/user_falls.csv

smnorris commented 1 month ago

Can user falls table be removed entirely? Probably not but it should be checked against CABD https://github.com/smnorris/bcfishpass/blob/v0.5.0/data/user_falls.csv

Feedback is that CABD will include Bonnington falls but I'll just leave the table as is for now.

Next redundant data clean should be user_barriers_definite_control.csv:

remove any records for falls where barrier status matches CABD barrier status UNLESS there are many observations upstream and the fix is to ensure that the falls is retained as barrier despite these observations
use CABD id when referring to a falls

Note that for falls, the usage of this table is now just to prevent falls with observations upstream from being ignored. Data and tables could probably be reorganized/renamed.

smnorris commented 1 month ago

For CABD features I'm thinking a table like below, supporting:

both dams and falls in one table (but I'd presume most/all records would be to falls)
barrier status adjustments
a boolean flag pinning existing CABD barrier status to ensure it is not overridden by upstream observations
linking of CABD feature to correct FWA blue_line_key
tracking what falls need adjustment in source CABD db (rows can be removed when fixed)

cabd_id
cabd_feature
barrier_ind
barrier_pin_ind
blue_line_key
reviewer_name
review_date
source
notes
source_fix_requested

smnorris commented 1 month ago

A barrier_pin column might have to be per species? Or we could presume that for falls, any pin related to observations is for salmon/steelhead only.

Currently, if a falls is flagged as a barrier in this table (based on salmon review, such as falls on blue_line_key=356364051, Herrick Ck), the stream is excluded from BT habitat output - regardless of how many observations are upstream. I'll treat this as a bug in the BT model for now but the fix table could be adjusted.