search

internetofwater / ref_gages

Reference Stream Gages
https://internetofwater.shinyapps.io/geoconnex-reference-features/
Creative Commons Zero v1.0 Universal
4 stars 4 forks source link

Need reference representations so we have something to cross reference to. #33

Closed dblodgett-usgs closed 3 months ago

dblodgett-usgs commented 1 year ago

Currently, the registry only includes a mapping from reference gage ID to provider/provider_id. There is no latest available representation tracked. We need to save the latest available representation such that we have something to use in the case that a provider's representation goes missing and so that we have something to cross reference to in the work flow.

ksonda commented 1 year ago

From ref_pws works like this:

  1. There is a catalog of versioned releases, with the last version duplicated as "latest". All releases include a field for date last successfully accessed from data contributor.
  2. Download current latest release
  3. Build new intermediate data product from existing contributor registry + new contributors for current round
  4. anti_join latest release with intermediate product
  5. Add anti_join to intermediate product
  6. intermediate product + anti_join creates new versioned release
  7. overwrite "latest" with the new version
dblodgett-usgs commented 1 year ago

This seems like what I was planning on hacking together, just haven't had the cycles to get to it.

dblodgett-usgs commented 3 months ago

Done in #31