matthewcornell
commented
3 years ago Results of today's meeting with @nickreich and @elray1 :
Updated examples: Zoltar as_of query examples
Results: 1a. in uploading a version (issue_date), users can mark a previous row as "deleted" by passing NULL for its value = Q: first cannot be NULL? = Q: other business rules?
1b. as_of will exclude NULL value rows from query results for any as_of >= that issue date
- when importing a forecast, rows with duplicate values are skipped. key: (fm.id, f.tz_id, pred.u_id, pred.tid, <sample/cat/value* columns>) = Q: class Meta constraint? = https://stackoverflow.com/questions/13947327/to-ignore-duplicate-keys-during-copy-from-in-postgresql = https://stackoverflow.com/questions/31639108/best-way-to-prevent-duplicate-data-on-copy-csv-postgresql = https://stackoverflow.com/questions/17267417/how-to-upsert-merge-insert-on-duplicate-update-in-postgresql = https://www.postgresql.org/message-id/CA%2BbJJbwjOgyy5%2BRFb%2BCVNntWFrwZKekG-qhcRy65HAA7FyLJJQ%40mail.gmail.com = https://dba.stackexchange.com/questions/30499/optimal-way-to-ignore-duplicate-inserts = https://dba.stackexchange.com/questions/13468/idiomatic-way-to-implement-upsert-in-postgresql
Right now
as_ofoperates at the at the forecast/timezero/issue_date level, not factoring in the unit/target level. This works OK with our current covidhub convention whereby forecast updates are done for all unit/target combinations. However, when we implement county-level truth forecasts, the files will be too large (~16MB) to upload with duplicates. Therefore we want to update/patch only "diffs". However, that will expose the weakness of the currentas_ofimplementation. See the example at Zoltar as_of query examples.A prerequisite to: [Think about representing truth as an oracle model #280].