search

IATI / refresher

A Python application which has the responsibility of tracking IATI data from around the Web and refreshing the core IATI software's data stores
GNU Affero General Public License v3.0
2 stars 0 forks source link

Unique auto-increment ID for validation #91

Closed akmiller01 closed 2 years ago

akmiller01 commented 2 years ago

This is working through a full refresher refresh, reload, validate, flatten, lakify, and solrize now. Creates auto-increment PK for validation and non-foreign-key publisher name so that re-uses of hashes don't overwrite validation, and deletion of publishers don't cascade delete validation:

# select id, document_hash, publisher_name from validation limit 10;
   id   |              document_hash               | publisher_name 
--------+------------------------------------------+----------------
     23 | 54cba8a8610e5eb231270cd090765212fc2e6ae8 | cku
 131854 | 21a4cd487af6a08564cd950c3b2b887c52f04c9c | sdi-int
 131855 | 22e8d0b81b065ecb6452c20829474fae829f49b8 | france
 131856 | 12a07a1e6357c297867f64ece41347493bd1c72d | unocha
 131857 | 59c6f632e070c4b63451ac2564c5f3eb8a938774 | unocha
     32 | 1c8edd9308a4ed8938943b1174f39d4aa879b39b | unocha
     33 | 9dec69d342589406d41ed80b5e88db84427f2cc8 | ifad
     35 | 327f3c8c9d67b4cf78863f245f96c239e0b67067 | civic_admin
 131858 | 68f590ab8bc8d09033d71519c19d63422e53697b | unocha
 131859 | 75dda1afbc934a02bd579f744444bc71db857c53 | solarnowsofa
nosvalds commented 2 years ago

Nice @akmiller01! I will try it out on my local setup tomorrow morning when I'm fresh and had a strong coffee.

One thing I will test tomorrow is this endpoint to non-destructively refresh a validation report: https://github.com/IATI/validator-services#pvt-patch-validation-regenerate-patch-pvtvalidationregenerate , did you test that yet?

akmiller01 commented 2 years ago

Just tested it now. Passing test in Postman image

Shouldn't be any ambiguous ID issues since it's not joining the validation table, and the validation column on document has changed to an integer now, from varchar, but it's still can be set to null.