Split NZVH data resource (dr2153) into individual data resources for each institution - Githubissues

AtlasOfLivingAustralia / data-management

Data management issue tracking

7 stars 0 forks source link

Split NZVH data resource (dr2153) into individual data resources for each institution #1002

Open nielsklazenga opened 7 months ago

nielsklazenga commented 7 months ago

Problem

AK and WELT want to deliver data directly to ALA but cannot do so because their data is already in the NZVH data resource
AK and WELT are not able to update their data in NZVH
ALA has not been able to upload the NZVH data set because of duplicate row keys.

Solution

Split the NZVH data set into individual data sets for each herbarium
Herbarium that has duplicate row keys in its data will have to do a new data delivery.

peggynewman commented 5 months ago

Dependent on https://github.com/AtlasOfLivingAustralia/preingestion/issues/228

peggynewman commented 2 months ago

Waiting on devops to allow access to EMR studio to manipulate UUID AVRO

rosemaryjoconnor commented 3 weeks ago

12/06/2024 Associated Freshdesk ticket: https://support.ehelp.edu.au/a/tickets/189866

New WELT data resource was created: dr26642

[x] Check against dr2153 - Allen Herbarium data
[x] Determine if dr26642 has new UUIDs
[x] If dr26642 has new UUIDs need to do avro updates from dr2153

rosemaryjoconnor commented 1 week ago

24/06/2024

Current status

Databox - Avros split and data updated

dr2153 - NZ Virtual Herbarium (790,126)
dr22717 - WELT (252,205)

Production

dr2153 - Allan Herbarium (1,005,799)
dr22717 - WELT (250,501)

To do:

[ ] Split Avros Production
[ ] Test locally double
[ ] Load and ingest Prod