TMAG - Investigate split dr347 to separate DRs

AtlasOfLivingAustralia / data-management

Data management issue tracking

7 stars 0 forks source link

TMAG - Investigate split dr347 to separate DRs #1071

Closed rosemaryjoconnor closed 6 days ago

rosemaryjoconnor commented 4 weeks ago

TMAG - dr347 is currently populated from 3 separate collections:

TMAG Bulk Samples Collection
TMAG Invertebrate Collection
TMAG Vertebrate Collection

Problems now arising with each of the collections being updated with new records and submitted to us standalone. That is, we'd need to merge into dr347 each time this happens. There is no easy way to manage this with one DR.

rosemaryjoconnor commented 4 weeks ago

04/06/2024 AVRO file processing

[x] Download UUID avro files for DR347

Write Script to:

[x] read avro files for dr347
[x] extract all UUIDs for records that are not 'invertebrate collection'
[x] update records with new dr values
[x] write to new avro file

rosemaryjoconnor commented 6 days ago

26/06/2024

TMAG dr347 not to be split

Process to update individual collections as they come in being finalised.
Latest invertebrates dataset from mobilisation complete