2023-2024 Mobilisation - TMAG - Githubissues

AtlasOfLivingAustralia / data-management

Data management issue tracking

7 stars 0 forks source link

2023-2024 Mobilisation - TMAG #1001

Closed rosemaryjoconnor closed 5 months ago

rosemaryjoconnor commented 11 months ago

Digitisation of specimen data from spider and other non-insect, non-crustacean, non-mollusc invertebrate specimens

rosemaryjoconnor commented 11 months ago

Fast-tracking delivery to the ALA of cleaned-up digital specimen data from spider and various other non-insect, non-crustacean, non-mollusc invertebrate specimens in the TMAG collections.

Delivery 15,000 new records to the ALA.

rosemaryjoconnor commented 11 months ago

19/12/2024

[x] Emailed to touch base re status

rosemaryjoconnor commented 10 months ago

30/01/2024

[x] Juliet emailed TMAG re update

rosemaryjoconnor commented 5 months ago

31/05/2024

[x] SG email indicated ready to send data sample - data received had no header row
[x] Dataset is a completely new Dataresource
[x] Sample of previous data format provided has been sent - not required
[x] Sample data successfully uploaded in databox - dr22716

rosemaryjoconnor commented 5 months ago

03/06/2024

Meeting with SG and KM. Discussed ongoing data updates - finalising with Mahmoud

Email follow up

To finalise the current mobilisation project:

Full dataset including approximately 20K new records will be delivered within the next week or so
I will take a look at the data in our test environment to ensure there are no major problems (I'm not envisaging that there will be)
We will load it into Production as soon as possible.
Juliet Seers can liaise with you regarding any contract/project finalisation details.
Just confirming that the dataset with approximately 2K invertebrate records is a likely a duplicate. I will confirm that all the records in that dataset, exist in the current full dataset load. Once that is determined we can remove the duplicate dataset.

rosemaryjoconnor commented 5 months ago

07/06/2024

[x] Confirmed that dr1840 are duplicate records. Dataset does not go to GBIF and is no longer updated
[x] TMAG indicated to remove dr1840
[x] Deleted dr1840 from test and production
[x] Final dataset due today or early next week - arrived 12/06/2024
[x] Confirm will still upload to dr347 as Incremental Load

rosemaryjoconnor commented 5 months ago

12/07/2024

[ ] Final datasets arrived - 5 CSV files
[ ] Needed code to run iconv over all files to remove hidden non-utf8 characters
[ ] Data requires a merge, but there are deletes

To do: Databox testin

[ ] Compare new and current CSV for deleted catalogNumbers.
[ ] Remove deleted catalogNumber records from full dr347 CSV and reingest

rosemaryjoconnor commented 5 months ago

26/06/2024

New invertebrates data in Prod - dr347

Previous record count: 204,897
New record count: 182,783

Invertebrates Collection

Previous record count: 190,911
New record count: 168,797
Records updated: 135,806
Records removed: 55,105
Records new: 32,991

rosemaryjoconnor commented 3 months ago

07/08/2024 One file of records missed being sent to us.

Now uploaded and live, images all fine too.
New record count: 224,534