Existing Data Load : Frog ID

[peggynewman]

• [x] After creation, add this issue to the Data Management project board https://github.com/orgs/AtlasOfLivingAustralia/projects/9

Metadata

• [x] Locate the existing data resource in the collectory and check metadata: https://collections.ala.org.au/dataResource/show/dr14760
• [x] Check the unique identifier matches that in the new dataset

Data Prep

• [ ] Map dataset to Darwin Core Terms in a CSV if necessary http://rs.tdwg.org/dwc/terms/. Package the CSV into a Darwin Core Archive
• [ ] Ensure the dataset has correctly specified unique identifier without duplicates, and matches the unique identifier in the collectory connection parameters
• [ ] Ensure images are specified in an images file in the DwCA with the identifier field populated
• [ ] Check that species, location, times are adequately specific. Check back with the data provider if necessary
• [ ] Download existing DwCA from hdfs and merge the two DwCAs

Data Load

• [ ] Upload the new DwCA to the dmgt server
• [ ] Run the Ingest_dataset job on Jenkins, setting the data resource UID (DRNNNN) parameter http://aws-spark-quoll-master.ala:9194/job/Ingest-dataset/
• [ ] Note the statistics for total and new records from the end of the log file for the load by pasting the relevant lines here:

  [Log entries..... ]

  QA

• [ ] Wait overnight for next complete reindex
• [ ] Check that the number of records accessible match the loaded numbers using https://biocache.ala.org.au/occurrence/search?q=data_resource_uid:drNNNN
• [ ] Ask the data provider to review the new records using the same URL

[peggynewman]

The new dataset completely replaces the old, so there's no need to merge with existing data in the ALA. But I would still check that the identifiers are ok (as per the list above).

[peggynewman]

Waiting: data is now loaded and AM informed. Data provider is considering registering this dataset with GBIF via the ALA but it came through a different organisation before, so GBIF will have to change the linkage in the Registry first. Waiting to hear back from them.

[djtfmartin]

For notes, updating collectory required the following SQL (which has been ran on production collectory):

update data_resource
set
    gbif_registry_key = '47c9fee2-619a-481c-a114-386bc4748256', 
    gbif_doi = '10.15468/wazqft'
where 
    uid='dr14760';

[peggynewman]

This is now published on GBIF and all is well. https://www.gbif.org/dataset/47c9fee2-619a-481c-a114-386bc4748256