cha801p
commented
4 months ago Ticket Update: April 29, 2024 (7 PM)
Issue: Load sample data (https://collections-test.ala.org.au/dataResource/show/dr22666)
Solution: Successfully loaded the new dataset on biocache-test/events-test
Actions Taken:
- [x] Data review
- [x] Code review
- [x] Setup code locally
- [x] Ran code locally - FAILED
- [x] Sample data extracted - 98 records
- [x] Date format fixed
- [x] DwCA created locally
- [x] File uploaded to collector-test
- [x] Triggered pre-ingestion
Issues Encountered:
_File "/Users/cha801/PycharmProjects/databox/data-resources/dr21639-test-Reef-life-survey-to-event-core/main.py", line 371, in
- The code kept failing with the above error.
- Needs further investigation
Status: Sample data loaded on biocache-test/events-test Further investigate code/ Rewrite code Waiting for review from Peggy
peggynewman
Summary
Work with marine.csiro.ipt to create a regular harvest for Reef Life Survey data into the ALA (incl Events system). Doug's data transform is here: https://github.com/AtlasOfLivingAustralia/databox/tree/master/reef-life-survey for the events data but also find and replace the old RLS data resource.
Email trail
From RLS: Datasets 5 and 6 below are not species records and are not needed here. You could use 1-4 as the basis for then filtering if you like, but it may be safest to start with 1 and 3 to represent the update of what you had, as we are only just starting with correcting a large backlog of errors, and 2 and 4 have more of these and more taxonomic uncertainties. It’s up to you though, and probably best if you do have a system for updating the records on a routine basis anyway – as our correction of errors does often include effectively removing some species observations and replacing them with a different (correct) one.
Either way, you’ll still need to filter on ‘program’ to get Reef Life Survey, as these new layers also have the ATRC and Parks Vic datasets, which we spoke about last time we talked (and which are not something that I can talk to re putting onto the ALA, GBIF and OBIS).
From Peggy: We have applied an older RLS dataset into our pilot events system as we discussed last year, and are now looking at updating occurrence records in the ALA. We sourced this data from here: https://metadata.imas.utas.edu.au/geonetwork/srv/eng/catalog.search#/metadata/9c766140-9e72-4bfb-8f04-d51038355c59
This seems to be made up of two datasets,
We'd like to update the ALA now (and set up regular updates), and I think we spoke on the phone about how to filter datasets from the NRM data in the AODN to pull those that you were comfortable with going on to the ALA (and subsequently GBIF and OBIS), but I haven't got this documented anywhere. When I search on "RLS" in AODN, I get these results:
From Dave: This image is of the 5 datasets from the IMOS National Reef Monitoring Network Sub-Facility in our internal dataset caalogue. This data import is dated from March 2023, and I have downloaded today's data which contains more rows. This I can convert and load into DwC and publish to you and OBIS via IPT (another 6 million records) The conversion code is done and dusted and can be applied imediately.