AtlasOfLivingAustralia / data-management

Data management issue tracking
7 stars 0 forks source link

Load Redmap into databox from AODN #846

Closed peggynewman closed 2 months ago

peggynewman commented 1 year ago

Redmap data is in AODN. Eventually we will also harvest ReefLife Survey from AODN as well. Work out how to pull and convert the data to DwC. If we need to consult with the data provider, call on Andrew/Erin for a contact. Let's see the data in Databox but we should check in with the data provider before going live as a matter of courtesy.

peggynewman commented 1 year ago

Redmap data is in AODN. Eventually we will also harvest ReefLife Survey from AODN as well. Work out how to pull and convert the data to DwC. If we need to consult with the data provider, call on Andrew/Erin for a contact. Let's see the data in Databox but we should check in with the data provider before going live as a matter of courtesy.

cha801p commented 1 year ago

-Loaded the data on databox -DynamicProperties not showing up and needs an investigation

cha801p commented 1 year ago

-This has been raised further: https://github.com/AtlasOfLivingAustralia/biocache-service/issues/787

peggynewman commented 1 year ago

In test here: https://collections-test.ala.org.au/public/show/dr18714 Needs

I emailed Peter W (Redmap Data Manager) about this 04/04

cha801p commented 1 year ago

https://collections-test.ala.org.au/dataResource/show/dr18714

peggynewman commented 1 year ago

Still waiting to hear back from Peter W

peggynewman commented 1 year ago

Can script this using the WFS call to AODN

collection_url = "https://geoserver.imas.utas.edu.au/geoserver/redmap/wfs?typeName=redmap:RM_SIGHTING_VIEW_DATA&SERVICE=WFS&outputFormat=csv&REQUEST=GetFeature&VERSION=1.0.0&CQL_FILTER=INTERSECTS(GEOM%2CPOLYGON((91.494140625%20-49.3955078125%2C91.494140625%20-3.7802734375000036%2C175.25390625%20-3.7802734375%2C175.25390625%20-49.3955078125%2C91.494140625%20-49.3955078125)))&userId=Guest"

peggynewman commented 1 year ago

We can't use FID as the unique identifier for this dataset - whenever AODN reload their datasets into the geoserver, the FID changes. We need to use the ID field as the occurrenceID.