Data load : Wingecarribee Shire Council Macroinvertebrate Sampling

peggynewman commented 3 years ago

Data in Support desk ticket: https://support.ehelp.edu.au/a/tickets/97728 DR: https://collections.ala.org.au/public/show/dr7949

[x] After creation, add this issue to the Data Management project board https://github.com/orgs/AtlasOfLivingAustralia/projects/9
[x] Map dataset to Darwin Core Terms if necessary http://rs.tdwg.org/dwc/terms/
[x] Upload the dataset to collections.ala.org.au
[x] Load/sample/process dataset http://aws-scjenkins.ala:9193/job/Parameterised%20Load%20Sample%20Process/
[x] Note the statistics for total and new records from the end of the log file for the load by pasting the relevant lines here:
```
[Log entries..... ]
```
[x] Wait for next complete reindex
[x] Check that the number of records accessible match the loaded numbers using https://biocache.ala.org.au/occurrence/search?q=data_resource_uid:drNNN
[ ] Ask the data provider to review the new records using the same URL

patkyn commented 3 years ago

Before load 427 records Loaded up dataset.

starRecord.core().rowType()=http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord
aws-bstore-4b 2021-01-07 15:51:27,568 INFO : [DataLoader] - Only loading multimedia for specific core record types, starRecord.core().rowType()=http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord
aws-bstore-4b 2021-01-07 15:51:27,578 INFO : [DataLoader] - Only loading multimedia for specific core record types, starRecord.core().rowType()=http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord
aws-bstore-4b 2021-01-07 15:51:27,583 INFO : [DataLoader] - Finished DwCA loader. Records loaded into the system: 73, records skipped: 0, new records: 73
aws-bstore-4b 2021-01-07 15:51:27,725 INFO : [DataLoader] - Registry response code: 200
aws-bstore-4b 2021-01-07 15:51:28,304 INFO : [DataLoader] - Registry response code: 200
aws-bstore-4b 2021-01-07 15:51:28,304 INFO : [Loader] - Completed loading resource: dr7949. Completed in 2.675seconds (0.04458333 minutes)
Finished: SUCCESS

patkyn commented 3 years ago

After load, there were 500 records.

New records that are loaded: https://biocache.ala.org.au/occurrences/search?q=data_resource_uid%3Adr7949&fq=last_load_date:[2021-01-07T00:00:00Z+TO+*]

However, it looks like the map is not displayed for the occurrences.

For example: https://biocache.ala.org.au/occurrences/e8f2d1b5-27ba-49bb-a050-5f6b7757636f The Geospatial info decimalLatitude and decimalLongtitude values are not correct (should not contain °S and °E)

Fixed up the data to reload.

starRecord.core().rowType()=http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord
aws-bstore-4b 2021-01-08 16:19:03,762 INFO : [DataLoader] - Only loading multimedia for specific core record types, starRecord.core().rowType()=http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord
aws-bstore-4b 2021-01-08 16:19:03,763 INFO : [DataLoader] - Only loading multimedia for specific core record types, starRecord.core().rowType()=http://rs.tdwg.org/dwc/xsd/simpledarwincore/SimpleDarwinRecord
aws-bstore-4b 2021-01-08 16:19:03,768 INFO : [DataLoader] - Finished DwCA loader. Records loaded into the system: 73, records skipped: 0, new records: 0
aws-bstore-4b 2021-01-08 16:19:03,917 INFO : [DataLoader] - Registry response code: 200
aws-bstore-4b 2021-01-08 16:19:04,490 INFO : [DataLoader] - Registry response code: 200
aws-bstore-4b 2021-01-08 16:19:04,490 INFO : [Loader] - Completed loading resource: dr7949. Completed in 2.756seconds (0.045933332 minutes)
Finished: SUCCESS

patkyn commented 3 years ago

Maps are appearing now on the latest load after update the decimalLatitude and decimalLongtitude https://biocache.ala.org.au/occurrences/search?q=data_resource_uid%3Adr7949&fq=last_load_date%3A%5B2021-01-07T00%3A00%3A00Z+TO+*%5D#tab_mapView

AtlasOfLivingAustralia / data-management

Data load : Wingecarribee Shire Council Macroinvertebrate Sampling #638