CSIRO Data Acess Portal

[M-Nicholls]

Review for applicable and open access data sets

[charvolant]

Contacted Anne Stevenson

• The DAP has recently implemented a tagging system (oddly enough, with the ALA in mind) so that datasets can be tagged as suitable for ingest by the ALA.
• Some datasets may have DwC terms as part of their metadata.
• The DAP has a web services API, documented on confluence under dmsdoc. Talk for David Benn or Dom Hogan for more information. We're asked to provide feedback on the documentation.
• One dataset is promising: http://dx.doi.org/10.4225/08/540F8063E8156 The plan for now is to see whether we can use that as an example for setting up automatic notifications of interesting datasets
• Automated translation into DwC is unlikely, since they're essentially ad-hoc data. But we can see what is available.

[charvolant]

DAP web services interface https://wiki.csiro.au/display/dmsdoc/Web+Services+Interface

[charvolant]

Contacted Chris Moeseneder re the Metallesthes data to use as a trial load via the data API

Moved Metallesthes data load to https://github.com/AtlasOfLivingAustralia/data-management/issues/47

[charvolant]

Developing DAP plugin for collectory to allow scanning of the DAP.

Requirements

• Most DAP entries are not useful for the ALA
• It requires a human to look at the entry and decise whether it can be included or not
• In most cases, the data provided will need to be manually mapped
• There needs to be some sort of tracability, so that the collectory keeps a record of what DAP datasets have been examined and
• All of this needs to integrate into the data management workflow

Implementation

• A grails plugin for the ala-collectory
• A separate table of DAP data resources, along with a lifecycle that can end up with either the resource being marked as not interesting or transformed into an actual data resource.
• Integration with the DAP web services API
• Integration with the GitHub API for creating issues. Issues should be created for each dataset an linked back to the collectory entries.
• The lifecycle is a state machine. To begin with, make this an embedded object model. If people want to get clever, they can make it something more sophisticated, with a database table, editors and the like, later.

User Interface

• A DAP instance is represented by an ALA data provider; new datasets are attached to the instance
• Editable pages for each candidate, similar to a data resource so that information can be edited
• The candidiate data resources make a complete copy of information needed by a data resource. This means that they can be edited without having to work out what changes have been made to the original data. (Additional functionality, a button to pull metadata from the DAP)
• Github issues are marked with labels showing the lifecycle state.
• Changing the state on the ALA causes changes to the github issues: the addition of a comment and changing of labels. Labels that are not part of the lifecycle should be left alone.

APIs

• DAP API https://wiki.csiro.au/display/dmsdoc/Web+Services+Interface
• GitHub API https://developer.github.com/v3/

[djtfmartin]

Sounds really good to me. Hopefully the integration with Github can be done behind an interface to hide details of github and so that switching to another issue tracker would be possible.

[charvolant]

Version v0.1 on git hub https://github.com/charvolant/data-repository-plugin/releases/tag/v0.1

[charvolant]

Informed Anne that plugin ready. The DAP now has tagging functionality, so searches can be made for "suitable for the ALA" tags. Not sure if there is an ALA tag just yet.

[peggynewman]

https://data.csiro.au/dap/ws/v2/tags/ALA DAP has a new API now. I'm keeping this ticket open! It's 8 years old now!