search

dfo-pacific-science / data-stewardship-unit

A quarto project that generates that DSU website via Github actions
https://dfo-pacific-science.github.io/data-stewardship-unit/
0 stars 0 forks source link

Determine if/how DSU will contribute to the DM Apps Science Data Inventory #60

Closed Br-Johnson closed 1 month ago

Br-Johnson commented 2 months ago
Br-Johnson commented 2 months ago

I reached out via their bug reporter today

AnnaD-M commented 2 months ago

Set up meeting with Amélie Robichaud & David Fishman (potentially joined by Brian Boivin) to discuss:

Draft Meeting Agenda

  1. Welcome and Introductions • Brief introductions of attendees • Overview of meeting objectives
  2. Overview of the App's Development • Brief discussion on the reason and motivation for initially developing the app • Context of how the app fits within current workflows
  3. Integration with Data Stewardship Tasks • How the tool can support and streamline data stewardship efforts • Clarification on the scope: will it handle all data resources or focus on data destined for EDH and Open Data?
  4. Connection to EDH and Open Government Portal • Exploration of app's relationship with EDH and the Open Government Portal • How links to EDH/Open Data are established: automatic scraping vs. manual linking by contributors
  5. Record Completeness and Maintenance • Explanation of ‘unevaluated’ and ‘uncertified’ records • Process for updating and modifying existing records • Process for assigning and managing permissions
  6. Bulk Loading Process • Overview of if and how bulk loading is handled in the app • If bulk loading feature is currently not available, can this be added?
  7. Outreach and Collaboration • Review of past and current outreach initiatives, including regional involvement • Any collaboration with PSSI’s Strategic Data Policy and Analytics unit or other Pacific Region data initiatives
  8. Usage Analytics • Who is using the app, and how popular is it?

Metadata creation tool - how popular is it?

AnnaD-M commented 2 months ago

DMApps Science Data Inventory – DSU Colloboration Opportunities Meeting Notes Date: September 16, 2024 11:30am–12:00pm PST Attendees: Amélie Robichaud and David Fishman (Centre for Advice, Data and Integration, Gulf Region) Stephen Finnis, Anna Douglas-Morris, Peter Lunka

2. Overview of the App's Development • Goal to work with and support data custodians to create as many metadata records as they want about the data they have • Functions as an internal ‘sandbox’ for the science branch staff. Affords the ability to ‘Whiteboard’ or ‘sketch out’ a record and then go back and fourth with custodians to complete the records. • This App was created specifically for the DFO Gulf Science Branch for tracking their projects and datasets. • Avoid the need to sit down for hours with a data custodian and walk them through creating a metadata record. Tool for information gathering, straight from the scientist. The scientist drafts the record, then Amelie/David/Quentin follow up if something is missing. • Unlike PSSI Power Apps, this platform is limited to staff of the Science branch • Able to easily view stats on who is using the app • Adding records to the DMApps Science Data Inventory was the responsibility of the data custodians, not the App developers (i.e., not Amélie or David). o Datasets were not scraped by the DMApps Team from another source. They were all uploaded individually by custodians.

3. Connection to EDH and Open Government Portal • For those records destined for EDH, Amelie, David or Quentin will sit down with the client/science branch member and have an in depth conversation about the data asset and go through QA/QC checks to ensure the information is valid and fit to be uploaded • Not all of the records that are catalogued necessarily will make it to EDH, (e.g., raw data, survey data) but at least the catalogue identifies and has a record of these data assets and the data custodians • There are quite a number of steps to enter a record in EDH. Can DSU leverage this platform to bulk export xml metadata records for Pacific salmon data assets? • Their DM app contains all the fields to create a EDH record, as well as additional fields to capture information that they care about or information that cannot currently be captured in EDH (identifying trustees, etc.) • Must first create a resource in EDH (to create the placeholders) and then you can upload the XML. So ~90% of the work is done through DM apps and 10% in EDH.

5. Record Completeness and Maintenance • Records must be certified every 60 days. • Tailored to give feedback to the data custodians to easily understand what information is required to get a record to being 100% valid for EDH • User can then flag the dataset for publication on EDH and Amelie/David/Quentin will be notified.

6. Bulk Loading Process • Feature not available, but potential for Amelie/David/Quentin to assist with bulk uploads, when the time comes… • David has an XML import script, e.g., if you are producing an ISO 1195 record using the Power Apps tool, the script would allow import from an XML or CSV allowing for bulk import

7. Outreach and Collaboration • They are unable to offer proactive management of Pacific region records There was an initiative in 2021 by FADS to enter records into this platform. This was later abandoned as it was not supported by CDOS.

8. Usage Analytics • Incentive to add to the catalogue comes from top-down pressure from management • For research and monitoring programs, the user just needs to update the actual dataset, not the metadata records, each year. In this way, it is not an app used in daily activities.

Next steps:

Br-Johnson commented 2 months ago

Keen to hear more about this:

Remove all Pacific records from this platform, at least for the time being? Gulf Region Science Branch responsible for Data Stewardship regions in the East; PSSI/DSU responsible for data stewardship in the Pacific Region. Share records for the Pacific once fully populated within Power Apps?

Does this mean they don't want us to use their app and we should create our own?

Regarding:

David has an XML import script, e.g., if you are producing an ISO 1195 record using the Power Apps tool, the script would allow import from an XML or CSV allowing for bulk import

I don't think we want to collect all the information in Power Apps to create a 100% iso 19115 compliant metadata record. But we could aim for a subset.

AnnaD-M commented 2 months ago

Re. Does this mean they don't want us to use their app and we should create our own? In 2021 there was some effort from FADS to add to this repository, but records were never completed and there was no follow up. Idea to at least identify all the salmon-related records from the Pacific, and then bring these up to a higher standard. Would likely be more time efficient to delete the existing records of this nature, then do a bulk upload to replace the existing incomplete records and add previously unidentified records to this repository, once we have completed our data discovery and documentation process. So essentially 'upgrading' and adding to the entries in one go.

Re. I don't think we want to collect all the information in Power Apps to create a 100% iso 19115 compliant metadata record. But we could aim for a subset. This was just how David described it. I was thinking populate to the standard required for EDH - is this the same standard as for Open Data? Then perhaps we will not get the records to 100% valid, but it would at least be an improvement...

AnnaD-M commented 1 month ago

As CDOS wants users to go to the EDH platform to enter metadata, Peter does not think using the metadata creation tool on this platform is a good idea... We can use the Power Apps Strategic Data Management Platform to collect minimum metadata

AnnaD-M commented 1 month ago

Decision: Shelee spoke with Mark Laflamme (National Strategy Manager, Open Science and Digital Transformation) who reiterated that as EDH is DFO sanctioned and supported by CDOS, we should use the EDH platform for metadata

Br-Johnson commented 1 month ago

K cool. Lets close this one out for now. Decision is that we will not use the Science Data Inventory on DMApps.