clean up and formalize the datastream harvester

[fostermh]

DataStream Metadata Comparison

Issues to resolve:

• [x] 1. Title[french] - All datastream records are missing French version of the title Current Action: Auto translate
• [x] 2. Notes[french] - All datastream records are missing French version of notes (description) Current Action: Auto translate
• [x] 3. Keywords[french] - All datastream records are missing French version of keywords Current Action: Auto translate
• [x] 4. Keywords[english] - Some datastream records contain no keywords Current Action: Default to “other”
• [x] 5. EOV - Some matching eov’s in datastream keywords but many are missing any matches
  1. Nutrients (20)
  2. Nitrate (3)
  3. Phosphate (3) Current Action: Default to “other”
• [x] 6. Resources - Datastream does not provide a download link for the data in there metadata record Current Action: Resources set to url of datastream metadata page
• [x] 7. Frequency of Update - Datastream does not appear to provide this field Current Action: Default to “As Needed”
• [x] 8. Vertical Extent - Not populated in DataStream metadata Current Action: ignored as it is not a required field
• [x] 9. Spatial Extent - Is available in DataStream metadata
• [x] 10. Organization - can be created from datastream metadata and every datastream record has an owner organization. Only the name of the organization is available however so organization metadata in CKAN is somewhat lacking. It would be great if the organizations had ROR or URL's attached so that we can uniquly identify them throughout the system. Current Action: Default the org to DataStream or create orgs with title only. Manually creating the orgs in ckan before harvesting is the other alternative. Dependent on 18 being fixed.
• [ ] 11. Cited Responsible Party - Individuals entered in DataStream metadata in the organization's field sometimes concatenated with the organization name as well E.g. organization-name = “Derek Gray, Wilfrid Laurier University”
• [x] 12. Metadata Point of Contact - is populated in DataStream metadata and is how the owner organization is determined. Only the organization name and email are filled out in most (all?) cases.
• [ ] 13. Licenses - Datstream uses the following unsupported licenses, several more are supported
  1. https://open.canada.ca/data/en/dataset/82927831-131a-433e-af26-745b9930cd65
  2. https://open.canada.ca/en/open-government-licence-canada.
  3. https://www.grca.on.ca/wp-content/uploads/2020/12/Ganaraska-Region-Conservation-Authority-Open-Data-Licence-V1.pdf
  4. ogl
• [ ] 14. XML files fail iso validation Current Action: skip validation.
• [ ] 15. Some keywords are a list of semicolon-separated strings rather than being there own XML entities. Currently Action: treat as one keyword and translate as needed.
• [ ] 16. some license URLS have an extra period '.' at the end. See 13
• [ ] 17. some license URLS are broken or wrong. See 13
• [ ] 18. Some organization names contain extra characters such as commas or spaces at the end of the string.
• [ ] 19. Duplicate organization names with different formating exist. will need to map them to the same organization in the harvester or fix in the xml. e.g. ECCC vs Environment and Climate Change Canada

[fostermh]

we will support automated translation during harvest. need to figure out how to do this without overloading the translation service.

[fostermh]

the automatic translation appears to be working. DataStream XML still fails validation mostly due to the order of fields in the XML. Attached are an original XML file from DataStream and a 'fixed' version that passes validation in Oxygen.

Original (failing ISO validation)

c6e71281-b732-47e7-8c24-f9430d7edf26.iso19115.xml.txt

Corrected (passing ISO validation)

c6e71281-b732-47e7-8c24-f9430d7edf26.iso19115.FIX.xml.txt

[willfarrell]

DataStream v3: We're actively working on the next version of DataStream to address a few items from your list. This includes allowing the entering metadata in english and french, changing organization from a simple text input to a proper object (to remove duplicates, attach RoR/URL and separate out individuals), along with many more improvements.

1-4: Will be supported in v3 5: What does EOV mean? 6: Currently all download links are signed URLs for security. There is a backlog item to allow direct download, currently a request to the API is required. 7: We do collect this, but may not be in the ISO 19115 template. I've created an issue to add this in. 8: We do generate this, but may not be in the ISO 19115 template. I've created an issue to add this in. 10: More organization metadata will be available in v3 11: Individuals will be separated in v3 12: data steward email is a required field, exists for all datasets 13: I've created an issue to address this. 14: I've created an issue to address this. I didn't know order in the xml was important to the standard. Any other validation issues you've seen? Is there an online tool to test this? I'll use your example to apply suggested changes. 15: I've created an issue to address this. 16: I've created an issue to address this. 17: I've created an issue to address this. We'll reach out to group that are affected by this. 18: I've created an issue to fix what I can now, the rest will have to be addressed in v3 19: I've created an issue to fix what I can now, the rest will have to be addressed in v3

I should be able to address the ones I've created an issue for this week.

[fostermh]

I recommend oxygen as a XML validator/editor you can get a free 30 day trial by providing an email address. It will read the namespaces in the XML header and run the appropriate validation for you. no need to download xsd's and such.

EOV stands for 'Essential Ocean Variables' basically it's how we group datasets in CIOOS. You can see them listed on the main page of the national catalogue. We likely need to add more options to capture freshwater sampling. The EOV's are populated by matching keywords.

[willfarrell]

Update: Bunch of fixes were deployed last week. All ISO 19115 files have been rebuilt over the weekend. I just need to run a few through the validator and make any necessary tweaks.

[willfarrell]

7. Should be good to go
8. Should be good to go
9. Should be good to go, spot tested a few - let me know if you find any more
10. Fixed
11. Fixed
12. Still waiting to hear back from data steward on a fix for their license, rest fixed
13. Should be good to go
14. Some fixes applied, rest may have to wait for v3

Let me know if there is anything else that I can address in the near term.

[fostermh]

Great! 271 datasets now pass validation and are harvested into my local test environment successfully. I have attached the most recent harvest report. 8 of the sitemap url's appear to be broken which leaves only 12 datasets with outstanding problems. Note that I thought I could split keywords on semicolons during harvest but it's not working out so keywords will need to be formatted as one keyword per xml keyword tag in v3, if possible.

datastream_harvest_job_report.json.txt

[willfarrell]

Thanks, I'll take a look through the list. At first glance it looks like some are due to cache, but do see some new ones.

[willfarrell]

More fixes pushed, hopefully that should do it. I have a fix for the sitemap too, but can't be deployed just yet. The json report was very helpful.

[fostermh]

I think, for now, everything is working as expected. harvester has been deployed to the development servers awaiting review. I'm going to close this issue for now. we can reopen if new problems are found.