Update dataverse plugin

NIAID-Data-Ecosystem / nde-crawlers

Harvesting infrastructure to collect and standardize dataset and computational tool metadata

Apache License 2.0

0 stars 0 forks source link

Update dataverse plugin #148

Closed NikkiBytes closed 4 days ago

NikkiBytes commented 5 days ago

In load_cache() the extraction of external metadata sources is removed, extracts now only the metadata from the dataverse not its external sources.
The handle.net resources ( a unique case ) is also adjusted with the change and does not extract the external source anymore.
Simple update to the parse() function to adjust with document load.
Added the extraction function to an external .txt for future reference if needed.

NikkiBytes commented 5 days ago

@jal347 @gtsueng I have an jupyter notebook if we need to go over some of the ways the documents look, plus I have lists of unique keys for the various data sources (harvard vs. the external data). But I didn't add any new keys, so it should still be schema.org compliant as before?