GREI 5: HDV Task - Indigenous Data Support

cmbz commented 10 months ago

Overview

TK Label Working Group developed an NIH GREI grant proposal to support indigenous data sovereignty. See: IDS SUPPORT AND NIH-GREI GRANT
TKLWG is developing a proposed integration with Dataverse with the following goals:
- Link to and incorporate identified sources for indigenous knowledge representation, such as TKLabels and Notices
- Curation processes for managing the creation, reading, updating and deleting of metadata (including an data access workflow); and
- Present curated metadata (e.g. TKLabels and TKNotices) in catalog records
- Allow external aggregators to harvest this metadata (specifically the IDN Data Catalog, but preferably standardized model that allows for multiple external parties to harvest).

Tasks

[x] Confirm that the goals described in IDS SUPPORT AND NIH-GREI GRANT are the final goals for the use case.
[x] Update proposed integration based upon Working Group and user feedback
[x] Coordinate with Ramin Soleymani-Fard, member of the LICCI team at ICTA-UAB, to evaluate his current work to develop a Dataverse metadata block for Local Context metadata
[x] Install on demo
[ ] #316
[x] Decide best UX approach (preferred, Dataverse looks for LC projects with dataset DOI)
[x] Design approach
[x] Implement approach
[ ] Retest - invite community on Zulip and Google Community to test and provide feedback in Issue
[ ] Add to Guide: New Feature, How to Use, and Tooltips on production
[ ] Install on HDV and promote
[ ] #322

Resources

TKLabels: Design Options for Integrating with Dataverse (Document)
Local Contexts repository: https://github.com/localcontexts/localcontexts-metadata
https://github.com/IQSS/dataverse-pm/issues/124
DRAFT: Year 3 Realignment Proposal Plan (Document)
NIH GREI Aim 2 Yearly Progress (Year 2) (Document)
Integration proposal by JM

cmbz commented 7 months ago

Status: April 2024

Stakeholder and Dataverse meeting to review proposed integration of TKlabels with DV: https://docs.google.com/document/d/1umSncPhXApNcvstzqMbqIfx87LCB3RqwS69gPCvbDVE/edit?pli=1#heading=h.w5gbovoa191i
Initiated plans to join the TK Labels Service Provider Early Adopter Program

sbarbosadataverse commented 6 months ago

Status: May 2024

Jim provided responses to the questions on the integration proposal (see resources above) provided by Ramin and Janet
Initiated plans to join the TK Labels Service Provider Early Adopter Program - meeting scheduled for June (SB will try to join and invited Laura (Brazil) and Amber (Borealis) to join the meeting

cmbz commented 4 months ago

Status: July 2024

Met the week of July 15th, including Local Context representative
Ramin made a pull request to query the external vocabulare to get the values to populate the metadata block: https://github.com/gdcc/dataverse-external-vocab-support/pull/23.
Issue created to install on demo.dataverse.org:
- https://github.com/IQSS/dataverse.harvard.edu/issues/297
Next meeting is August 1st.

cmbz commented 3 months ago

Status: August 2024

Local Context controlled vocabulary support implemented on demo.dataverse.org
- https://github.com/IQSS/dataverse.harvard.edu/issues/297
Shared the announcement with TKLabel interest group and Brazilian Installation that has been waiting on this feature.
Shared with TKLabel working group and Brazilian Installations for testing

amberleahey commented 2 months ago

Thanks for adding this into the demo repository. I tested the LC metadata block and field for a dataset and noticed that when I search for term "name" I was able to make a linkage to a record in the LC Hub. May want to think about how to instruct people to use this, perhaps encourage them to visit the LC Hub for more information about registering projects for linkage before adding to Dataverse metadata? After adding, I wasn't able to delete the linkage it just appeared as greyed out once added. After saving, this did appear to retrieve the TK Labels image and name for display in DV Metadata. I couldn't figure out how to enable this field for display in the Terms or the main dataset landing page citation area for appearing alongside other dataset terms such as licenses.

qqmyers commented 2 months ago

FYI: While there were more issues than I expected, I've been able to update the Local Contexts script to address the problem of not being able to delete a project, and have made it configurable so test instances can point to the Local Contexts sandbox rather than their production site. I also managed to find a work-around to enable the Local Contexts information to appear on the main dataset page without having to wait for a change in Dataverse core (rather than just far down on the metadata pane).

I've updated https://docs.google.com/document/d/1umSncPhXApNcvstzqMbqIfx87LCB3RqwS69gPCvbDVE/edit?usp=sharing to include instructions on how to deploy the LC capability and to include more information about what has been fixed/improved in the script.

I'll push these changes to github next week and coordinate with @stevenwinship to deploy them to https://demo.dataverse.org .

qqmyers commented 2 months ago

Demo has been updated by doing the following:

Updating the LocalContexts External Vocabulary configuration: curl -X PUT --upload-file combined-cvoc-conf.9.3.24.json http://localhost:8080/api/admin/settings/:CVocConf (/home/rocky/combined-cvoc-conf.9.3.24.json has the same set of fields as the previous setting (from combined-cvoc-conf.json) with the changes for LocalContexts from the https://github.com/gdcc/dataverse-external-vocab-support repo.)
Adding display of LocalContexts info above-the-fold on the dataset page: curl -X PUT -d "dsDescription,subject,keyword,publication,notesText,LCProjectUrl" http://localhost:8080/api/admin/settings/:CustomDatasetSummaryFields
Addition of a published https://demo.dataverse.org/dataverse/LCtests collection for testing.
Turned off the LocalContexts metadatablock at the root collection level (mirroring how we expect this to be used - with LocalContexts only used on some child collections).

amberleahey commented 2 months ago

Looks great, I was able to remove the linkage in demo now! I'll check out the new testing collection in demo as well.

qqmyers commented 2 months ago

In the call today: Ashley showed us how DOI lookup almost works with their current code and she is going to make some updates in a week or so to allow it to work (two issues: they allow DOIs in multiple fields, but right now only the provider_id field is indexed, and, while they allow free form text, so we could get https://doi.org/10.5072/ABCDEF or doi:10:5072/ABCDEF etc, a search for 10.5072/ABCDEF doesn't work right now because they aren't breaking the strings up when indexing.) It sounds like searching per field will be allowed in v2, and there will be changes in the returned JSON, so we should expect some details to change. We also discussed how a link might be reported to DataCite or in the schema.org export, etc. and Ashley said she'd get us info on the draft recommendation for the DataCite mapping. It sounds like the schema.org mapping discussion is not as far along.

I said I would go ahead with the current script to add some accessibility improvements I just made for the ROR and ORCID scripts.
Ramin is going to create a PR to add a config change that will make it possible to add an LC Project id via the API.
When Ashley's changes are made, we can look at updating how input is done along the lines discussed in the design doc.
If getting a DataCite mapping in is a priority (i.e. if we want it in v6.4), we'll need to move very quickly as v6.4 is starting to get frozen (no new work).

cmbz commented 1 month ago

Status: September 2024

Demo.dataverse.org was updated with new TK labels metadata block, see:
- https://github.com/IQSS/dataverse.harvard.edu/issues/303
- https://github.com/gdcc/dataverse-external-vocab-support/issues/24
Working group members continued to test and provide feedback on the current implementation on demo.dataverse.org

sbarbosadataverse commented 18 hours ago

-OSF met with DV team to discuss the implementation of this support -The tasks were updated to include a broader retest by the DV community and documentation work for the Guide. An invite to test and input feedback will be added to Zulip and DV Google Group, with a deadline. -Consider engaging with the DDI community around Copyright field use in DV

IQSS / dataverse-pm