ROR affiliations/funders update

[slint]

• Make sure the datastreams workflow works (new and updated entries)

[ptamarit]

funders vocabulary testing

Issue with ROR data dump now containing 2 JSON files

This issue was fixed in #309

convert command

• The command works: $ invenio vocabularies convert -v funders -o v1.46-2024-05-02-ror-data.zip -t funders.yaml
• Data comparison
  • The default file only has 21 entries; a blog post explaining the addition of extra funders indicated that it is manually curated.
  • From the data dump, we get extra identifiers: isni, doi, grid
  • We now include title in other languages than en (e.g. fi: Suomen Akatemia and sv: Finlands Akademi for https://ror.org/05k73zm37)
  • Some names changed (e.g. 05k73zm37 is now Research Council of Finland instead of Academy of Finland, https://ror.org/05k73zm37 shows the "old" name as an alias, but we don't use them in the funders transformation)
  • The country appears before the id; order other than alphabetical can be achieved by setting sort_keys=False in yaml.safe_dump
  • 2 tabs indented instead of 4 tabs indented; did not manage to change this, but it could be done with Prettier.

import command

• The command works: invenio vocabularies import -v funders -o ../v1.46-2024-05-02-ror-data.zip
• Performance
  • The command takes more than 30 minutes on the first import (when only 21 entries exist and 100'000+ entries are inserted)
  • The command takes about 15 minutes on the second import (when all 100'000+ entries already exist and nothing is inserted)
• It shows warnings about existing entries and continues to proceed with next entries:

  FundersServiceWriter: ["Vocabulary entry already exists: {'id': '000000000', 'name': 'Name', 'country': 'CH', 'title': {'en': 'Name'}, 'identifiers': [{'identifier': '000000000', 'scheme': 'ror'}, {'identifier': '0000 0000 0000 0000', 'scheme': 'isni'}, {'identifier': '10.13039/000000000000', 'scheme': 'doi'}, {'identifier': 'grid.00000.00', 'scheme': 'grid'}]}"]

  • Using the update command instead of the import command should update existing entries instead of showing warnings (did not test it yet)

[ptamarit]

Closing for now since the ROR datastreams funders update works with the fix for the 2 JSON files linked to this issue. Other tasks like migrating affiliations to use datastreams, or upgrading to ROR v2 format, will be done as separated tasks.