whole-tale / girder_wholetale

Girder plugin providing basic Whole Tale functionality
BSD 3-Clause "New" or "Revised" License
3 stars 5 forks source link

Annotate registered data with location labels #542

Closed Xarthisius closed 2 years ago

Xarthisius commented 2 years ago

What's new?

For Zenodo, Dataverse, DataONE and Globus providers this PR annotates registered data with a) location of a file/folder relative to the original dataset root b) identifier of root dataset (in case there's a specific file/folder identifier it's added as directIdentifer to metadata)

Why?

Currently this information is derived ad hoc from the structure of WholeTale Catalog using a lot of implicit assumptions regarding how registering works for a given provider. With this PR it's going to be easy to export / import external data regardless of its location in girder.

How to test?

  1. Register some data. I used:
    1. https://zenodo.org/record/3463499 (nested zenodo)
    2. doi:10.7910/DVN/5WTSUZ (nested DV)
    3. https://search.dataone.org/view/urn:uuid:6f5533ab-6508-4ac7-82a3-1df88ed4580e
    4. doi:10.18126/M2301J (globus)
  2. Inspect registered data in girder. Look for dsRelPath in item/folder metadata.

TODO

codecov[bot] commented 2 years ago

Codecov Report

Merging #542 (98da4b6) into master (0186a99) will decrease coverage by 0.19%. The diff coverage is 72.41%.

@@            Coverage Diff             @@
##           master     #542      +/-   ##
==========================================
- Coverage   92.87%   92.67%   -0.20%     
==========================================
  Files          58       58              
  Lines        4560     4575      +15     
==========================================
+ Hits         4235     4240       +5     
- Misses        325      335      +10     
Impacted Files Coverage Δ
server/rest/dataset.py 88.09% <ø> (ø)
server/lib/globus/globus_provider.py 62.16% <20.00%> (-1.39%) :arrow_down:
server/lib/dataone/provider.py 98.82% <100.00%> (+0.01%) :arrow_up:
server/lib/dataverse/provider.py 95.98% <100.00%> (-0.74%) :arrow_down:
server/lib/zenodo/provider.py 94.02% <100.00%> (-3.65%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 0186a99...98da4b6. Read the comment docs.