Enchance download to process distributions separately

[eleftherioszisis]

Enhance download to allow processing more types of downloadable KG resources within reason.

Features

• Option to create links wherever possible.
• Handle both gpfs files and content only available via contentUrl.
• download() returns a dictionary with the downloaded paths.

Assumptions:

• Downloadable resources always have a DataDownload type.
• Downloadable resources always have a contentUrl attribute. This is true for all store types.
• Distributions with gpfs paths should satisfy distribution.atLocation.location.startswith("file:///gpfs")
• Distributions always have a name (questionable).

Restrictions

• More than one distribution with identical files (extension included) are not allowed

Questions

• Are there resources with contentUrl but no distribution?
• Are there other types of downloadable content that should be considered?
• Are other cases currently not taken care of (see below for the current list)?
• What do we do with the url entry. Is it sensible/expected to have downloadable content?
• Other features of interest? e.g filter by mime type, add user-provided target renaming, etc.?

Example Cases

Two distributions, but one is not like the other

https://bbp.epfl.ch/neurosciencegraph/data/1e2d3085-b591-4041-a33f-d8220cd72a61?rev=12

```python "distribution": [ { "@type": "DataDownload", "contentUrl": "https://bbp.epfl.ch/nexus/v1/files/bbp/neocortex/e3fbe8c5-dd8c-41b1-9953-e22738f1405b", "encodingFormat": "application/nwb", "name": "534524026.nwb" }, { "@type": "DataDownload", "repository": { "@id": "http://celltypes.brain-map.org/", "name": "Allen Cell Types Database" }, "url": "http://celltypes.brain-map.org/experiment/electrophysiology/534524026" } ], ```

Behavior:

• Distribution 1: contentUrl -> forge.download to output directory
• Distribution 2: No contentUrl -> skip

Distributions with atLocation but with a DiskStorage type of store

https://bbp.epfl.ch/neurosciencegraph/data/neuronmorphologies/a62417c8-704a-44fe-9088-854ed8bb2a35?rev=9

```python "distribution": [ { "@type": "DataDownload", "atLocation": { "@type": "Location", "store": { "@id": "https://bluebrain.github.io/nexus/vocabulary/diskStorageDefault", "@type": "DiskStorage", "_rev": 1 } }, "contentUrl": "https://bbp.epfl.ch/nexus/v1/files/bbp/mouselight/be3cf189-776e-4915-b27d-c5c84448b422", "encodingFormat": "application/swc", "name": "AA0390.swc" }, { "@type": "DataDownload", "atLocation": { "@type": "Location", "store": { "@id": "https://bluebrain.github.io/nexus/vocabulary/diskStorageDefault", "@type": "DiskStorage", "_rev": 1 } }, "contentUrl": "https://bbp.epfl.ch/nexus/v1/files/bbp/mouselight/1b77f913-dcd8-4706-a596-c6f7f6063d33", "encodingFormat": "application/asc", "name": "AA0390.asc" }, { "@type": "DataDownload", "atLocation": { "@type": "Location", "store": { "@id": "https://bluebrain.github.io/nexus/vocabulary/diskStorageDefault", "@type": "DiskStorage", "_rev": 1 } }, "contentUrl": "https://bbp.epfl.ch/nexus/v1/files/bbp/mouselight/bd1a34ad-5bc4-4856-b53a-144f74d3aac7", "encodingFormat": "application/h5", "name": "AA0390.h5" } ], ```

Behavior:

• Given that there is no gpfs path -> forge.download contentUrl

Distribution with a RemoteDiskStorage and atLocation.location gpfs path

https://bbp.epfl.ch/neurosciencegraph/data/d6fe7775-0f20-48f5-8819-825d392122d9?rev=1

```python "distribution": { "@type": "DataDownload", "atLocation": { "@type": "Location", "location": "file:///gpfs/bbp.cscs.ch/data/project/proj134/nexus/bbp/mmb-point-neuron-framework-model/0/1/3/d/d/d/a/8/bNAC.hoc", "store": { "@id": "https://bbp.epfl.ch/neurosciencegraph/data/10419671-da26-48d3-abf9-51eb54759ca4", "@type": "RemoteDiskStorage", "_rev": 1 } }, "contentUrl": "https://bbp.epfl.ch/nexus/v1/files/bbp/mmb-point-neuron-framework-model/1e7334b5-bc82-47b6-b963-f24144bb741f", } ```

Behavior:

• copy to output directory if create_links_if_possible is False
• create symlink to output directory if create_links_if_possilbe is True

[eleftherioszisis]

@mgeplf , @joni-herttuainen , @GianlucaFicarelli , please feel free to leave your feedback and discuss what directions would be interesting/useful for download().

[mgeplf]

I think this is good

The https://bbp.epfl.ch/neurosciencegraph/data/1e2d3085-b591-4041-a33f-d8220cd72a61?rev=12 downloads a file, but names it file; I have opened a bug for that here: https://bbpteam.epfl.ch/project/issues/browse/DKE-1094