Open jsheunis opened 1 year ago
Ideas for updating the frontend functionality.
A "catalog" contains all the html and JS assets for the frontend as well as the catalog entries that need to be rendered. The catalog assumes all JSON blobs of the metadata entries are located locally in the metadata
directory relative to the index.html
file. Metadata entries are in a hierarchy of nodes, a node could be of type dataset
or directory
. Here's an example of the hierarchy and a directory
node displayed:
In the hierarchy, each node location is identified by 1) the dataset_id
, 2) the dataset_version
and 3) a hash of the dataset_id
+ dataset_version
+ additionally the node_path
in the case of a directory
-type node. The entry blob (that is fetched via GET request) is inside the file with (part of) said hash in the filename.
The workflow is:
For purposes of rendering content in the frontend, there might also be a config.json
file on the dataset level (at the location my_catalog/metadata/dataset_id/dataset_version/config.json
). There is always a config.json
on the catalog-level (at the location my_catalog/config.json
). There's an existing and functional inheritance principle between catalog-level and dataset-level config: if not specified on dataset level during catalog entry generation, it get's inherited from catalog level.
Let's first write down a few assumptions about the future setup:
The first idea is to make as little as possible changes to the current structure. Therefore:
This means that a whole metadata
directory containing all node files will still be hosted with the catalog. It seems pretty similar conceptually to the git-annex way of replacing file content with symlinks, although here a file is replaced with another file which contains a URL to the actual file content.
The workflow is:
The implication for catalog generation:
my_catalog/metadata/dataset_id/dataset_version/node_file.json
The second idea assumes the granularity of catalog entry distribution and access is at the dataset-level:
dataset_id/dataset_version
config.json
on the dataset+version-level to specify the base URL of the relevant dataset
-node and all related directory
-nodes.This means that the metadata
directory that is hosted alongside the catalog only contains config files per dataset id and version.
The workflow is:
(again, some caching or other mechanism could be used so as not to require fetching same content multiple times)
The implication for catalog generation:
metadata
can be transported to a long-term location and subsequently removed from the catalog (although, importantly, the config.json
files at the dataset_id/dataset_version
path location should remain)???
This issue served its purpose as a discussion starter. The whole idea of separating the "catalog" (of metadata) from the "frontend" (e.g. browser-based viewer) is part of the design process of the continued work in https://github.com/psychoinformatics-de/shacl-vue and https://github.com/datalad/datalad-catalog/tree/revolution.
The context
Let's consider the role of
datalad-catalog
in the context of data discoverability. It currently has two main functions:Even though data discoverability via a browsable, user-friendly catalog is an important use for harmonized metadata, it is not necessarily the only use case, and the harmonization part should be able to happen irrespective of which frontend or client will ingest the catalog entries. E.g. it could be fed into a queryable graph-database.
On the front-end side, an important challenge that
datalad-catalog
needs to be able to deal with in this context is the need for different access URLs for different metadata entries. There are multiple datasets and files in a catalog and in order to allow varying degrees of access permissions (per user-dataset or user-file combination) these entries might be hosted at different locations, each with their own set of access-requirements.The current state
The two functions of data harmonization and browsable frontend are quite inter-dependent. E.g.:
What needs to happen