dandi / dandi-infrastructure

A repository to collect docs/issues on DANDI project infrastructure
Apache License 2.0
0 stars 6 forks source link

dandi-derived instance of DANDI? #172

Open yarikoptic opened 8 months ago

yarikoptic commented 8 months ago

Given that there could be a multiverse (large number) of derived dandisets, especially if we automate that, would it make sense to create an explicitly separate DANDI instance with/for them?

attn @satra - let's discuss on some use cases (e.g. spike sorting for all ephys data on dandi)

kabilar commented 6 months ago

Thanks Yarik. I would prefer to build this functionality into the DANDI Archive instance.

From a user perspective, having the data on the same platform would make it easier to navigate the original and derived Dandisets. We would need to spend time on the UI/UX to ensure that storing and navigating these derived Dandisets is streamlined.

kabilar commented 6 months ago

OpenNeuro mentioned that they are going to follow the GitHub model and allow users to fork datasets. Do we want to consider a similar model?

satra commented 6 months ago

this bit has a lot of different implications, and unlike openneuro datasets are much larger. this probably requires some discussion.

yarikoptic commented 6 months ago

OpenNeuro mentioned that they are going to follow the GitHub model and allow users to fork datasets. Do we want to consider a similar model?

note that in their case, their underlying dataset model is a git repository on github: https://github.com/OpenNeuroDatasets are the "source" (not just replica like we have for https://github.com/dandisets). So they can allow forking and merging "natively" . Moreover people already do that (e.g. those on https://github.com/OpenNeuroDatasets-JSONLD). The question becomes for them just to facilitate that forking on github, track available forks, and possibly allow for accepting "patches" back.

For us it would be an entire new development and figuring out what it would mean and how to organize the flow of contributions back.... But in principle we could also enable something like that in the scope of