kousu
commented
3 years ago Software Design
Global Architecture
bidirectional mirroring
│
│ ┌───────────────┐
┌────┼──────────────────────────▲│ │
│┼───────────────────────────────┤ data.site4.ca │
││ │ │
││ ├────────┬────┬─┘
││ ┌─────┴────────┘ │
││ │ │
││ │ │
┌───────────────┘▼┐ │ │
│ │ │ │
│ images.site5.ca │ │ │
│ │ │ │
└─────────────────┴─┐ │ │
│ │ ├──────mirroring
│ ├────notifying │
│ │ │
│ │ │
│ │ │
xxx│xxxxxxxxxxxxxxxxxxxxxx▼xxxxxx │
x┌─▼───────────────────────────┐x ▼
┌─────────────────┐ x│ │x ┌─────────────────┐
│ │ x│ data.praxisinstitute.org │x │imagerie.site7.ca│
│ spines.site6.ca ├─────────────────►│ │◄─────────┴─┬───────────────┘
│ │ x└─────────────────────────────┘x │
└──────────┬────┬─┘ xxxx▲xxxxxxx▲xxxxxxxxxxxxxxx▲xxxx │
│ │ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │ ┌───────────────┐
│ │ │ │ ├───────────────┼──────┘ data.site3.ca │
│ └───────────────────────┼───────┼───────────────┴───────────────┼──────▼─┬─────────────┘
▼ │ │ │ │
┌────────────────┐ │ │ │ │
│ ├─────────────────────────┘ │ │ │
│ data.site1.ca │ │ │ │
│ │ │ │ │
└─────────────┬─▲┘ │ │ │
│ │ │ │ │
│ │ │ │ │
│ │ ┌────────────┴┐ │ │
│ └─────────────────────┤ │ │ │
│ │data.site2.ca◄──────────────────────────────┼────────┘
└──────────────────────►│ │ │
└─────────────┘◄─────────────────────────────┘Per-Site Architecture
┌──────────────────┐ ┌──────────────────┐ ┌───────────────────┐ ┌───────────────────────────┐
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │
│ PACS ├─────┬──────► BIDS ├─────┬────►│ data.example.ca ├─────┬────►│ data.praxisinstitute.ca │
│ │ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ (data server) │ │ │ (portal) │
│ │ │ │ │ │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │
└──────────────────┘ │ └──────────────────┘ │ └───────────────────┘ │ └───────────────────────────┘
│ │ │
│ │ │
│ │ │
│ │ │
export scripts uploader notifier
written by each site. written by us written by us
(`git push`, `rsync -a`, etc) (cronjob: curl data.praxisinstitute.ca/notify ...)Components
It's easy to spend a lot of money writing software from scratch. I don't think we should do that. I think what we should do is design some customizations of existing software and build packages that deploy them.
Data Servers
I have two options in mind for the data servers:
GIN is itself built on top of pre-existing open source projects: Gogs, git-annex, datalad, git, combined in a customized package. We would take it and further customize it. It is a little more sprawling than NextCloud. Being focused on neuroscience, we could easily upstream customizations we design back to them to help out the broader community.
NextCloud is a lot simpler to use than datalad. You can mount it onto Windows, Linux, and macOS as a cloud disk (via WebDAV). It also has a strong company behind it, lots of users, good apps. It's meant for more general use than science; actually, it was never designed for science. It would be harder to share any improvements we make to it; though we could publish our packages and any plugins back to the wider NextCloud ecosystem. It has some other weaknesses too.
Uploading/Downloading
With GIN, uploading can be done with git:
- Follow https://github.com/neuropoly/data-management/blob/master/git-annex.md#new-repo, or something like it (these are our instructions; different sites might disagree; this is one of the ways GIN/datalad is a bit more ad-hoc and difficult)
- Do this:
git remote add origin git@data.site1.ca:my-new-dataset.git git push -u origin master # or, actually, maybe GIN forces you to first make the remote repo in its UI? Unsure git annex sync --content
Downloading is replacing the first two lines with git clone.
Windows and macOS do not have a git client built in.
With NextCloud, uploading can be done with davfs2 + rsync:
mount -t davfs2 data.site1.ca /mnt/data # something close to this anyway
rsync -av my-new-dataset /mnt/data/Downloading is just reversing the arguments to rsync.
There's also cadaver, and Windows and macOS have WebDAV built in.
Versioning
GIN is based on git, so it has very strong versioning.
There are git fsck and git annex fsck to validate that what's on-disk is as expected.
NextCloud only supports weak versioning.
But maybe we can write a plugin that improves that. Somehow. We would have to figure out a way to mount an old version of a dataset.
Permissions
NextCloud has federated ACLs built in: users on data.site1.ca can grant chosen users on spines.site6.ca access to specific folders A/, B/ and D/.
I am unsure what GIN has; since it's based on Gogs, it probably has public/private/protected datasets, all the same controls that Github and Gitlab implement, but I don't think it supports federated ACLs. Federation with GIN might look like everyone having to have one account per site.
But maybe we could improve that; perhaps we could patch GIN to supported federated ACLs as well. We would need to study how NextCloud does it, how GIN does it, and see where we can fit them in.
Sharing
In a federated model, data-sharing is done bidirectionally: humans at each site grant each other access to data sets, one at a time.
We should encourage the actual data sharing to happen via mirroring, for the sake of encouraging resiliancy in the network.
I think we should encourage the
Gitlab supports mirroring; on https://gitlab.com/$user/$repo/-/settings/repository you will find the mirror settings
Gitlab Example
We need to replicate this kind of UI in whatever we deploy.
Portal
For the portal, we can probably write most of it using a static site generator like hugo, plus a small bit of code to add (and remove) datasets.
The dataset list can be generated either by pushing or pulling: the data servers could notify the portal (this is how I've drawn the diagrams above) or the portal could connect to the data servers in a cronjob to ask them what datasets they have. Generally, the latter is more stable, but the former is more accurate.
It should be possible to keep the list of datasets, one per file, in a folder on the portal site, and have hugo automatically read them all and produce a big, cross-referenced, searchable index.
Packaging
We should provide an automated installer for each site to deploy the software parts. It should be as automated as possible: no more than 5 manual install steps.
We can build the packages in:
- ansible
- conda
- custom bash install scripts
- .deb
I think .deb is the smoothest option here; I have some experience using pacur, and we can use their package server to host the .debs. Whatever we pick, the important thing is that we deliver the software reproducibly and with as little manual customization as possible.
I think we should produce two packages:
praxis-data-server- the per-site softwarepraxis-index-server- the portal site
We might also need to produce uploader scripts; but because praxis-data-server will use standard upload protocols it's not as important to do so; moreover, because the uploader will be run directly by users, it will need to deal with cross-platform issues which makes it harder to package. I think, at least as a first draft, we should just document what command lines will get your data uploaded
Software Development Tasks
-
write specification for what a proper dataset looks like; probably BIDS, maybe with some extra restrictions
-
deploy data server prototypes
-
write data server customizations
-
write uploader scripts?
-
write PACS -> BIDS scripts that conform to that specification
- This is an ad-hoc, per site task. We can help, but we can't write these scripts ourselves.
-
write portal site
-
write data server -> portal notifier
-
sender
-
receiver
-
write packages
- data server
- portal server
-
write uploading documentation
-
write downloading documentation
-
write ACL documentation
-
Open Questions
- is
datalad(the GIN client) compatible with Windows? Is it compatible with Windows without using WSL? - Is there a way to build software support for attaching data sharing agreements to the permissions?
- I'm envisioning a situation where, next to a mirror of a dataset, its data sharing agreement .pdf is linked; perhaps even some of the terms, like expiry dates, can be programmed in.
- NextCloud or GIN? Or something else?
- Should we focus on trying to develop federated ACLs for GIN, or
Summary
We can build a federated data system on either GIN or nextcloud. Either one will require some software development to tune it for our use case, but much less than writing a system from scratch. Both support widely supported network protocols which makes them cross-compatible, reliable, and avoids the cost of developing custom clients.
jcohenadad
\
taowa
mguaypaq
+ 
https://praxisinstitute.org wants to fund a Canada-wide spine scan sharing platform.
They were considering paying OBI as a vendor, and having them set up a neuroimaging repository. But had doubts about the quality of that solution and have looked around for others, and have landed on asking us for help.
We've proposed a federated data sharing plan and they are interested in pursuing this line.
Needs