search

dandi / dandi-archive

DANDI API server and Web app
https://dandiarchive.org
13 stars 10 forks source link

minio option #1918

Open magland opened 2 months ago

magland commented 2 months ago

I noticed in the source code there is mention of Minio storage. Is that a working option - to use Minio rather than AWS?

I ask because we are thinking of creating an internal dandi instance at Flatiron. I will probably need some help exploring this option, so any resources you could point to would be helpful.

satra commented 2 months ago

@magland - we use the docker compose setup for testing and local development. this uses minio. so the testing workflows are probably good places to check it out.

@aaronkanzer is currently documenting the process to replicate dandi (we are running a version for a different project - as an internal repo) and we want to figure out how to simplify deployment of non-dandi instances. currently it requires a fair bit of hand holding and we are looking to abstract out pieces that can allow easier deployment. the docker use-case with minio is probably one of those. so if you do deploy at flatiron, do work with aaron to update the document.

magland commented 2 months ago

Thanks @satra !

@aaronkanzer It would be great if you could keep me updated as you make progress on that document. I'll hold off for now.

yarikoptic commented 2 months ago

FWIW, just to complement @satra's comment -- AFAIK nobody used minio backend fully in production yet. Would be great to see even simply on a sample deployment which would be toyed with on some data in "near real life" scenarios and see how it copes.

Meanwhile initiated https://github.com/orgs/dandi/projects/20/views/1 to collate issues etc relating to multi-instance deployment support.