search

phac-nml / irida

Canada’s Integrated Rapid Infectious Disease Analysis Platform for Genomic Epidemiology
https://irida.ca
Apache License 2.0
40 stars 31 forks source link

Object store support #228

Open happykhan opened 5 years ago

happykhan commented 5 years ago

As our file storage increases I suspect we will want to move our file storage (particularly of fastq) to object store. Is this on the roadmap for IRIDA?

How are files fetched in IRIDA? Is it a few select methods that deal with files from the file system? It should be able to have it parse s3:// instead for instance.

tom114 commented 5 years ago

Hi @happykhan. Thanks for the suggestion. Object store support is something we've been thinking about but it would be a ways off. As we continue looking at cloud storage solutions it would definitely be something we need to address.

At the moment files in IRIDA are managed with regular linux file management. Part of this is to facilitate easy file sharing with Galaxy through NFS and a shared file system, and sharing with some custom workflows we have at some of our installations. In theory (famous last words) enabling something with S3 (or other object stores) shouldn't be super hard as most file management is handled within a few repository classes, but it would involve some changes with our communication with Galaxy.