search

scimma / blast

Django web app for the automatic characterization of supernova hosts
MIT License
1 stars 2 forks source link

Implement option to use S3-compatible object storage #240

Open manning-ncsa opened 5 months ago

manning-ncsa commented 5 months ago

The static cutout files downloaded for each transient could be more efficiently and economically stored using S3-compatible object storage, particularly in production deployments. Implement an option to enable object storage instead of filesystem storage. Currently the use of FileField types in the relevant Django models that store data, such as Cutout, should make this an easier task. Combined with the django-storages Django app, it should be possible to rewrite the read/write functions such that they are agnostic to the selected storage backend.

manning-ncsa commented 2 months ago

Before we dive in the code to implement a bunch of changes, it occurs to me that we should experiment first with s3fs to transparently mount the S3 backend to the local filesystem for Blast. I have tested this method on OSN and it works; the only question is whether the performance is sufficient.