search

dirac-institute / trailblazer

An open data repository for astronomical data products affected by satellites
MIT License
8 stars 2 forks source link

Implement S3 storage. #4

Open DinoBektesevic opened 3 years ago

DinoBektesevic commented 3 years ago

Abandon the "static" storage workarounds we have now in favor of hosting data on S3.

Can this be made to load from the browser-side and not go through server-side or will we have to give too many access privileges to our buckets/objects?

spenczar commented 3 years ago

I would recommend against this, and for using WhiteNoise instead. See http://whitenoise.evans.io/en/stable/#shouldn-t-i-be-pushing-my-static-files-to-s3-using-something-like-django-storages.

DinoBektesevic commented 3 years ago

Oh sorry, this is my mistake because I did a drive by and opened a bunch of issues today. I should have been clearer when explaining them.

As it stands, for faster turnaround I store all uploaded fits files to static/upload/fits. This is a no-no. I store the thumbnails into media dir. This can pass.

Fits files need to be stored in S3, otherwise we will soon-ish (hopefully) be renting some very large drives to keep up with demand. This is why static is in quotation marks. These aren't your classic files, these files shouldn't be in "static".
It helpful when debugging because you can see them from the browser and that's essentially why they are there now.

I was considering moving thumbnails to S3, but given your link - perhaps we shouldn't? Do you think it's smarter to leave them in media? They are about 4 times smaller than the original image, with some leeway to make them even smaller, but they could grow to be sizable in the long term. It would be a fixable problem though...

spenczar commented 3 years ago

Ah, got it! Yeah, static is not the right place, as you say.

I don't think anything user-provided belongs in static, yeah. You probably want to just store them with FileFields. Django has a lot of good stuff for managing uploaded files, see https://docs.djangoproject.com/en/3.2/topics/files.

I think media is probably fine, even for large files.