Image storage and handling

[gedw99]

A really common pattern is where you need to allow users to upload images and i need to save its existence in the DB for later use.

For example you might have a web page with a form, and as part of that they need to upload a screenshot or a list of images.

Then you want to save the image in a local FS or S3, and record the URL ( or RID - resource ID ) in the DB, so that you have a record of the form data and also the image that they also uploaded with the Form.

So you need a Web template for image uploads also that can be easily included in other golang templates. This is because you need image upload GUI that can be used with different form golang templates.

is there a reusable pattern here ?

The above example if a very simple one. A generic pattern of having a "blobs" table in the DB that is just for holding a mapping of files to the db can also be useful because it can be reused for lots of use cases. You can then reference that "blobs" table from other tables in your DB.

In terms of transactional integrity, you save the file first and if no errors then save the RID into the "blobs" table.

There is also a side effect with this pattern that can occur if you allow mutation access to S3. An image is changed, and you need to tell the DB that it has changed. File systems and S3 have a Notification API for this. Here is the minio docs on this to illustrate: https://docs.min.io/docs/minio-bucket-notification-guide.html In this case, you woudl want a webhook to call back into the Golang API.

Hope that i explained myself ok and that you think this is generic enough and crosses enough use case that its worth including.

For anyone reading this, please let me know what you think. Is this something you need ? Is it better to do this in a different way ?

feel free to ask me if i have not explained this very well.

[mikestefanello]

Thanks for suggesting this. I definitely forgot about including something related to files. I think it would be a great feature to include since it's a very common requirement. Have you researched any existing Go modules that could help? It would be nice if there's a module that provides interfaces for the common file storage providers plus local, so you can easily use the one you want without needing to change the code or implementation.

[gedw99]

https://github.com/hack-pad/hackpadfs is very flexible.

Its supports s3, local and also browsers. It also can do overlays.

I have used this and its well done and well supported.

[gedw99]

There is a really high quality file system in ocis . I am using it on different projects.

mit has a gui that basically is like google drive without of course the lock in

The way its architecture is designed is quite cool because it can run as a single binary or be broken into many micro services . It’s the type of pattern I use on almost all projects . Even the cli is designed that way .

Under the hood it’s uses a golang implementation of the erlang supervisor trees, which allows it to run and yet control many processes or one process.

https://github.com/owncloud/ocis

It supports file upload / download/ sharing and workspaces .

the file system provider is agnostic and can use local file system, s3, ceph, s3 ( minio and others ) or others by extension.

it uses tusd for the core file upload / download which supports chunked and retry and so can scale to really large files and handle retryable uploading if the network connection fails.

Its is based on the cs3 stardard also.

m the system is extensible such that open office or any app / project can tap into it .

i am only suggesting this because it’s very well done. It might not be your cup of tea though.

The htmx pattern for how you do your gui is probably not going to work with pagoda but perhaps a minimal file upload / download can be done that leverages reva / ocis ?

https://reva.link/

https://github.com/cs3org/reva

Basically it’s Design as a way for other systems to tap into a cs3 based system for which ocis is one of them.

It’s also incorporates IDP being separated and supports openID, ldap , etc