Feature: Local Storage with S3-compatible

[chiqors]

Is your feature request related to a problem?

I am not sure, if this has implemented the storage feature or not. But, it would be great if we can decide the files for pushing into cdn or even AWS S3 later.

Describe the solution

Implementing storage feature with S3-compatible might be a good solution for people who prefer to host files on their machine first, before using cdn.

Alternative methods

If you don't have an idea for implementing it. Take a look of these 2 open source repo https://github.com/ceph/ceph https://github.com/minio/minio

SurrealDB version

surreal v1.0.0-beta7 on Windows

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[JABirchall]

This isnt the resposability of a database, If you need to host files on your local machine, do that.

[chiqors]

This isnt the resposability of a database, If you need to host files on your local machine, do that.

It's still relatable, imo. Take an example for blog posting, ecommerce, etc. I know, this might not be their priority. But, i would like to see this in the future.

[itsezc]

This isnt the resposability of a database.

How come? Files are a collection of data, so I don't see why not.

Also would love support for Storj (https://storj.io) which supports an S3 API but with limited functionality, perhaps the API could support BYO provider so we can easily build integrations?

[uahic]

If you implement support for storages like minio you basically have ( I think) everything in place to beat solutions like supabase, nhost, firebase, appwrite etc.

The integration would somehow require that user roles/access rights of files or buckets of files are considered

[ansarizafar]

Garage would be a better option https://garagehq.deuxfleurs.fr/

[Olyno]

I would more see Seaweedfs instead.

[JABirchall]

The goal of surreal is for Data storage not file storage.

If you need a file storage service, you should use such services you have recommended.

[itsezc]

The goal of surreal is for Data storage not file storage.

If you need a file storage service, you should use such services you have recommended.

I don't think that goal is listed anywhere explicitly, there have been several conversations in the discord server that suggest files storage / uploads are being considered, and since a file is "an entity of" data, it would fall under the database scope - there are several databases that handle files. Additionally, this request has gained traction in terms of positive support, which would suggest there is a proper use case for it.

Reference: https://discord.com/channels/902568124350599239/970336107206176768/1016961315807432774

To quote Tobie: "... We do intent to support file upload/storage in due course, but this is still a way off we expect..."

[tigawanna]

It's also doing a lot of non conventional database things like direct queries which are more of anti pattern than let's say for example exposing an API that lets you define a file column that will saves the path to the uploaded resources either saved where the database is being hosted (useful for self hosting and on premise hosting) or a cloud storage provider . Just saying

[MatthewAry]

I said this in #1652 but I will say it here.

Consider how Supabase handles storage. https://supabase.com/docs/guides/storage it seems that it does this by using the DB layer (which for supabase is PostgreSQL) to store the file metadata and Supabase handles the sync logic. While I have not used Supabase, I imagine that this solution is great for DX as it makes handling files something that seems almost a native feature for the database requiring some initial configuration to setup. Supabase ends up abstracting away the complexity of dealing with the object storage service.

However, while S3's API is seemingly the industry standard for file storage services (Because of all the S3 clones out there), Google's Cloud Storage system is different. If SurrealDB decides it would want to pursue a solution that is similar to what Supabase is doing, then hopefully it will be platform agnostic for the storage layer.

[amiyatulu]

S3 compatibility or direct storage to IPFS using Iroh within the Substrate blockchain, can be valuable additional features. It would be highly beneficial if there were a no_std version compatible with the Phala Network or Integritee SDK that includes data encryption and confidentiality.

[maxwellflitton]

Hey guys, I'm currently working on the ML integration into the database and alongside this I will be integrating object storage as you can see here. This is using the object_store crate which supports s3, Azure, and GCP. Right now we're just getting it to work with local storage but will add the other engines when working. We are also working on supporting file storage streaming here. Right now I'm going to get it working with storing ML files but as you can see with the isolated nature of the object storage module it will not be hard to add general file storage once this is merged. The loading of the ML files is also going to be integrated into the SQL as well.

[sandros94]

@maxwellflitton sorry for the ping, any updates on this?

While digging around I found a SURREAL_OBJECT_CACHE and SURREAL_OBJECT_STORE env variables that do refer to specifying local storage paths. Are these the ones used by ML under the hood atm (1.4.0)? Would it be possible to provide an initial example for how to use local storage until it will be officially supported?

I also see a reference to file_store functions https://github.com/surrealdb/surrealdb/blob/b9f02d146da2222e14c22264283297ff678a59e2/core/src/dbs/store.rs#L60-L420