search

nextcloud / server

☁️ Nextcloud server, a safe home for all your data
https://nextcloud.com
GNU Affero General Public License v3.0
27.49k stars 4.08k forks source link

Preserve file/folder structure in S3 without urn:oid #49416

Open keithkhl opened 1 day ago

keithkhl commented 1 day ago

How to use GitHub

Is your feature request related to a problem? Please describe.

I try to use S3 as a primary object storage, but file names and folder structures are not preserved. All files in S3 are in urn:oid w/ appended numbers, which seems like a database entry for metadata.

With this limitation, I cannot access files from S3. I always have to go to Nextcloud's console. File sharing outside of Nextcloud is also way more difficult than local storage case.

Without S3 storage, unless I have a large local storage, it is not easy to extend user base as well.

Describe the solution you'd like

Help us to find the files/folders on S3 bucket, just like it is done in local storage. Although /data folder in local server is not accessible by default, as a server admin, I can still see what's going on in there. The same privilege should be reserved for S3.

Describe alternatives you've considered

Not much. I wonder how other people use S3, if this is not supported.

Additional context

If file/folder access is limited to Nextcloud only, I have to look for other solutions for file stroage + team collaboration. Despite all wonderful functions that I push forward Nextcloud to the team, I think shared external storage is a key for running a server, especially in case of server failure.

keithkhl commented 1 day ago

seems like there have been similar requests multiple times here and there.

I think identical copy of /data folder solves all our needs.

joshtrichards commented 23 hours ago

With this limitation, I cannot access files from S3. I always have to go to Nextcloud's console. File sharing outside of Nextcloud is also way more difficult than local storage case.

This only applies when using S3 as Primary Storage. You might find External Storage (which also supports S3 and does not depend on urn:oid identifiers nor require exclusive control over the metadata) more suitable if the goal is to access the underlying data repository outside of Nextcloud.

keithkhl commented 23 hours ago

Well thanks for the comment, but urn:oid was indeed created for external storage cases as well. It's just that the urn:oid files are not real contents but database meta in that case.

File sharing is only a tiny piece of my puzzle. I need S3 for server scaleability. The S3 integration as primary storage is a key, unless I run manual rsync across multiple servers.

For files in S3, without Nextcloud, I cannot share, copy, read, and identify what it does, unless I match the oid from database tables. In addition to token, more than 3 datatables are needed to see the file details, if I remember correctly.

Not sure how difficult is this request gets done, but I doubt it is an incredibly difficult job.

keithkhl commented 23 hours ago

One other thing that S3 as the external storage is not a solution to my case. Except the primary storage, for all other external ones, it seems like it is user specific. For other users, they also have to connect S3 by themselves.

Say there are 100 users, then the basic files for each new user is about 34MB, thus 3GB+ is required in the local server, and unless there is a way to make S3 as an external storage as default for all users, setting up each person's configuration is kinda nonsense.

Hope this make sense. It would be deeply gratifying, if there is a reasonable solution for this.

joshtrichards commented 13 hours ago

Well thanks for the comment, but urn:oid was indeed created for external storage cases as well. It's just that the urn:oid files are not real contents but database meta in that case.

I assure you that is not the case. That is not how External Storage works. Please re-check/re-test.