TASK: Add MinIO compatibility

flownative / flow-aws-s3

Amazon S3 adaptor for Neos and Flow

MIT License

18 stars 33 forks source link

TASK: Add MinIO compatibility #37

Open etlam opened 4 years ago

etlam commented 4 years ago

This pull requests adds compatibilty for MinIO.

Problems with MinIO:

resources beneath resources are not supported by MinIO
use_path_style_endpoint=true must be used

dfeyer commented 3 years ago

@robertlemke @kdambekalns I can confirm that Minio did not have the limitation that this PR try to solve. Running a project and currently URL like http://storage-medialib.cms.ttree.localhost/public/sites/core/c31d44f75b3ba322d5219cd1e17c2ba933a6db5a/9WO0CELoWq6tSdjqCs34Y0qek7l-420x236.jpg are perfectly valid and works with latest docker image of Minio

But use_path_style_endpoint is required and the README update can be nice even if the Minio document is clear about this https://docs.min.io/docs/how-to-use-aws-sdk-for-php-with-minio-server.html

etlam commented 3 years ago

If you use for example Sitegeist.Kaleidoscope the original image gets saved as jhgsjsjhkshgezwwef and the smaller thumbnails try to get saved as jhgsjsjhkshgezwwef/<hash_of_variant>[...].
So the original image should be used as file and folder at the same time. This is not possible with MinIO because of the used underlying linux file system.

A folder and a file can not have the same name and be located in the same folder:
https://github.com/minio/minio/issues/7335#issuecomment-470683398
https://github.com/minio/minio/issues/10160

Maybe there is a more elegant way to prevent that there is a file and a folder with the same name.

kdambekalns commented 3 years ago

See #42

etlam commented 3 years ago

Yes, you have to use use_path_style_endpoint=true for MinIO.
But the problem with same file and folder names remains, as linked in my comment above: https://github.com/minio/minio/issues/7335#issuecomment-470683398 https://github.com/minio/minio/issues/10160

AWS S3 can use a file as a folder and store more files inside of it, MinIO can not do this.

We can close the pull request if using Sitegeist.Kaleidoscope with MinIO is a special case.

kdambekalns commented 3 years ago

Still not sure this is an issue with Sitegeist.Kaleidoscope, nor that it is a general issue:

Flow stores persistent in resources in a storage using just the hash
Resources and variants are published in a target using a pattern of hash/name

Since the hash is dependent on the file contents, two different files will have a different hash, and as long storage and target have separate buckets (as required for the S3 adapter), there should never be a clash…

Are you in fact using seperate buckets for storage and target, @etlam?