search

directus / v8-archive

Directus Database API — Wraps Custom SQL Databases with a REST/GraphQL API
https://docs.directus.io/api/reference.html
507 stars 203 forks source link

Question: S3 image hosting #1341

Open raftAtGit opened 5 years ago

raftAtGit commented 5 years ago

When using S3 storage adapter, I've noticed thumbnail URL's are at API, acting as a proxy to S3, like /thumbnail/_/200/200/crop/good/xx.png but full size image URL's are actual S3 URL's.

Is this a misconfiguration on our side or is it intentionally like this (for performance etc)?

Thanks

jbalatero commented 5 years ago

I think this is intentional since the thumbnails sizes are variable. So thumbnails are generated on the fly on the config

raftAtGit commented 5 years ago

Are we talking about same thing?

All the images are stored at S3, either uploaded (original) or generated on the fly (thumbnails).

When accessing them, for thumbnails the URL's are at API server, so to my understanding, it proxies to S3. But for original images, the URL's are directly pointing to S3.

benhaynes commented 5 years ago

The way it should work is that all assets (original and thumbnail) should be saved on your storage adapter (eg: S3), but there might be an issue with thumbnails only saving locally due to how those process. Checking/fixing this is on our high-priority list... so we'll find out soon.

🔔 @rijkvanzanten @bjgajjar

raftAtGit commented 5 years ago

In my case, all assets (original and thumbnail) are saved on S3 (Minio actually), I can verify it.

As mentioned, when I query Directus API for /_/files, returned thumbnail URL's are on Directus API /thumbnail/_/200/200/crop/good/xx.png but original image URL's are on Minio.

I want to access original images via Directus API too, if this wouldnt impact performance. I dont want to expose Minio to outer world. Is this possible?

jbalatero commented 5 years ago

I want to access original images via Directus API too, if this wouldnt impact performance. I dont want to expose Minio to outer world. Is this possible?

I am currently on the same page, I don't want to expose minio to outer world also. The way it looks right now, it should be exposed.

WoLfulus commented 5 years ago

I agree it would be nice to proxy original images too (it's also a case for me), but as an optional feature. Making this non-optional (even though I don't see how we would make it a requirement) would break a lot of installations.

Doesn't seem that hard to implement though. I can give it a try later today if there's no on willing to do it right now.

benhaynes commented 5 years ago

This middleware proxy would also be required to secure assets through permissions, since we can't do that if we expose the actual file's URL.

binal-7span commented 5 years ago

@raftAtGit - Maybe this will help you out to figure out the issue https://github.com/directus/api/issues/1127#issuecomment-513741367

everyx commented 4 years ago

@raftAtGit at v8, api will not return thumbnail url like /thumbnail/_/200/200/crop/good/xx.png but /assets/t2701u06z9c000ko?key=directus-medium-contain, so this is a v7 problem?

everyx commented 4 years ago

I agree it would be nice to proxy original images too (it's also a case for me), but as an optional feature. Making this non-optional (even though I don't see how we would make it a requirement) would break a lot of installations.

Doesn't seem that hard to implement though. I can give it a try later today if there's no on willing to do it right now.

@WoLfulus Maybe we should add a option to let user can get a CDN url for files, It's too heavy to let php handle static file request.

raftAtGit commented 4 years ago

@everyx right, this is was the issue with v7, didnt try v8 yet

WoLfulus commented 4 years ago

I totally forgot about this one. My bad sorry. I'll take a look as soon as I can.

rijkvanzanten commented 4 years ago

The actual thumbnails are still stored on the same location as the full files, so if you want, you can circumvent using the thumbnail URL completely and go straight to the source (like your CDN). As for

It's too heavy to let php handle static file request.

This is very true. Currently, the API reads the file into memory and then returns it. We have to update this to stream the file straight from disk to response.

The API generates thumbnails on request, instead of on upload (by design). We can consider adding an option that it will redirect with a 302 to the "full" thumbnail URL (on CDN) if it already exists 🤔 @WoLfulus

everyx commented 4 years ago

@rijkvanzanten

The API generates thumbnails on request, instead of on upload (by design). We can consider adding an option that it will redirect with a 302 to the "full" thumbnail URL (on CDN) if it already exists 🤔

Add a option to use "full" thumbnail is great, if this option is be set, and if we need return the thumbnail url, maybe we should:

  1. if the thumbnail is not exist in filesystem, return a url to api can generate it and then 302 redirect it to full thumbnail URL.
  2. if file exist, we return the full thumbnail URL.

We should also consider the same when we return the original link to the file with full url (on CDN).

rijkvanzanten commented 4 years ago

We should keep it as an option though. Using the /assets/ endpoint at all time gives us the ability to have private files. The private hash can change while the file stays the same, allowing you to prevent people from accessing the file through the assets endpoint at any point :)

everyx commented 4 years ago

@rijkvanzanten Another question, why assets endpoint url scheme is /<PROJECT>/assets/..., but download endpoint is /download/<PROJECT>/..., why don't the two remain unified as /assets/<PROJECT>/... and /download/<PROJECT>/...?

rijkvanzanten commented 4 years ago

I don't have the answer. They should be the same. Everything is scoped /<project>/<endpoint>