search

foundryvtt / foundryvtt

Public issue tracking and documentation for Foundry Virtual Tabletop - software connecting RPG gamers in a shared multiplayer environment with an intuitive interface and powerful API.
https://foundryvtt.com/
235 stars 9 forks source link

Enable Single-Bucket endpoints in S3 #9021

Open abbottmg opened 1 year ago

abbottmg commented 1 year ago

User Experience

I've been working on adding S3 storage to my personal server, and from what I can tell the Foundry GUI assumes it can get a list of buckets from its configured S3 connection by calling against the "root" endpoint, so that it can display the bucket name(s) in a dropdown within the file dialog. For my use case, I have a single bucket to store all user-uploaded Foundry content, with a custom domain CNAMEd to the bucket's URL. This is so that I can change S3 providers without having to rewrite the URLs for each file or force users to re-configure any objects that use a particular image etc.

This pattern is supported by my S3 provider, including support for SSL certificates at that domain, etc. From what I can tell, the AWS SDK also supports this use case via the config option s3BucketEndpoint: true, which tells the client object to treat the endpoint as a single bucket. Unfortunately, this simply disables the ability to query a list of buckets on that object (instead of returning the name of the single bucket located there...). This seems to break initialization of the "Amazon S3" tab of the file dialog. Even if the dialog initializes with a blank dropdown, the server rejects user requests sent without a bucket identifier.

If Foundry can check for that same config option, it could disable the bucket dropdown and the validation check for a non-blank bucket ID.

This would allow me to enable SSL on my config, and would keep the user-visible URLs for resources to a more manageable https://files.foundry.example.org/path.png rather than https://files.foundry.example.org.us-region-1.provider.com/path.png

I can provide examples of both the working and ideal was-config.json if needed.

DavidArchibald commented 1 year ago

I would just like to add I believe this would help with using Cloudflare R2 as well, though potentially this is the provider in question. You can see here that they actually most favor you using your own domain for the bucket. From their perspective it's not that unusual because you probably already are putting your domain under Cloudflare. To set all this up (as I have) is still only a 15 minute task at the most.

R2 is cheaper to store in than S3, doesn't charge for egress, can be distributed in edge servers, and makes it easy to migrate from AWS so I imagine it would be a very attractive option to users of Foundry since it seems objectively better for this use case. Obviously I don't mention this to attempt to pitch but simply explain why I favor R2.

abbottmg commented 1 year ago

I'm not entirely sure why I was shy about naming my S3 provider before, but I actually use Linode. They seem to follow more or less the same workflow as Cloudflare.

I tried removing my custom certificate from the bucket in order to allow TLS. Unfortunately, the certificates provided by Linode seem not to play well with multiple levels of subdomain introduced by a bucket name (probably because they expect you to provide your own cert...) so my instance cannot serve up S3 via HTTPS at all unless I'm willing to create a separate bucket with a simpler name and migrate everything should this feature be implemented.

To echo @DavidArchibald's logic, I chose Linode for S3 because they have a small, flat monthly charge with generous transfer budget I'm nowhere near hitting on other web applications, even with a much heavier media use case than a handful of people pulling the same few dozen token and map images.

aaclayton commented 5 months ago

Deprioritizing from V12 since this fell below the line.

DavidArchibald commented 5 hours ago

Is the hope to get to this in v13 then?

Only tangentially related to this but I have to admit I chuckle a bit every time I see a link like: https://r2.foundryvtt.com/website-uploads-public/screen/user_1/ember-cosmos-loop-2024-06-27.webm; evidently Foundry itself favors r2. I understand the obvious difference between wanting to use it personally for storing assets and spending the development time to make it easy to use in the core software, haha.