S3 Support

[Jotschi]

Gentics Mesh can potentially make use of S3 in various ways. Each enhances the functionality in various ways.

Options

A - S3 Binary Storage

S3 can be used as a binary storage replacement. This means that every upload to Mesh for binary fields is passed onto the S3 server. The binaries will be stored using the binary field UUID in S3.

The path to the S3 object can be either determined via the binaryField Uuid or an S3 path property within the binary response JSON. This allows client applications to render S3 URLs for web clients.

Image Resize requests still need to be answered by Gentics Mesh. (Dynamically resized image data is not placed on S3)

B - External S3 URL Support

Instead of uploading a file to Mesh you can include a S3 URL in your binary field JSON to update the field. Gentics Mesh will utilize this field when it needs to process binary data.

This way existing S3 data can be utilized in Gentics Mesh without the need to re-upload everything in mesh.

C - S3 Link Resolving

Gentics Mesh will resolve mesh links to binary nodes into S3 URLs. This way asset links will directly point to S3 instead to Gentics Mesh. This allows web CDN usage of assets in S3.

Feedback

We still need feedback before we can fully implement S3 support. We are curious whether the mentioned options would solve your specific use-case for S3 with Gentics Mesh. Please let us know if we missed something.

Questions

• Is it acceptable that all assets are placed in a single bucket?
• How should / could permissions be handled for assets in the S3 bucket?
• How could image resizing be handled for S3 assets?

[mephinet]

Just discussed with a potential customer, who is looking for A+C (thereby using S3 as a geo-distributed CDN for binaries).

[ReneVolution]

I would also vote for A+C.

From other CMS, I have seen S3 Binary Storage implementations, where the client would upload binary files to a transient S3 bucket and once the upload is complete, the CMS would move the file to the final storage bucket.

That client flow could look as follows:

• Initiate a new Upload with Mesh (the response contains a temporary upload token for the transient S3 bucket)
• Upload file directly into the transient S3 bucket with previously received temporary credentials
• Notify Mesh about the upload completion.

The biggest benefit of this approach is that logic can be implemented to prevent accidentally overwriting files. Also, from a security point of view this would add the ability to only provide "write only" access to the client application.

For S3 Link Resolving, I have had good experience with redirects to pre-signed S3 urls. These expire after a configurable time and also normally keep the permissions system in tact.

[ReneVolution]

Oh, and just for the other questions:

Is it acceptable that all assets are placed in a single bucket? Yes How should / could permissions be handled for assets in the S3 bucket? As explained before using pre-signed url redirects How could image resizing be handled for S3 assets? when implemented using the previously described flow, the complete event could kick off image resize operations

[plyhun]

Partially implemented as A variant, with requested resizes cached locally, with no S3 permission manipulations (controlled by Mesh owner node by now).