Decide between presigned url vs direct upload approach for content in S3

kurtisassad commented 3 months ago

There would basically be 2 approaches for uploading content to s3. We could either send a presigned url to the user for them to upload the content themselves, or the user can pass in the content to our server, and we make a direct upload to s3.

Presigned URL

Description

So the idea would be the user has content they want to upload, say a thread body. They request for a presigned url from our server which then points them to an s3 bucket which the can upload that content to. During this presigned url request, our server would fire an event to a queue containing the s3 url of the content for further processing (indexing, ...) by some asynchronous consumer processes.

Pros

Consistent model with our image uploads
No bottlenecks (Only bottleneck would be s3). Cheaper since we don't need to pass along the data.
Faster response times on content upload, since the processing would be done asynchronously.

Cons

More difficult to implement, because now our processing will be done asynchronously.
Not sure how to deal with "ghost" content where the user gets the presigned url, but fails to upload the content
Might not be compatible with our transactional outbox model

Direct Upload

Description

The user uploads the content directly to our server. Our server then uploads the data directly to S3. The server would perform the further processing step as part of this upload request.

Pros

More simple to implement since it follows our current pattern (We directly upload to postgresql)
No edge cases, if the user fails to upload the content our server will not store "ghost" content.

Cons

Not as efficient, our server needs to pass along the data to s3.
Slower response times as it will do the further processing portion asynchronously.

timolegros commented 1 month ago

I'm not sure if we landed on a specific approach here but I do believe we agreed that the direct upload method is easier to implement. As such I think for now we should move forward with the simple approach and we can transition to a pre-signed URL method in the future if necessary. @kurtisassad please let me know ASAP if you disagree because I will be working on the bucket ticket starting today.

jnaviask commented 1 month ago

I'm OK with the direct upload approach here for the time being. Closing this ticket. We can revisit once initial MVP is in place.

hicommonwealth / commonwealth