[Architectural question] Recommendations for creating a fallback bucket / host?

humanmade / S3-Uploads

The WordPress Plugin to Store Uploads on Amazon S3

1.95k stars 392 forks source link

[Architectural question] Recommendations for creating a fallback bucket / host? #476

Closed lstellway closed 3 years ago

lstellway commented 3 years ago

Description

I recently experienced an issue with a DigitalOcean space having maintenance problems. The issue caused interrupted service to and from my S3 bucket.

In light of this being a possibility, I am exploring what my options are for preventing this in the future.

Questions

Would it be possible to upload to multiple S3 buckets?
- If so, I could create a proxy / load balancer configuration to serve files from the buckets.
- I see some discussion (elsewhere) about the possibility of syncing / mirroring S3 buckets. If that was possible, it probably wouldn't be necessary to hook into / modify the plugin.
- I also came across a tool (Zenko) that seems to be geared towards my use-case.
Would it be possible to keep files locally in addition to uploading them to S3?
- I could do similar server configuration to fall back to my web instance
- This could work for me currently as we only have a single web host instance, but it's probably not a scalable solution.
Are there any other thoughts / ideas / recommendations?

joehoyle commented 3 years ago

Would it be possible to upload to multiple S3 buckets?

That wouldn't be readily easy with S3 Uploads. there is some actions for when the file is pushed to s3 (see https://github.com/humanmade/S3-Uploads/blob/master/inc/class-stream-wrapper.php#L188) where you could push to an additional service manually. Though your syncing methods look like a better solution to me.

Would it be possible to keep files locally in addition to uploading them to S3?

Not really, s3-uploads is meant to work in cases where this isn't even a writable local system, so it bypasses any of the wp-content/uploads writing.

Are there any other thoughts / ideas / recommendations?

IMO cross bucket or cross service syncing by another system (s3 putObject policy hooked into Lambda for example) would be the best way to go here, rather than push it twice from the application layer.

lstellway commented 3 years ago

Thanks for the quick answer!

IMO cross bucket or cross service syncing by another system (s3 putObject policy hooked into Lambda for example) would be the best way to go here, rather than push it twice from the application layer.

That's a great idea! I'll definitely consider it.

joehoyle commented 3 years ago

👍 sounds good, going to close this issue out.