DigitalOcean Spaces Rate Limiting

mattrothenberg commented 4 years ago

Bug Description

When using DigitalOcean Spaces as my site's remote asset source, I'm consistently running into rate limiting issues (as shown by the screenshot below). This error was triggered by trying to edit an entry from the control panel. Worth noting is that my site has zero web traffic, and my Spaces bucket has < 2mb of content in it.

How to Reproduce

Given an S3 asset source When I try to edit an entry that's using one of these assets (e.g., a "cover photo" or a "featured image") in its content Then I consistently see these 503 Rate Limiting error messages, even over the course of multiple days

Extra Detail

Conversation with DO Support

Hello there, thank you for contacting DigitalOcean. 

Looking over today's logs on our backend I can confirm that this is a case of rate-limiting. I see you are being rate limited on  HEAD requests from IP 161.35.60.117. 

How many requests are you sending per second to your bucket?

750 requests (any operation) per IP address per second to all Spaces on an account
150 PUTs, 150 DELETEs, 150 LISTs, and 240 other requests per second to any individual Space
2 COPYs per 5 minutes on any individual object in a Space

Environment

Statamic version: 3.0.1.6

PHP version: 7.4.11

Install method (choose one):

Fresh install from statamic/statamic

duncanmcclean commented 4 years ago

I've not used DigitalOcean spaces but I don't think Statamic directly is the cause of this issue. I think it's probably the Flysystem S3 driver that's used to allow Statamic to talk to AWS S3/DO Spaces.

From a bit of googling, looks like it's related to these threads on various forums:

mattrothenberg commented 4 years ago

@damcclean I already opened an issue on that project and was assured that it's likely not an issue to that end

https://github.com/thephpleague/flysystem-aws-s3-v3/issues/221

Moreover, this doesn't really help explain why accessing a single asset would require exponential backoff / throttling.

duncanmcclean commented 4 years ago

@damcclean I already opened an issue on that project and was assured that it's likely not an issue to that end

thephpleague/flysystem-aws-s3-v3#221

Moreover, this doesn't really help explain why accessing a single asset would require exponential backoff / throttling.

Good point! I'll see if I can replicate the issue myself to see what's going on.

martink635 commented 3 years ago

We are running into the same issue. We have approx 200 files on the Spaces and it seems we get sometimes throttled when just listing entries, that contain those assets. The entry list is also super slow to load due to fetching content from Spaces.

If you need more information regarding the setup etc let me know 🙂

github-actions[bot] commented 3 years ago

This issue has not had recent activity and has been marked as stale — by me, a robot. Simply reply to keep it open and send me away. If you do nothing, I will close it in a week. I have no feelings, so whatever you do is fine by me.

GioChocolateBro commented 3 years ago

@mattrothenberg I found this issue through your flysystem issue. Did you figure it out yet?

DigitalOcean doesn't have a very high rate limit, this is why you have to use the CDN at all times. Never load an image directly from the DO Space. Always through the CDN. Before using Statamic I was also always running into the rate limits, back then we used a sort of caching system instead of hitting DO every time. But with the filemanager DO Space will get hammered with requests every time!

To do this you have to change the ASSET_URL found in the app.php config in Laravel.

Afterwards the entire Statamic CP will be broken since it won't be able to load the css/js/fonts etc.. You'll need to upload these to DO Spaces as well.

We build a special publish command to automate this.

remove old statamic assets
upload new ones
purge the digitalocean CDN (to get the CDN id you need to do more crappy DO api code since it's very hard to get a hold of this 😅 )

Theres a couple more caveats to this. If you want/need to know more drop me a line.

mattrothenberg commented 3 years ago

@GioChocolateBro Interesting. I didn't have to do any of these things. https://github.com/statamic/cms/pull/2828 seems to have resolved this issue just fine for me.

GioChocolateBro commented 3 years ago

@mattrothenberg Yeah #2828 deff saved the same issues for us, file manager wouldn't even load before.

But loading the file manager/asset container instantly drains the request limit for us right now when we hit the space directly, how many files are in your container? We do everything through the CDN now and don't have issues anymore. Figured this was the same issue you were facing..

During development we didn't have any issues, because we just had a couple of uploads. But when we deployed to a server and our clients started testing/uploading loads of files this started to become a problem.

mattrothenberg commented 3 years ago

Odd. Not sure what's going on there. I have thousands of assets and it seems to "just" work 🤷‍♂️

GioChocolateBro commented 3 years ago

Been racking my brain over why, but just remembered we extended Glide to do everything through digitalocean (or any other remote flysystem disk)! By default Statamic doesn't allow you to set a different disk to save all manipulations/caching (its saved to your local storage folder).

We want every file to be saved in our bucket also the glide crops. This is where we hit these troubles.. Sorry the confusion 😅.

We now have a separate bucket/CDN for all our projects glide crops/manipulations, it makes a folder for each project/website. So glide crops come from a special CDN, while the original uploads are saved in a different bucket thats specific to that project/website.

I do think it would be a nice feature request for Statamic btw, being able to set the glide disk! @jasonvarga

*sorry for getting way off topic..

duncanmcclean commented 3 years ago

Looks like there's already a feature request open for using different Flysystem filesystems for Glide, see statamic/ideas#341.

GioChocolateBro commented 3 years ago

Also looks like I already jumped in here 2 months ago, cant recall 😆. Don't mind me ya'll..

jasonvarga commented 3 years ago

Just a heads up - there's a bunch of asset optimizations making it into the next 3.1 prerelease. It should solve most if not all of your rate limiting issues.

statamic / cms