sirrus233 / twilight-imperium-helper

Game assistant for Twilight Imperium
1 stars 0 forks source link

Fix 503 errors due to lambda throttling #34

Closed sirrus233 closed 3 months ago

sirrus233 commented 3 months ago

Happy to have finally tracked this down! This change bundles our code differently, grouping all our images under an assets/ path, with different cloudfront behavior (no lambda invocation) when fetching those assets. For the long description why we need this -- read on!


I noticed when loading the page when the static content is not cached (so say, after a hard refresh in the browser), sometimes a few of the ship images would fail to load. Seems we were getting 503 errors and some sort of header in the response indicating that lambda was throttling us due to too many concurrent invocations.

I dug around in the AWS docs and in our CloudWatch metrics, and I did notice that invocations were weirdly high. It turns out that we were hitting our www redirect lambda on every request -- which means 1 invocation per file. With ~10 images plus the bundle, the index, and the favicon, this comes out to 14 invocations per page load (!)

Even so - being throttled off that traffic still struck me as odd. Our account settings show a very low concurrent invocation limit of only 10:

$ aws --profile twilightimperium lambda get-account-settings
{
    "AccountLimit": {
        "TotalCodeSize": 80530636800,
        "CodeSizeUnzipped": 262144000,
        "CodeSizeZipped": 52428800,
        "ConcurrentExecutions": 10,
        "UnreservedConcurrentExecutions": 10
    },
    "AccountUsage": {
        "TotalCodeSize": 2016,
        "FunctionCount": 1
    }

I thought this should be much higher, but it looks like you get a lower limit for new AWS accounts and then this limit is raised either when the account is more trusted, or when you request an increase.

SO - whether or not we want to do that, I figure fix the dumb behavior. We don't actually need to redirect when serving images, we only care about the root path here. So splitting the dist folder into bundle and assets/ lets us specify different cloudfront behavior per path, and thus only run the redirect lambda when we actually need to.

image

sirrus233 commented 3 months ago

This fix was all infra - so it's deployed and live already :)

chelseaSchmidt commented 3 months ago

👀 image

Sending you some virtual tea/coffee :( ☕ ☕