503 error on accessing the webpage

[surfd4wg]

The cloudformation template worked great. I approved the SNS message from my email. I also sent a request to get SES into production. But I was getting a 503 on kikrr.social. I noticed that the WebService task tried to start several times but couldn't. The only cloudwatch logs in Alarm are the following:

mastodon-on-aws-Alb-14D3IVKFCK6QL-HTTPCodeELB5XXTooHighAlarm-AB5QEJNE5UMD

mastodon-on-aws-Database-129D3MPJPEB2S-Instance-ET9T36H2L845-AlarmCPUCreditBalanceTooLow-YMN7OELIO1UB

I downloaded the logs for the failed WebService, https://gist.github.com/surfd4wg/dc664827e696907344353e80ae4bd245.

Looking for some help. Thank you.

[andreaswittig]

Hm, the Rails app cannot connect to the RDS database. Could you please check the status of the RDS database instance?

[surfd4wg]

The RDS is available. I did see an alarm, and it doesn't make sense. Attaching here.

[scrappydog]

t2 and t3 instances use “burstable” CPU credits

"To prevent throttling when credits run out, you can enable T2/T3 Unlimited."

https://medium.com/dnx-labs/understanding-credit-balances-on-aws-services-ec2-and-rds-2c1d4cfeb8e0

(I need to go make this change on my instance! ;-)

[surfd4wg]

I saw that, but unclear on how to set the fargate instance to use that. Maybe the cloud formation template needs to be modified to specify unlimited.

[scrappydog]

Amazon RDS T4g and T3 DB instances run in Unlimited mode, which means that you will be charged if your average CPU utilization over a rolling 24-hour period exceeds the baseline of the instance. CPU Credits are charged at $0.075 per vCPU-Hour. The CPU Credit pricing is the same for all T4g and T3 instance sizes across all regions and is not covered by Reserved Instances.

[scrappydog]

I think this is basically a false positive new instance alarm... it will clear as soon as you balance climbs above 20 without any action on your part... and you will probably never see it again.

[surfd4wg]

I re-deployed the WebService task, and it started this time. So it looks good for now. However, this brings up the question of stability and scalability...

[surfd4wg]

Closing but still curious about stability and scalability based on what I experienced. I think the general infrastructure and cloudformation templates are good, just might need tweaking to get around the AWS quirks. As for scalability, time will tell. I guess I have a question for the authors regarding "was this built with the intention of scaling to thousands of users?

[michaelwittig]

@surfd4wg our own instance runs with 2 users so we can not say that we have experience running a mastodon instance with 1000 users. Do you plan to run a larger instance? The obvious issue at the moment is #1 and you also want to change the cache/database types to something bigger and not a t* family member, not cache.t4g.micro or 1db.t4g.micro`.

[surfd4wg]

Where would I change the container type?Sent from my iPhoneOn Dec 2, 2022, at 5:39 AM, Michael Wittig @.**> wrote: @surfd4wg our own instance runs with 2 users so we can not say that we have experience running a mastodon instance with 1000 users. Do you plan to run a larger instance? The obvious issue at the moment is #1 and you also want to change the cache/database types to something bigger and not a t family member, not cache.t4g.micro or 1db.t4g.micro`.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

[michaelwittig]

There is no such thing as a container type. You can tweak the following in the template you downloaded and update.

Cpu: '0.25'
Memory: '1'

but check allowed values first https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-cpu-memory-error.html

[surfd4wg]

Assuming I upgrade the database instance the same way?

[michaelwittig]

@surfd4wg yes :+1: