Service periodically going down

[jcarbelbide]

Service is periodically going down. Many times, the service recovers on its own, but below are the times where it hasn't:

• First time, this issue was created: https://github.com/jcarbelbide/tog-crowdsourcing/issues/10
  • Took two weeks to respond, but was resolved in around 10 min.
  • The solution was to stop the VM and restart it
  • After this, a monitor on the service was set up so that response times will be at minimum 5 minutes
• Second time happened on 10/25/23. Monitor alerted at 11:25AM PST. Service was restored at 11:53AM PST.
  • The issue this time was the same as last time. VM unresponsive, stopping and rebooting restored service.
  • Monitor is helping, but underlying issue of service periodically going down remains
  • This happened this time around when the osrs servers reset. When the service started up again, there was no stream order data. Could be related somehow.
  • Update: Checking metrics on cloudflare reveals that we received an uptick of 700k requests at 11AM. Normal volume throughout the day is around 15k. This seems like an attack, but not sure.
• Third time again on 10/25/23 at around 11:43PM PST, service was brought down again
  • Metrics show spike in CPU utilization
  • This time when service was restored, it was not around a reset. Data remains unchanged after coming back up.

As to WHY this is happening, I am not sure. Some contributing/non-contributing factors:

• Service is behind Cloudflare for DDOS protection. It is not entirely impossible that the service is being attacked, but Cloudflare should be helping a bit to protect it.
• The plugin has grown in users and its possible that the server can't handle the large request volume.
  • The instance that I am paying for is AWS's cheapest option on lightsail. I'm sure paying for a better machine will help, but will probably only delay the problem
• The client does not seem to be sending an excess number of requests. I believe it is only sending 1 request on startup, twice when a stream order is detected, and 1 every 10 seconds if the initial get request failed.

As I find out more, or if this happens again, I'll update the post with more information.

[samszotkowski]

Appears to be down currently, not sure how long it's been like this https://www.reddit.com/r/2007scape/s/N3HaTPPnNg

[jcarbelbide]

Hey, thanks for flagging this. Should be back up. I have a monitor running on this, but it's been noisy, and seems i have missed this being down for so long. I'm going to look into improving that monitor so it doesn't raise alarms every time the server lags a bit

I think what happened was that AWS was going through an upgrade, and had to shut down a few servers. This one was on that list.

I've seen people concerned about the services availability. Definitely, this service does go down fairly often unfortunately. I do want to apologize for that. It's being used by a lot more people than I initially thought, and I think it sometimes has a hard time keeping up with the traffic. It's not an incredibly powerful machine that hosts the server, since I'm currently optimizing for cost. There's definitely some stuff I can do on my side to make it a bit better, so I'll look into this.

[jcarbelbide]

I've changed our monitoring tool to one that has a little more customization options that will alert only on real incidents instead of every time there is a failed request that recovers a few minutes after