investigate random slowdowns

[domino14]

9/25/23 @ 08:15 pm and @ 08:31 pm (EST) or so there were lag spikes. People reported "Connecting to server..." and moves not registering on the app.

Logs show some extra SLOW SQL statements. But this could be because requests are getting backed up and coming all at once. We need more monitoring.

Things to monitor:

• [x] Back pressure on our internal go channels in bus.go (On the socket too for good measure)
• [ ] Number of SQL queries per second/minute/etc
• [ ] Slow SQL queries on non-Gorm connections (only Gorm seems to print out the SLOW SQL logs)
• [ ] Average time to connect to socket (figure out exactly when "Connecting to server..." shows up -- what causes it to finally disappear)
• [ ] HTTP requests per second/minute/etc (Can probably make a Logs Insights query for this)
• [ ] Consider moving to connect sooner (#1079) - besides having to do this anyway, we can experiment with reusing SSL connections etc

Also see #1136. This may still be a significant cause of lag as it causes a lot of channel pressure.

[domino14]

a lot of these "context canceled" errors occur. Is this because people keep refreshing and it kills the Twirp connection?