biblicabeebli
commented
7 months ago https://instances.vantage.sh/ (formerly ec2instances.info) is a good source for viewing server pricing.
- The m7a.medium is probably better performance than a t3 instance but is slightly (on the order of $10 a month) more expensive; it only has 1 core, but T3's 2nd core is SMT.
- T3 can have serious throttling issues (there is a scenario where it can get stuck at a fractional compute rate, like 20% speed), so the m7a.medium is Almost Definitely better than that scenario.
- T3 is also slow in single threaded performance, compared to m5 instances (they literally run on the same cpus), the T series cores are not 100% of an m5 cpu core.
- The m7 is a 2023 cpu microarchitecture (almost definitely Epyc 7004 Zen-4c cores) and the t3 is a 2015 microarchitecture (skylake).
Really shifting to containers would more fully solve our problems, but that's a huge lift and addition/change/support cost for the deploy scripts.
Now that we are running Forest tasks on the data processing servers we have a lot more room for stability issues to affect the data processing manager servers.
The fix is to adopt a new data processing manager/worker server structure where the manager server handles task distribution and push notifications, and the worker servers handles data processing, forest tasks, and the miscellaneous periodic scripts.
Manager still needs to run the log-uploading script, hmmmm. make a new server-task task-queue?
This will increase server costs, but unless we swap to a container-based backend its going to be painful to solve this. We can default this server type to a cheapo server though, push notifications currently take a matter of seconds every few minutes - But I would need to test this.