The WebSocket server causes a lot of load. Before #566, it regularly pinned a core on the AT instance. That change improves the situation, but it's not a real solution, especially not in the longer term if capacity were to grow further.
Some ideas:
Batch messages so they can be compressed and sent at once.
Reimplement in a different language that's more efficient at the task than Python.
Reduce the data sent to the clients. The obvious idea here would be turning it into pub-sub, i.e. making clients subscribe to jobs. For example, the dashboard would subscribe to any job that's currently visible per the filters. This may necessitate buffering messages on the server so they can be sent when the user changes the filter. Stats updates would be sent for any jobs that clients haven't subscribed to.
The WebSocket server causes a lot of load. Before #566, it regularly pinned a core on the AT instance. That change improves the situation, but it's not a real solution, especially not in the longer term if capacity were to grow further.
Some ideas: