The performance degradation from the previous approach seems to be coming from running the bot as a service. After this pull request we'll instead run the bot as a normal process and, in parallel, also have a monitoring service dedicated for feeding the bot's output to the systemd journal, that way it'll be available on Grafana's logs.
This MR's main goal is to refactor bench-bot's setup so that it'll be possible to have the logs + monitoring without overhead.
After https://github.com/paritytech/substrate/pull/9507#issuecomment-909633326 was reported, I tried to look into the logs to but the bot is currently running in @shawntabrizi's session which has limited buffer history size (I think it's 2000 lines). We once had the bot running in a service with full access to the logs but apparently that introduced some overhead (https://github.com/paritytech/bench-bot/pull/47#issuecomment-879392756).
The performance degradation from the previous approach seems to be coming from running the bot as a service. After this pull request we'll instead run the bot as a normal process and, in parallel, also have a monitoring service dedicated for feeding the bot's output to the systemd journal, that way it'll be available on Grafana's logs.