load generator may perform too much work on the main thread

[MonsieurNicolas]

Right now it looks like loadgen performs a lot (too much?) of heavy work on the main thread:

• it fetches account data for all accounts from the database when waiting for the simulation to end using point loads instead of batching.
  • Maybe we could both limit the number of queries performed and remove accounts from that list when they reach the right sequence number (to avoid querying them all the time)?
• it fetches accounts from the database for the current "step" using "point loads" (ie, not using batch loading)
• it's generating transactions, in particular digital signatures, for each transaction for the current "step"

As loadgen is scheduled like anything else, I wonder if there is some weird scheduling interaction between the node doing "real work" and performing loadgen related work.

I don't know if it's easy to quantify that effect, or if we should just mitigate all of those issues somehow.

[marta-lokhova]

this came up again during theoretical TPS measurements that @bboston7 did, so should probably be prioritized soon

[bboston7]

Another acceptable solution (if it works) that came up during the planning meeting is to adjust our tests to generate load from watcher nodes and increase the resources available to those watcher nodes in SSC. This might be a good approach if backgrounding loadgen tasks is painful.