search

fsdonks / proc

MARATHON post proccessor
0 stars 0 forks source link

Optimize runtime #13

Open fs-tom opened 7 years ago

fs-tom commented 7 years ago

Profile intermediate steps in computing fills files, see where we're bottlenecking and reduce CPU footprint (or do things in parallel).

fs-tom commented 7 years ago

Most bottlenecking is occurring after we load entity locations, and try to recreate history for each entity. The problem we have now is we do a bulk computation, using group-by to get each entity's location and related info into its history. we could be streaming this, using a variant of the stuff in spork.util.temporal, or even something akin to spork.sim.simcontext, or even just recreating the simulation state from the textual history [and stepping through/collecting sequentially], OR read from a binary history. Many possibilities to explore; all of them should avoid doing too many group-by and such aggregate operations that require us to step through the entire history for an individual entity.