search

fsdonks / proc

MARATHON post proccessor
0 stars 0 forks source link

Optimize chart creation / incanter dataset parsing #18

Closed fs-tom closed 7 years ago

fs-tom commented 7 years ago

I plugged in an optimization for proc.core/dwell-over-fill charts in m4, specifically m4.vnv.

We a) use string pools to eliminate string bloat, and b) parse faster using spork.util.table with parsing schemas, and c) extend proc.util/as-dataset to cover spork.util.table tables.

This reduces the parsing time drastically, which is currently what appears to be dominating our ability to render said charts (not terrible, ~17 minutes for 16 charts, but should be much faster).

So, migrate the patches from m4 .vnv over to proc [as a near-term optimization] and re-test performance.

fs-tom commented 7 years ago

Migrated to a slightly customized incanter 1.9.1-SNAPSHOT build, based off the dev branch of incanter. The 1.9.1 release on clojars is broken, since it has a janky dependency to an ancient clojure via swingrepl. weird! This snapshot is just a current build of dev.

Got most of the chart-io related bottlenecks sorted out. Stuff from M4 vnv is ported and working. Without any additional tweaks, we're post-processing an entire large dataset and generating canonical charts in a self-contained .pptx file, without any user interaction. Takes ~55s for pptx generation (building and render charts and the like), and a couple of minutes for post processing (this is a major bottleneck).