search

mezuro / kalibro_processor

Reimplementation of Kalibro processing
GNU Affero General Public License v3.0
2 stars 10 forks source link

Rewrite aggregation processing to be more efficient #223

Closed danielkza closed 8 years ago

danielkza commented 8 years ago

Previously aggregation worked by traversing the modules tree in pre-order. But to ensure that children are aggregated before their parents, we can relax that order a bit to just processes all the results on the same level before all of those on a level above it (the topmost level consisting of the root).

This allows fetching much more results at once and significantly reduce the number of trips to the database - from a number proportional to the number of nodes, to exactly and no more than the maximum depth of the tree.

It also makes it much easier to accumulate the created tree metric results to be created all at once. That also saves a huge number of trips to the database.

Regarding tests: a complete refactor was necessary, and made possible by the module results tree factory. The tests ended up much cleaner and arguably better, as they can verify the actual values being aggregated while mocking only the necessary data accesses.

rafamanzo commented 8 years ago

I believe I've split this into: https://github.com/mezuro/kalibro_processor/pull/222, https://github.com/mezuro/kalibro_processor/pull/224 and https://github.com/mezuro/kalibro_processor/pull/225. What do you think? If you feel like, please update their descriptions as you wish.

danielkza commented 8 years ago

Closed by splitting into #222, #224 and #225.