I have a query returning 600000 cells. I want to export result to excel.
Export was performed to long and consumed to much cpu.
Profiller shows that hotspot is org.pivot4j.util.TreeNode.getWidthhttps://github.com/mysticfall/pivot4j/blob/master/pivot4j-core/src/main/java/org/pivot4j/util/TreeNode.java#L155
The method called recursively many times. Much more than count of cells. The same TreeNodes width is calculated many times for different parents. We need "cache" result ofTreeNode.getWidthand reuse calculated value for subsequent calls. TreeNode.getMaxDescendantLevel` is not so hot spot as getWidth, but is hot too. We need "cache" it too.
After doing that export time and cpu consumption drammatically decreased. More than 10 times.
We need "invalidate cache" when children collection changed. It is easy because getChildren returns unmodifiableList so We have control all modifications of children;
Memory profiler shows tonns of garbage UnmodifiableRandomAccessList produced by getChildren().
https://github.com/mysticfall/pivot4j/blob/master/pivot4j-core/src/main/java/org/pivot4j/util/TreeNode.java#L215
It is immediately collected. But that volumes of garbage triggers to many gc. So export performed slowly.
Imho, in this case, we should allocate private List<TreeNode<T>> unmodifiableChildren = Collections.unmodifiableList(children) at creation time and return it with getChildren. Rather than allocate on each call.
mysticfall
commented
6 years ago
I have a query returning 600000 cells. I want to export result to excel. Export was performed to long and consumed to much cpu.
org.pivot4j.util.TreeNode.getWidthhttps://github.com/mysticfall/pivot4j/blob/master/pivot4j-core/src/main/java/org/pivot4j/util/TreeNode.java#L155 The method called recursively many times. Much more than count of cells. The same TreeNodes width is calculated many times for different parents. We need "cache" result ofTreeNode.getWidthand reuse calculated value for subsequent calls.TreeNode.getMaxDescendantLevel` is not so hot spot as getWidth, but is hot too. We need "cache" it too. After doing that export time and cpu consumption drammatically decreased. More than 10 times.We need "invalidate cache" when children collection changed. It is easy because getChildren returns unmodifiableList so We have control all modifications of
children;UnmodifiableRandomAccessListproduced bygetChildren(). https://github.com/mysticfall/pivot4j/blob/master/pivot4j-core/src/main/java/org/pivot4j/util/TreeNode.java#L215 It is immediately collected. But that volumes of garbage triggers to many gc. So export performed slowly. Imho, in this case, we should allocateprivate List<TreeNode<T>> unmodifiableChildren = Collections.unmodifiableList(children)at creation time and return it withgetChildren. Rather than allocate on each call.