[KIEKER-671] GraphElement.origins causes heavy memory usage with large monitoring logs

[rju]

JIRA Issue: KIEKER-671 GraphElement.origins causes heavy memory usage with large monitoring logs Original Reporter: Andre van Hoorn

---

For example, when plotting dependency graphs, a reference to each MessageTrace observed is kept. This causes the memory usage to increase by each new trace.

Temporarily deactivated the origin feature in GraphElement:

   /**
     * Adds a new origin object to this element.
     * 
     * @param origin
     *            The origin object
     */
    public void addOrigin(final O origin) {
        //this.origins.add(origin);
    }

Effect:

We might want to add an option for graph filters whether or not the origin should be kept?

[rju]

author holgerknoche -- Thu, 13 Sep 2012 14:45:49 +0200

Planned solution: Save the trace information data (and not the trace itself) as the origin

[rju]

author holgerknoche -- Thu, 13 Sep 2012 16:56:26 +0200

Graph elements now refer to the trace metadata instead of the trace itself. Thus, the actual trace data can be removed from memory when desired.

[rju]

author André van Hoorn -- Fri, 14 Sep 2012 16:48:04 +0200

Reopening because problem remains.

[rju]

author André van Hoorn -- Mon, 17 Sep 2012 19:00:27 +0200

Replying to [avh|comment:4]:
> Reopening because problem remains.

Some measurements with the data set from above:

Unable to render embedded object: File (20120917-origin-bug-mem-cpu.png, width=100%) not found.

Unable to render embedded object: File (20120917-origin-bug-object-sizes.png, width=100%) not found.

[rju]

author holgerknoche -- Wed, 26 Sep 2012 16:05:42 +0200

Looks like the real problem is the size of the hash sets which are used to manage the graph element's origins. I will prepare a fix which stores the origin data only when they are explicitly requested by a consumer plugin.

[rju]

author André van Hoorn -- Thu, 4 Oct 2012 12:05:14 +0200

Holger's version from the branch ConnectionNotification executed successfully and also the mem usage looks OK (see below).

voornpc-vanhoorn:0:~/git_work/kieker <ConnectionNotification> $ date
Do 4. Okt 12:04:31 CEST 2012
voornpc-vanhoorn:0:~/git_work/kieker <ConnectionNotification> $ time bin/trace-analysis.sh -i $KIEKER_DIRS -o /tmp/kieker-out/ --plot-Deployment-Component-Dependency-Graph --plot-Assembly-Component-Dependency-Graph --plot-Deployment-Operation-Dependency-Graph --plot-Assembly-Operation-Dependency-Graph --plot-Container-Dependency-Graph --print-invalid-Execution-Traces --ignore-invalid-traces -p ewe-complete- --max-trace-duration 18000000

...

real    43m0.089s
user    42m45.008s
sys     6m19.048s
voornpc-vanhoorn:0:~/git_work/kieker <ConnectionNotification> $

Unable to render embedded object: File (583-ConnectionNotification-cpuheap.png,width=100%) not found.

Unable to render embedded object: File (583-ConnectionNotification-objects.png,width=100%) not found.

[rju]

author holgerknoche -- Fri, 5 Oct 2012 17:46:29 +0200

Bug has been fixed in conjunction with KIEKER-514 Done .