enabling session serialization debug triggers OOM

[jorgheymans]

We are trying to troubleshoot session serialization errors, but when navigating to the suspected pages with session serde debug enabled, we immediately get OOM. We tried increasing heap to 1G, 2G, 3G but it does not make a difference which could indicate memory is being leaked somewhere.

Note that the application itself can start and run without any Xmx setting, so it's not memory intensive by nature.

Heapdumps available on request, in case that helps.

[mcollovati]

@jorgheymans thanks for reporting. Heap dumps will be helpful to investigating the issue indeed.

Please contact me at marco at vaadin dot com, so we can agree on a way to share the dumps.

[mcollovati]

Thanks, @jorgheymans, for providing useful details

[mcollovati]

One of the potential causes of the OOM seems to be that to provide sensible information in case of deserialization errors, the tool collects the trace info produced by ObjectOutputStream.DebugTraceInfoStack for every processed object. Getting it as a string produces a new String object for every traced object. Instead, storing the information as a Collection<String> should reuse the existing instances and save memory.

[jorgheymans]

You mean allowing the jdk to do it's String interning thing? That will help, yet i do wonder how it fills up 3G of heap so rapidly, it's really 1-2 seconds at most.

[mcollovati]

No, I mean that JDK ObjectOutputStream is storing traces in a List<String>, but the debug tool is doing a join of the list for every serialized object, so generating additional strings (or at least it seems so inspecting a heap dump). What we can do is reuse the existing Strings, avoiding the call to DebugTraceInfoStack.toString().

[mcollovati]

To be noted that all the collected strings are currently also serialized, thus increasing the size of the in memory binary data. And this should be multiplied for the number of concurrent requests. We are working on several fixes.

[mcollovati]

Here's a log excerpt from the current version of the kit.

Serialization of attributes [com.vaadin.flow.server.VaadinSession.springServlet, springServlet.lock, clusterKey] for session DEBUG-SERIALIZE-47C51F88917092CCC072627D0EEE31CF with distributed key 4f8dc544-7f86-48be-940b-fda84cb54dc6_SOURCE:47C51F88917092CCC072627D0EEE31CF completed in 3161ms (740145896 bytes)

And this one is after applying the PRs under review

Serialization of attributes [com.vaadin.flow.server.VaadinSession.springServlet, springServlet.lock, clusterKey] for session DEBUG-SERIALIZE-51E5857518441999AE33880A93B736F1 with distributed key 95f35c86-f693-4d52-bea1-f736558f8757_SOURCE:51E5857518441999AE33880A93B736F1 completed in 47ms (418430 bytes)

Both processing times and serialized data sizes look a little better :grin:

Time: 3161ms -> 47ms Size: 740 MB -> 418 KB

[jorgheymans]

That looks like a great (and much needed?) improvement !