OOME Error on pull request

[marakiis]

It seems there is a memory leak on the R66 client. While transmitting a big file (>400Mo) an OOME occurs.

client.log.2020-07-01.0.log

I suspect data packets are not discarded correctly.

[fredericBregier]

I've looked at it. I've made a IT test for that:

• limitation of bandwhidth (seems to be used in your configuration ?) : 10 MBps and also without any limitation
• file size : 500 MB
• Using Direct Send/Recv, Submit Send/Recv, TLS/No-TLS
• Recv and Send: both at around 10 MBps without any OOME (roughly 50s each), without limitation around 25 MBps (disk speed = 400 Mbps) (roughly 20 s each)

Could you provide the configuration and scenario?

[fredericBregier]

And of course, which version (3.3.4) ?

[bcarlin]

The version is 3.3.4, with an Xmx at 512m.

[marakiis]

The case is on a msend with a recv rule

[fredericBregier]

Tests done, using -Xmx512m, send and recv, tls and not tls, limit bandwidth and no limit, direct and submit, no issue until now.

Waiting for more information... ;-)

[marakiis]

Hi, It seems the OOME was caused by an insufficient amount of RAM on the machine (only 400Mo were available). However it doesn't feel right that a R66Client needs that amount of memory.

[fredericBregier]

Well, we made already a lot of memory optimization (more than 40% less). Memory footprint is a combination of active services:

• Of course R66 itself: roughly 200 MB (well, one can limit more choosing to activate only one among TLS or NO_TLS)
• Then other services: Web Interface (mandatory), REST V1, REST V2, SNMP, THRIFT, Monitoring, CPU limitation, Bandwidth limitation : roughly 50MB each

400 MB is really a very short amount of memory for usual server. If everything is deactivated, it might be OK.

My feeling is that 1GB for modern servers is quite normal for a Java program, including all those options.

However, REST V2 for instance could be optimized a bit (no configuration at all on thread or memory consumption were made).

[bcarlin]

I concur with you: If Waarp R66 is started as a server, esp. for a production server, 1GB is a minimum. but for a client (here it is a client that OOMEed), there is no web interfaces, REST APIs etc...

It is all very empirical, but it seems I need to put my Xmx at 128m to pull a 1.5GB file from a server without TLS, where my client is configured with runlimit=1, serverthread=1 and clientthread=1. This seems to me a little weird, even if for a java app, it is not that big...

[fredericBregier]

Well, Java memory footprint by itself is quite huge (note that there are about 30 MB of native jar, then the JRE itself that shall be loaded). By default, Java needs at least 100 MB (so -Xmx128m is the minimum).

Nothing to do about that...

[bcarlin]

I reckon...

[fredericBregier]

Small improvement in last MR.

Doing the following works for a client (direct transfer or submit transfer):

• Add -Xmx128m to java line command
• Set in the XML client configuration:
  • runlimit to 1
  • serverthread to 1
  • clientthread to 1

Doing the following works for a Server according to related needs:

• For server, it is still recommended to have 1GB, but one can limit the memory fooprint by:
  • Add -Xmx512m to java line command (minimum recommended)
  • Adapt the XML configuration according to context:
  • runlimit to a number between 100 to 1000 (at most 1000)
  • serverthread to 1 to 32 (at most 2x number of cores)
  • clientthread to 10 to 100 (at most 10x serverthread)
  • memorylimit to a number between 100000 to 1000000 (100MB to 1GB) for web and Rest purpose only
  • Don't activate unused sub-services such as: REST (V1 and V2), SNMP, THRIFT, CPU limitation, Bandwidth limitation

[bcarlin]

Ok... So I have played a little and made some tests, just for fun.

With a small script, I benchmarked the RAM usage of some JVM (mainly distributions of OpenJDK).

The client was setup with runlimit=1, serverthread=1 and clientthread=1. It used a rule in received mode without any task, to get a 1.5GB file without TLS from a server installed on my host machine.

I recorded the apparent RAM usage as it is viewed by the kernel (RSS memory). Here are the results :

The x axis is the time elapsed from the start of the container (roughly 1 data point every 0.5 second).

All the Hotspot based JVM are in the same range, with Amazon Corretto running on Amazon Linux being slightly below.

JVMs based on OpenJ9, however, are a surprise as far as memory consumption is concerned, since they used ~50% less RAM, without a too significative impact on transfer duration.

Each container was started with the default settings. I did not include Java 7 (I was curious) because the transfer failed with UnsupportedClassVersionError: org/snmp4j/agent/MOGroup : Unsupported major.minor version 52.0.

[fredericBregier]

Added small improvements on Hash computation (limit byte copies, so memory footprint and cpu usage)

[fredericBregier]

This bug should be closed once the MR is merged. A reminder:

• client only: 128M is possible, while 512M is still recommended
• server: 512M is possible, while 1G is still recommended as the minimum (2 to 4G for heavy loaded server)

[bcarlin]

Yes. I'm closing it since the issue was not bound to the application but to the system: there was less RAM available on the server than the Xmx configured (The JVM requested 512Mo, but the system only had 400Mo available).

I'm sending a PR to include the RAM requirements in the doc.