Optimize GzipMaxInputIT in order to reduce the JVM memory consumption

[jcarranzan]

Summary

Refactors GzipMaxInputIT to reduce memory usage, avoiding spikes in memory consumption.

I compared prints on our Jenkins jobs in the previous one GzipMaxInputIT and observed a rapid spike in memory consumption, but currently, these changes are almost negligible.

This approach addresses the memory issues observed in Podman Jenkins jobs related to Keycloak container failures.

Please select the relevant options.

• [ ] Bug fix (non-breaking change which fixes an issue)
• [ ] Dependency update
• [x] Refactoring
• [ ] Backport
• [ ] New scenario (non-breaking change which adds functionality)
• [ ] This change requires a documentation update
• [ ] This change requires execution against OCP (use run tests phrase in comment)

Checklist:

• [x] Methods and classes used in PR scenarios are meaningful
• [x] Commits are well encapsulated and follow the best practices

[rsvoboda]

Do you have jvisualvm or JDK Mission Control screenshots of memory usage for case before and after? So the impact of your changes can be more clear.

[jcarranzan]

From my jcarranz-baremetal-ts-jvm-podman-nightly (33) job execution (you look there for more details) :

12:04:01 [After sendPayloadBelowMaxInputLimit Test]
12:04:01 === JVM Memory Info ===
12:04:01 Free Memory: 554 MB
12:04:01 Total Memory: 1002 MB
12:04:01 Max Memory: 1924 MB
12:04:01 === System Memory Info ===
12:04:01 Total Physical Memory: 7695 MB
12:04:01 Free Physical Memory: 1359 MB
12:04:01 --------------------------------------------------
12:04:01 11:04:01,234 INFO  ## Running test GzipMaxInputIT.sendInvalidContent()
12:04:01 [After sendInvalidContent Test]
12:04:01 === JVM Memory Info ===
12:04:01 Free Memory: 260 MB
12:04:01 Total Memory: 1002 MB
12:04:01 Max Memory: 1924 MB
12:04:01 === System Memory Info ===
12:04:01 Total Physical Memory: 7695 MB
12:04:01 Free Physical Memory: 1257 MB
12:04:01 --------------------------------------------------
12:04:01 11:04:01,394 INFO  ## Running test GzipMaxInputIT.sendMaximumAllowedPayload()
12:04:02 [After sendMaximumAllowedPayload Test]
12:04:02 === JVM Memory Info ===
12:04:02 Free Memory: 347 MB
12:04:02 Total Memory: 1365 MB
12:04:02 Max Memory: 1924 MB
12:04:02 === System Memory Info ===
12:04:02 Total Physical Memory: 7695 MB
12:04:02 Free Physical Memory: 794 MB
12:04:02 --------------------------------------------------
12:04:02 11:04:02,397 INFO  ## Running test GzipMaxInputIT.sendMoreThanMaximumAllowedPayload()
12:04:03 [After sendMoreThanMaximumAllowedPayload Test]
12:04:03 === JVM Memory Info ===
12:04:03 Free Memory: 245 MB
12:04:03 Total Memory: 1566 MB
12:04:03 Max Memory: 1924 MB
12:04:03 === System Memory Info ===
12:04:03 Total Physical Memory: 7695 MB
12:04:03 Free Physical Memory: 364 MB
12:04:03 --------------------------------------------------
12:04:03 11:04:03,973 INFO  ## Running test GzipMaxInputIT.sendZeroBytesPayload()
12:04:03 [After sendZeroBytesPayload Test]
12:04:03 === JVM Memory Info ===
12:04:03 Free Memory: 244 MB
12:04:03 Total Memory: 1868 MB
12:04:03 Max Memory: 1924 MB
12:04:03 === System Memory Info ===
12:04:03 Total Physical Memory: 7695 MB
12:04:03 Free Physical Memory: 130 MB

Currently with my changes (you can see in the same job ( jcarranz-baremetal-ts-jvm-podman-nightly 33 execution) :

10:12:24 === JVM Memory Info ===
10:12:24 Free Memory: 393 MB
10:12:24 Total Memory: 1029 MB
10:12:24 Max Memory: 1924 MB
10:12:24 === System Memory Info ===
10:12:24 Total Physical Memory: 7695 MB
10:12:24 Free Physical Memory: 1443 MB
10:12:24 --------------------------------------------------
10:12:24 [After sending invalid content]
10:12:24 === JVM Memory Info ===
10:12:24 Free Memory: 390 MB
10:12:24 Total Memory: 1029 MB
10:12:24 Max Memory: 1924 MB
10:12:24 === System Memory Info ===
10:12:24 Total Physical Memory: 7695 MB
10:12:24 Free Physical Memory: 1425 MB
10:12:24 --------------------------------------------------
10:12:24 09:12:24,456 INFO  ## Running test GzipMaxInputIT.sendInvalidContent()
10:12:24 [Before sending maximum allowed payload]
10:12:24 === JVM Memory Info ===
10:12:24 Free Memory: 390 MB
10:12:24 Total Memory: 1029 MB
10:12:24 Max Memory: 1924 MB
10:12:24 === System Memory Info ===
10:12:24 Total Physical Memory: 7695 MB
10:12:24 Free Physical Memory: 1425 MB
10:12:24 --------------------------------------------------
10:12:24 [After sending invalid content]
10:12:24 === JVM Memory Info ===
10:12:24 Free Memory: 390 MB
10:12:24 Total Memory: 1029 MB
10:12:24 Max Memory: 1924 MB
10:12:24 === System Memory Info ===
10:12:24 Total Physical Memory: 7695 MB
10:12:24 Free Physical Memory: 1425 MB
10:12:24 --------------------------------------------------
10:12:24 09:12:24,473 INFO  ## Running test GzipMaxInputIT.sendMaximumAllowedPayload()
10:12:24 [Before sending maximum allowed payload]
10:12:24 === JVM Memory Info ===
10:12:24 Free Memory: 390 MB
10:12:24 Total Memory: 1029 MB
10:12:24 Max Memory: 1924 MB
10:12:24 === System Memory Info ===
10:12:24 Total Physical Memory: 7695 MB
10:12:24 Free Physical Memory: 1423 MB
10:12:24 --------------------------------------------------
10:12:25 [After sending invalid content]
10:12:25 === JVM Memory Info ===
10:12:25 Free Memory: 389 MB
10:12:25 Total Memory: 1029 MB
10:12:25 Max Memory: 1924 MB
10:12:25 === System Memory Info ===
10:12:25 Total Physical Memory: 7695 MB
10:12:25 Free Physical Memory: 1068 MB
10:12:25 --------------------------------------------------
10:12:25 09:12:25,423 INFO  ## Running test GzipMaxInputIT.sendMoreThanMaximumAllowedPayload()
10:12:25 [Before sending maximum allowed payload]
10:12:25 === JVM Memory Info ===
10:12:25 Free Memory: 389 MB
10:12:25 Total Memory: 1029 MB
10:12:25 Max Memory: 1924 MB
10:12:25 === System Memory Info ===
10:12:25 Total Physical Memory: 7695 MB
10:12:25 Free Physical Memory: 1068 MB
10:12:25 --------------------------------------------------
10:12:26 [After sending invalid content]
10:12:26 === JVM Memory Info ===
10:12:26 Free Memory: 387 MB
10:12:26 Total Memory: 1029 MB
10:12:26 Max Memory: 1924 MB
10:12:26 === System Memory Info ===
10:12:26 Total Physical Memory: 7695 MB
10:12:26 Free Physical Memory: 1062 MB
10:12:26 --------------------------------------------------
10:12:26 09:12:26,897 INFO  ## Running test GzipMaxInputIT.sendZeroBytesPayload()
10:12:26 [Before sending maximum allowed payload]
10:12:26 === JVM Memory Info ===
10:12:26 Free Memory: 387 MB
10:12:26 Total Memory: 1029 MB
10:12:26 Max Memory: 1924 MB
10:12:26 === System Memory Info ===
10:12:26 Total Physical Memory: 7695 MB
10:12:26 Free Physical Memory: 1062 MB
10:12:26 --------------------------------------------------
10:12:26 [After sending invalid content]
10:12:26 === JVM Memory Info ===
10:12:26 Free Memory: 387 MB
10:12:26 Total Memory: 1029 MB
10:12:26 Max Memory: 1924 MB
10:12:26 === System Memory Info ===
10:12:26 Total Physical Memory: 7695 MB
10:12:26 Free Physical Memory: 1062 MB
10:12:26 --------------------------------------------------
10:12:28 09:12:28,962 INFO  [app] Service stopped (Quarkus JVM mode)
10:12:28 [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time

[michalvavrik]

I wonder why many-extensions appeared between changed files, please try to rebase n current main.

[jcarranzan]

I wonder why many-extensions appeared between changed files, please try to rebase n current main.

I don't see it, are u checking with a git command?

[michalvavrik]

I wonder why many-extensions appeared between changed files, please try to rebase n current main.

I don't see it, are u checking with a git command?

CI failed

[jcarranzan]

@michalvavrik related to your questions:

• How many times have you tried it?

  I think memory is funny thing, I'd like to know that this is consistent improvement. I am running again some Jenkins jobs, but this morning and yesterday locally I also did some runs. I agree wh you regarding to get this as a consistent improvement.

• Where is remaining 6633 MB? While I included also the printout of both memories(JVM and System Memory) this PR is just about JVM memory usage within our tests. That 6633MB is system memory, probably used by other processes and so on.