[Functions] Function workers require more memory on Pulsar 2.8.x+

[lhotari]

Describe the bug Function workers consume more resident RAM memory on Pulsar 2.8.x compared to Pulsar 2.7.x . The "built-in" NAR files from the connectors directory are loaded and retained in memory as NARClassLoaders. This change was made as part of #9413. This could lead to OOM killer killing the function worker on Kubernetes when there are low limits for the function worker. ​

Expected behavior The expectation is that a function worker on Pulsar 2.8.x+ wouldn't require more memory that Pulsar 2.7.x .

[eolivelli]

This is for the Function Worker ? not for the Process that is running the Function. So it is not a big deal, as you usually run only one Worker per machine.

On the Function Process we observe more Heap usage on JDK11 but this is due to how the JVM stores the Jar file contents and we cannot do much.

[lhotari]

This is for the Function Worker ? not for the Process that is running the Function. So it is not a big deal, as you usually run only one Worker per machine.

I didn't measure how big of a deal it is. The impact was that on CircleCI I couldn't get DataStax Pulsar Helm Chart to run with Functions Worker with all .nar files. It seems to be a big deal since CircleCI has 7.5GB RAM by default for machines.

On the Function Process we observe more Heap usage on JDK11 but this is due to how the JVM stores the Jar file contents and we cannot do much.

There is something that can be done. In Pulsar 2.7.x, the NarClassloader was closed after usage. Similar approach could be made possible. It should be possible to resolve the issue that #9413 intended to fix by loading all required information for the built-in connectors once and then closing the NarClassloaders. This would result in lower memory consumption since all NarClassloaders for all connectors wouldn't have to be kept in memory at once.

[github-actions[bot]]

The issue had no activity for 30 days, mark with Stale label.