NVIDIA / spark-rapids-tools

User tools for Spark RAPIDS
Apache License 2.0
44 stars 34 forks source link

[BUG] Reduce the maximum number of Java threads #1079

Closed amahussein closed 4 weeks ago

amahussein commented 4 weeks ago

Describe the bug

Currently, there CLI sets the number of threads of the Java cmd based on available physical memory. The calculation will set a cap not to exceed 6 parallel threads.

On big machines, we noticed that large number of parallel threads reading from CSP eventlogs could trigger Linux OOM because of the huge memory consumption. The work around was to lower the number of threads to 1.

This issue is to reduce the maximum number of threads from 6 to 3; so that the Tools java process won't cause OS resource starvation when the users are not setting the thread count.