[X] I have searched in the issues and found no similar issues.
What would you like to be improved?
We experienced and suffered from AMS full GC for a long period. we dig into this case and finally found out local spark session terminal triggers java.lang.System.gc() every 30 minutes to clean contexts. As arthas stack traces as shown below:
How should we improve?
For external kyuubi session terminal, there is no need to revises anything. The problem is only related to local session which is embedding into AMS itself. Simply add config 'spark.cleaner.referenceTracking' = 'false' to solve the problem.
Search before asking
What would you like to be improved?
We experienced and suffered from AMS full GC for a long period. we dig into this case and finally found out local spark session terminal triggers java.lang.System.gc() every 30 minutes to clean contexts. As arthas stack traces as shown below:
How should we improve?
For external kyuubi session terminal, there is no need to revises anything. The problem is only related to local session which is embedding into AMS itself. Simply add config 'spark.cleaner.referenceTracking' = 'false' to solve the problem.
Are you willing to submit PR?
Subtasks
No response
Code of Conduct