apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
8.06k stars 1.83k forks source link

[Bug] [zeta] java.lang.OutOfMemoryError:Metaspace #7972

Open wssmao opened 2 weeks ago

wssmao commented 2 weeks ago

Search before asking

What happened

OOM appear, when I run st cluster after a long time .

SeaTunnel Version

2.3.5

SeaTunnel Config

# JVM Heap
-Xms6g
-Xmx6g

# JVM Dump
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/tmp/seatunnel/dump/zeta-server

# Metaspace
-XX:MaxMetaspaceSize=6g

# G1GC
-XX:+UseG1GC

Running Command

nohup ./bin/seatunnel.sh

Error Exception

Exception in thread "main" org.apache.seatunnel.core.starter.exception.CommandExecuteException: SeaTunnel job executed failed
        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:202)
        at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
        at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
    Caused by: org.apache.seatunnel.engine.common.exception.SeaTunnelEngineException: org.apache.seatunnel.engine.common.exception.TaskGroupDeployException: java.lang.OutOfMemoryError: Metaspace

        at org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex.stateProcess(PhysicalVertex.java:561)
[INFO] 2024-11-04 09:20:38.450  - [taskAppId=TASK-1519-5417926-7668604]:[127] -  ->     at org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex.updateTaskState(PhysicalVertex.java:392)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex.makeTaskGroupDeploy(PhysicalVertex.java:319)
        at org.apache.seatunnel.engine.server.dag.physical.SubPlan.lambda$stateProcess$19(SubPlan.java:627)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.apache.seatunnel.engine.server.dag.physical.SubPlan.stateProcess(SubPlan.java:623)
        at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:360)
        at org.apache.seatunnel.engine.server.dag.physical.SubPlan.stateProcess(SubPlan.java:609)
        at org.apache.seatunnel.engine.server.dag.physical.SubPlan.updatePipelineState(SubPlan.java:360)
        at org.apache.seatunnel.engine.server.dag.physical.SubPlan.stateProcess(SubPlan.java:596)
        at org.apache.seatunnel.engine.server.dag.physical.SubPlan.startSubPlanStateProcess(SubPlan.java:580)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan.lambda$stateProcess$3(PhysicalPlan.java:312)
        at java.util.ArrayList.forEach(ArrayList.java:1257)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan.stateProcess(PhysicalPlan.java:308)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan.updateJobState(PhysicalPlan.java:260)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan.stateProcess(PhysicalPlan.java:304)
        at org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan.startJob(PhysicalPlan.java:289)
        at org.apache.seatunnel.engine.server.master.JobMaster.run(JobMaster.java:315)
        at org.apache.seatunnel.engine.server.CoordinatorService.lambda$submitJob$3(CoordinatorService.java:488)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

        at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:194)
        ... 2 more

Zeta or Flink or Spark Version

No response

Java or Scala Version

1.8

Screenshots

No response

Are you willing to submit PR?

Code of Conduct

Hisoka-X commented 2 weeks ago

Could you try enable this? https://seatunnel.apache.org/docs/2.3.5/seatunnel-engine/deployment#45-classloader-cache-mode