search

apache / dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code
https://dolphinscheduler.apache.org/
Apache License 2.0
12.73k stars 4.58k forks source link

[Bug] [api-server, worker, alert, master] direct memory keeps increasing #16661

Open 1017005846 opened 4 days ago

1017005846 commented 4 days ago

Search before asking

What happened

服务启动之后直接内存一直会增加，直到超过容器限制进行重启

What you expected to happen

内存不溢出

How to reproduce

服务启动后通过arthas，用memory命令可以看到direct区域一直在增加

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

Code of Conduct

github-actions[bot] commented 4 days ago

Search before asking

What happened

After the service is started, the direct memory will continue to increase until it exceeds the container limit and is restarted.

What you expected to happen

Memory does not overflow

How to reproduce

After the service is started, through arthas and the memory command, you can see that the direct area has been increasing.

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

Code of Conduct

SbloodyS commented 3 days ago

Can you provide the reproduction steps? @1017005846

1017005846 commented 3 days ago

Can you provide the reproduction steps? @1017005846

我用源码进行打包部署，分别部署dolphin-api、work、master 三个容器，jvm配置参数为-Xmx2648m -Xms2648m -Xmn960m -XX:MaxMetaspaceSize=512m -XX:MetaspaceSize=512m ，启动后通过arthas的memory命令查看服务内存，发现direct占用800+M，并且一直在增加，当超过pod限制的内存后pod触发重启

SbloodyS commented 3 days ago

Please using english to describe.

1017005846 commented 3 days ago

Please using english to describe.

I use source-code to package and deploy,i deploy three pod dolphin-api,worker and master, jvm config is -Xmx2648m -Xms2648m -Xmn960m -XX:MaxMetaspaceSize=512m -XX:MetaspaceSize=512m . when i start the service i use arthas to monitor the memory ,i found direct memory used 800+M and Continuously increasing when it over pod limit it will restart. Another, i use prometheus to monitor it suddenly increase from 2.5G to 4G image

SbloodyS commented 3 days ago

Which version you are using? The direct memory is off-heap space used by nio. It will not be used when starting with the default parameters. Please provide your usage mode and parameters.