search

apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.7k stars 3.28k forks source link

[Bug] Doris fe 自己莫名其妙关闭了 Doris fe closed it herself 感觉是内存溢出了， top看一切正常，但是服务器只有刚才be分配的8GB内存空闲，且be fe都退出了 #33409

Open KongFuPandaa opened 7 months ago

KongFuPandaa commented 7 months ago

Search before asking

Version

doris-1.2.3-rc02

What's Wrong?

doris fs stop it's self over 10 times recently

2024-04-09 05:01:04,817 WARN (thrift-server-pool-1|190) [TIOStreamTransport.close():110] Error closing output stream.
java.net.SocketException: Socket closed
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118) ~[?:1.8.0_312]
    at java.net.SocketOutputStream.write(SocketOutputStream.java:155) ~[?:1.8.0_312]
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[?:1.8.0_312]
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[?:1.8.0_312]
    at java.io.FilterOutputStream.close(FilterOutputStream.java:158) ~[?:1.8.0_312]
    at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:108) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.transport.TSocket.close(TSocket.java:235) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:328) ~[libthrift-0.13.0.jar:0.13.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
2024-04-09 05:01:05,050 ERROR (thrift-server-pool-23|1872) [TThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:411) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:254) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) ~[libthrift-0.13.0.jar:0.13.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_312]
    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_312]
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[?:1.8.0_312]
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_312]
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
    ... 8 more
2024-04-09 05:01:05,195 ERROR (thrift-server-pool-27|14029) [TThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:127) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:411) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:254) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:27) ~[libthrift-0.13.0.jar:0.13.0]
    at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:313) ~[libthrift-0.13.0.jar:0.13.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
    at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
Caused by: java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210) ~[?:1.8.0_312]
    at java.net.SocketInputStream.read(SocketInputStream.java:141) ~[?:1.8.0_312]
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:284) ~[?:1.8.0_312]
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345) ~[?:1.8.0_312]
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125) ~[libthrift-0.13.0.jar:0.13.0]
    ... 8 more

What You Expected?

save my doris

How to Reproduce?

No response

Anything Else?

Server Config: DD: 4TB MEM: 32G CPU: 16 core

Are you willing to submit PR?

Code of Conduct

KongFuPandaa commented 7 months ago

有大人管管这个问题吗