search

ContentSquare / chproxy

Open-Source ClickHouse http proxy and load balancer
https://www.chproxy.org/
MIT License
1.29k stars 258 forks source link

[BUG] Healthcheck generate tcp connection reset by peer on clickhouse server #452

Open lchabert opened 3 months ago

lchabert commented 3 months ago

Describe the bug When i run clickhouse proxy, whithout any usefull trafic (only heatbeat), it seems tcp session is broken prematuraly. The TCP connection is raised as an error on clickhouse logs. I'm running clickhouse with TLS.

To Reproduce Run a clickhouse server and clickhouse proxy. Do not run any query on client side. Just run the natural chproxy healthchecks.

Expected behavior Probably tcp connection should be closed correctly.

Logs Clickhouse error logs:

2024.08.07 08:15:27.726323 [ 2443 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket (peer: [::ffff:10.0.223.151]:43942, local: [::ffff:10.0.223.153]:8443). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d01e6db
1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000d16043e
2. DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000d15ff1a
3. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr<DB::IHTTPContext>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000012afde82
4. DB::HTTPServerConnection::run() @ 0x0000000012afc6b8
5. Poco::Net::TCPServerConnection::start() @ 0x00000000153ce307
6. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153ce799
7. Poco::PooledThread::run() @ 0x000000001539b4a1
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015399a5d
9. ? @ 0x00007f2770f9d609
10. ? @ 0x00007f2770eb8353
 (version 24.6.2.17 (official build))
2024.08.07 08:18:52.836394 [ 2443 ] {} <Error> ServerErrorHandler: Code: 210. DB::NetException: Connection reset by peer, while reading from socket (peer: [::ffff:10.0.223.151]:39624, local: [::ffff:10.0.223.153]:8443). (NETWORK_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000d01e6db
1. DB::NetException::NetException<String, String, String>(int, FormatStringHelperImpl<std::type_identity<String>::type, std::type_identity<String>::type, std::type_identity<String>::type>, String&&, String&&, String&&) @ 0x000000000d16043e
2. DB::ReadBufferFromPocoSocket::nextImpl() @ 0x000000000d15ff1a
3. DB::HTTPServerRequest::HTTPServerRequest(std::shared_ptr<DB::IHTTPContext>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&, StrongTypedef<unsigned long, ProfileEvents::EventTag> const&) @ 0x0000000012afde82
4. DB::HTTPServerConnection::run() @ 0x0000000012afc6b8
5. Poco::Net::TCPServerConnection::start() @ 0x00000000153ce307
6. Poco::Net::TCPServerDispatcher::run() @ 0x00000000153ce799
7. Poco::PooledThread::run() @ 0x000000001539b4a1
8. Poco::ThreadImpl::runnableEntry(void*) @ 0x0000000015399a5d
9. ? @ 0x00007f2770f9d609
10. ? @ 0x00007f2770eb8353
 (version 24.6.2.17 (official build))

Environment information Clickhouse: run in container, version 24.6.2-alpine Clickhouse proxy: run in container, version v1.26.4