i receive an HTTP response problem when integrate ClickHouse with MinIO

[nozebraline]

I am Integrating ClickHouse with MinIO（https://clickhouse.com/docs/en/integrations/s3/s3-merge-tree） and occasionally I am receiving an HTTPResponse with a StatusCode = -1.

Below are the entries from the clickhouselog:

2022.09.09 11:01:52.990116 [ 11439 ] {0a3da8d4-ef5b-4bba-8944-43fe1276c0e3} AWSClient: Failed to make request to: http://172.16.124.43:9000/henry/var/mtwmtkhrsaauunchovzqouftnwpoouyk: Poco::Exception. Code: 1000, e.code() = 0, Timeout: connect timed out: 172.16.124.43:9000, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&) in /usr/bin/clickhouse
1. Poco::Net::HTTPSession::connect(Poco::Net::SocketAddress const&) in /usr/bin/clickhouse
2. Poco::Net::HTTPClientSession::reconnect() in /usr/bin/clickhouse
3. Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&) in /usr/bin/clickhouse
4. DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const in /usr/bin/clickhouse
5. DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr const&, Aws::Utils::RateLimits::RateLimiterInterface, Aws::Utils::RateLimits::RateLimiterInterface) const in /usr/bin/clickhouse
6. Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr const&, Aws::AmazonWebServiceRequest const&, char const, char const, char const*) const in /usr/bin/clickhouse
7. Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const, char const, char const*) const in /usr/bin/clickhouse
8. Aws::Client::AWSClient::MakeRequestWithUnparsedResponse(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const, char const, char const*) const in /usr/bin/clickhouse
9. Aws::S3::S3Client::GetObject(Aws::S3::Model::GetObjectRequest const&) const in /usr/bin/clickhouse
10. DB::ReadBufferFromS3::initialize() in /usr/bin/clickhouse
11. DB::ReadBufferFromS3::nextImpl() in /usr/bin/clickhouse
12. DB::ReadBufferFromRemoteFSGather::readImpl() in /usr/bin/clickhouse
13. DB::ReadBufferFromRemoteFSGather::nextImpl() in /usr/bin/clickhouse
14. DB::ReadBufferFromRemoteFSGather::readInto(char*, unsigned long, unsigned long, unsigned long) in /usr/bin/clickhouse
15. ? in /usr/bin/clickhouse
16. std::__1::packaged_task<DB::IAsynchronousReader::Result ()>::operator()() in /usr/bin/clickhouse
17. ThreadPoolImpl::worker(std::1::list_iterator<ThreadFromGlobalPool, void*>) in /usr/bin/clickhouse
18. void std::1::function::policy_invoker<void ()>::__call_impl<std::1::function::default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, int, std::1::optional)::'lambda0'()>(void&&)::'lambda'(), void ()> >(std::1::function::__policy_storage const*) in /usr/bin/clickhouse
19. ThreadPoolImpl::worker(std::1::list_iterator<std::__1::thread, void*>) in /usr/bin/clickhouse
20. ? in /usr/bin/clickhouse Connection reset by 172.16.127.240 port 22libpthread-2.27.so

21. clone in /lib/x86_64-linux-gnu/libc-2.27.so ✘ (version 22.8.4.7 (official build))  09/09/2022   14:00.44   /home/mobaxterm  4bba-8944-43fe1276c0e3} AWSClient: HTTP response code: -1 Resolved remote host IP address: Request ID: Exception name: Error message: Poco::Exception. Code: 1000, e.code() = 0, Timeout: connect timed out: 172.16.124.43:9000 (version 22.8.4.7 (official build)) 0 response headers:

    I suspect the problem is with the network bandwidth, and I try to modify the parameters connect_timeout_ms and request_timeout_ms （https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/#table_engine-mergetree-s3）. but it seems not work.

Please help me solve the problem

[melvynator]

Are you using zero copy replication: https://clickhouse.com/docs/en/guides/sre/s3-multi-region#disable-zero-copy-replication ?

[alesapin]

HTTP code -1 is just default code specified by AWS SDK when it creates HTTPResponse object. If it doesn't receive any response it will return "empty" object with code -1.

[nozebraline]

Are you using zero copy replication: https://clickhouse.com/docs/en/guides/sre/s3-multi-region#disable-zero-copy-replication ?

Although I didn't know why, I solved the problem by increasing the bandwidth to 25Gbps. Thank you for your help.