search

ClickHouse / ClickHouse

ClickHouse® is a real-time analytics DBMS
https://clickhouse.com
Apache License 2.0
36.92k stars 6.82k forks source link

[Delta Lake] Infinite (?) retries in AWSClient #49959

Open melvynator opened 1 year ago

melvynator commented 1 year ago

Query to delta lake is returning timeout.

Even simple queries like: select count(1)

It might be related to: https://github.com/ClickHouse/ClickHouse/issues/48867

tavplubix commented 1 year ago

Provided information is not enough to do anything. Please provide:

alifirat commented 1 year ago

Hey @tavplubix We have generated a dataset with 1776 data files (Parquet format) and we have started to read the data using ClickHouse.

First, the time to create the table takes

clickhouse1 :) SET send_logs_level = 'trace';
clickhouse1:) create table test Engine = DeltaLake('https://cs-tmp.s3.eu-west-1.amazonaws.com/callouis/delta/')
CREATE TABLE test
ENGINE = DeltaLake('https://cs-tmp.s3.eu-west-1.amazonaws.com/callouis/delta/')

Query id: a6342c0c-fa3b-404e-b075-894c8fcd4be4

[clickhouse1] 2023.05.19 07:31:05.766713 [ 9 ] {a6342c0c-fa3b-404e-b075-894c8fcd4be4} <Debug> executeQuery: (from 127.0.0.1:43818) create table test Engine = DeltaLake('https://cs-tmp.s3.eu-west-1.amazonaws.com/callouis/delta/') (stage: Complete)

Timeout exceeded while receiving data from server. Waited for 300 seconds, timeout is 300 seconds.
Cancelling query.

SELECT
    count(),
    arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n    ', addressToLine(x)), trace), '\n') AS sym
FROM system.trace_log
WHERE (query_id = 'a6342c0c-fa3b-404e-b075-894c8fcd4be4') AND (event_date = today())
GROUP BY trace
ORDER BY count() DESC
LIMIT 5
SETTINGS allow_introspection_functions = 1

Query id: 946ed89f-a7a3-48ef-97a2-bbd72be18faf

┌─count()─┬─sym────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│     146 │ __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse
bssl::ssl_run_handshake(bssl::SSL_HANDSHAKE*, bool*)
    /usr/bin/clickhouse
SSL_do_handshake
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connectSSL(bool)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│     116 │ poll
    /lib/libc.so.6
Poco::Net::SocketImpl::pollImpl(Poco::Timespan&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::poll(Poco::Timespan const&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│     102 │ __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
SSL_read
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::receiveBytes(void*, int, int)
    /usr/bin/clickhouse
Poco::Net::HTTPSession::refill()
    /usr/bin/clickhouse
Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
std::__1::basic_streambuf<char, std::__1::char_traits<char>>::uflow()
    /usr/bin/clickhouse
std::__1::basic_istream<char, std::__1::char_traits<char>>::get()
    /usr/bin/clickhouse
Poco::Net::HTTPResponse::read(std::__1::basic_istream<char, std::__1::char_traits<char>>&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::receiveResponse(Poco::Net::HTTPResponse&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│      24 │ MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
│      24 │ MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6 │
└─────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

5 rows in set. Elapsed: 0.044 sec. Processed 4.66 thousand rows, 1.48 MB (105.07 thousand rows/s., 33.28 MB/s.)

clickhouse1 :) SELECT
    count(),
    arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n    ', addressToLine(x)), trace), '\n') AS sym
FROM system.trace_log
WHERE (query_id = 'a6342c0c-fa3b-404e-b075-894c8fcd4be4') AND (event_date = today())
GROUP BY trace
ORDER BY count() DESC
LIMIT 5 SETTINGS allow_introspection_functions = true Format Vertical

SELECT
    count(),
    arrayStringConcat(arrayMap(x -> concat(demangle(addressToSymbol(x)), '\n    ', addressToLine(x)), trace), '\n') AS sym
FROM system.trace_log
WHERE (query_id = 'a6342c0c-fa3b-404e-b075-894c8fcd4be4') AND (event_date = today())
GROUP BY trace
ORDER BY count() DESC
LIMIT 5
SETTINGS allow_introspection_functions = 1
FORMAT Vertical

Query id: e7682830-ccf7-480c-a245-7d019c032c14

Row 1:
──────
count(): 146
sym:     __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse
bssl::ssl_run_handshake(bssl::SSL_HANDSHAKE*, bool*)
    /usr/bin/clickhouse
SSL_do_handshake
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connectSSL(bool)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 2:
──────
count(): 116
sym:     poll
    /lib/libc.so.6
Poco::Net::SocketImpl::pollImpl(Poco::Timespan&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::poll(Poco::Timespan const&, int)
    /usr/bin/clickhouse
Poco::Net::SocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool)
    /usr/bin/clickhouse
Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&)
    /usr/bin/clickhouse
Poco::Net::HTTPSClientSession::connect(Poco::Net::SocketAddress const&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::reconnect()
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::sendRequest(Poco::Net::HTTPRequest&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 3:
──────
count(): 102
sym:     __read
    /lib/libpthread.so.0

    /usr/bin/clickhouse
bssl::ssl_read_buffer_extend_to(ssl_st*, unsigned long)
    /usr/bin/clickhouse
bssl::ssl_handle_open_record(ssl_st*, bool*, bssl::ssl_open_record_t, unsigned long, unsigned char)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
SSL_read
    /usr/bin/clickhouse
Poco::Net::SecureSocketImpl::receiveBytes(void*, int, int)
    /usr/bin/clickhouse
Poco::Net::HTTPSession::refill()
    /usr/bin/clickhouse
Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
std::__1::basic_streambuf<char, std::__1::char_traits<char>>::uflow()
    /usr/bin/clickhouse
std::__1::basic_istream<char, std::__1::char_traits<char>>::get()
    /usr/bin/clickhouse
Poco::Net::HTTPResponse::read(std::__1::basic_istream<char, std::__1::char_traits<char>>&)
    /usr/bin/clickhouse
Poco::Net::HTTPClientSession::receiveResponse(Poco::Net::HTTPResponse&)
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const
    /usr/bin/clickhouse
Aws::S3::S3Client::HeadObject(Aws::S3::Model::HeadObjectRequest const&) const
    /usr/bin/clickhouse
DB::S3::Client::HeadObject(DB::S3::ExtendedRequest<Aws::S3::Model::HeadObjectRequest> const&) const
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::S3::getObjectInfo(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, bool, bool, bool)
    /usr/bin/clickhouse
DB::StorageS3Source::KeysIterator::KeysIterator(DB::S3::Client const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, DB::S3Settings::RequestSettings const&, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::shared_ptr<DB::Context const>, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::createFileIterator(DB::StorageS3::Configuration const&, bool, std::__1::shared_ptr<DB::Context const>, std::__1::shared_ptr<DB::IAST>, DB::Block const&, std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>>*)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 4:
──────
count(): 24
sym:     MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Row 5:
──────
count(): 24
sym:     MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
MemoryTracker::allocImpl(long, bool, MemoryTracker*)
    /usr/bin/clickhouse
operator new(unsigned long)
    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::SchemaCache::addUnlocked(DB::SchemaCache::Key const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::SchemaCache::addMany(std::__1::vector<DB::SchemaCache::Key, std::__1::allocator<DB::SchemaCache::Key>> const&, DB::ColumnsDescription const&)
    /usr/bin/clickhouse
DB::StorageS3::addColumnsToCache(std::__1::vector<DB::StorageS3Source::KeyWithInfo, std::__1::allocator<DB::StorageS3Source::KeyWithInfo>> const&, DB::StorageS3::Configuration const&, DB::ColumnsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const> const&)
    /usr/bin/clickhouse
DB::StorageS3::getTableStructureFromDataImpl(DB::StorageS3::Configuration const&, std::__1::optional<DB::FormatSettings> const&, std::__1::shared_ptr<DB::Context const>)
    /usr/bin/clickhouse
DB::StorageS3::StorageS3(DB::StorageS3::Configuration const&, std::__1::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::optional<DB::FormatSettings>, bool, std::__1::shared_ptr<DB::IAST>)
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::StorageFactory::get(DB::ASTCreateQuery const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, std::__1::shared_ptr<DB::Context>, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, bool) const
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::__1::unique_ptr<DB::DDLGuard, std::__1::default_delete<DB::DDLGuard>>&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&)
    /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute()
    /usr/bin/clickhouse

    /usr/bin/clickhouse
DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum)
    /usr/bin/clickhouse
DB::TCPHandler::runImpl()
    /usr/bin/clickhouse
DB::TCPHandler::run()
    /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start()
    /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run()
    /usr/bin/clickhouse
Poco::PooledThread::run()
    /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*)
    /usr/bin/clickhouse
start_thread
    /lib/libpthread.so.0

    /lib/libc.so.6

Not sure to understand what happens when creating the table, maybe infering the schema from all the data files take to much time ? I now that for Apache Iceberg, we do a head-object request to every data file, maybe it's the same for Delta Lake ?

alifirat commented 1 year ago

One line in the error stack trace may me think that we have the same issues between Iceberg and Delta Lake. I see the function getTableStructureFromDataImpl that calls the keys iterator, I guess it's done for the schema inference. While the issue on Iceberg is different, I still think when there is a plenty of data files, it takes too much time so solving this schema inference must solve the issues for both Iceberg / Delta Lake. I don't know much for Apache Hudi.