search

jaegertracing / jaeger-clickhouse

Jaeger ClickHouse storage plugin implementation
Apache License 2.0
247 stars 51 forks source link

Running with hotrod results in Too many simultaneous queries. Maximum: 100 #2

Closed pavolloffay closed 3 years ago

pavolloffay commented 3 years ago 
2021.07.14 17:06:49.783711 [ 219 ] {11925d3b-7684-4919-827b-319af811c400} <Debug> MemoryTracker: Peak memory usage (for query): 0.00 B.
2021.07.14 17:06:49.783769 [ 1010 ] {d4de6e5e-6305-4802-842e-13c660886ef2} <Error> TCPHandler: Code: 202, e.displayText() = DB::Exception: Too many simultaneous queries. Maximum: 100, Stack trace:

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x8d31b5a in /usr/bin/clickhouse
1. DB::ProcessList::insert(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::IAST const*, std::__1::shared_ptr<DB::Context const>) @ 0xfcd6802 in /usr/bin/clickhouse
2. ? @ 0xfe21ab3 in /usr/bin/clickhouse
3. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, bool) @ 0xfe208e3 in /usr/bin/clickhouse
4. DB::TCPHandler::runImpl() @ 0x1069f6c2 in /usr/bin/clickhouse
5. DB::TCPHandler::run() @ 0x106b25d9 in /usr/bin/clickhouse
6. Poco::Net::TCPServerConnection::start() @ 0x1338b30f in /usr/bin/clickhouse
7. Poco::Net::TCPServerDispatcher::run() @ 0x1338cd9a in /usr/bin/clickhouse
8. Poco::PooledThread::run() @ 0x134bfc19 in /usr/bin/clickhouse
9. Poco::ThreadImpl::runnableEntry(void*) @ 0x134bbeaa in /usr/bin/clickhouse
10. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
11. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so

The DB is started as docker run --rm -it -p9000:9000 --name some-clickhouse-server --ulimit nofile=262144:262144 yandex/clickhouse-server:21

chhetripradeep commented 3 years ago

Thank you for creating this storage plugin.

I think this can happen due to the batch size (default: 10000) We can try increasing it and see if it help.

chhetripradeep commented 3 years ago

Minor Feedback:

After each Jaeger run the ClickHouse server has to be restarted or changed sql_scripts_dir to a directory with no SLQ scripts.

We can add IF NOT EXISTS in the sql files to prevent this.

pavolloffay commented 3 years ago

We can add IF NOT EXISTS in the sql files to prevent this.

Added in https://github.com/pavolloffay/jaeger-clickhouse/commit/7b283586e13bb2d617f4420067db106de8a0c746

pavolloffay commented 3 years ago

I think this can happen due to the batch size (default: 10000) We can try increasing it and see if it help.

Increasing batch size does not help, not even decreasing to e.g. 5.

pavolloffay commented 3 years ago

My bad fixed in https://github.com/pavolloffay/jaeger-clickhouse/commit/99ca6441590c37c75653d9039607db3054d0371a