Clickhouse DB::Exception: Not found column process_command_line in block

[patrijua]

Bug description

Clickhouse is consuming a lot of CPU constantly, basically taking all CPU available on the node. In clickhouse logs the following keeps repeating:

2023.05.25 06:42:05.424741 [ 16 ] {af7401d3-045e-4b43-8bab-ccced506a55f::20230520_1222264_1222267_1_1288240} <Error> MutatePlainMergeTreeTask: Code: 10. DB::Exception: Not found column process_command_line in block. There are only columns: timestamp, id, severity_text, severity_number, body, exception_message, exception_stacktrace, service_name, exception_type, k8s_container_name, k8s_namespace_name, time, observed_timestamp, trace_id, span_id, trace_flags, resources_string_key, resources_string_value, attributes_string_key, attributes_string_value, attributes_int64_key, attributes_int64_value, attributes_float64_key, attributes_float64_value, host_name, k8s_pod_name. (NOT_FOUND_COLUMN_IN_BLOCK) (version 22.8.8.3 (official build))
2023.05.25 06:42:05.427550 [ 16 ] {af7401d3-045e-4b43-8bab-ccced506a55f::20230520_1222264_1222267_1_1288240} <Error> virtual bool DB::MutatePlainMergeTreeTask::executeStep(): Code: 10. DB::Exception: Not found column process_command_line in block. There are only columns: timestamp, id, severity_text, severity_number, body, exception_message, exception_stacktrace, service_name, exception_type, k8s_container_name, k8s_namespace_name, time, observed_timestamp, trace_id, span_id, trace_flags, resources_string_key, resources_string_value, attributes_string_key, attributes_string_value, attributes_int64_key, attributes_int64_value, attributes_float64_key, attributes_float64_value, host_name, k8s_pod_name. (NOT_FOUND_COLUMN_IN_BLOCK), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xa3ef75a in /usr/bin/clickhouse
1. DB::Block::getByName(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool) const @ 0x13ef0872 in /usr/bin/clickhouse
2. DB::getBlockAndPermute(DB::Block const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, DB::PODArray<unsigned long, 4096ul, Allocator<false, false>, 15ul, 16ul> const*) @ 0x158db96f in /usr/bin/clickhouse
3. DB::MergeTreeDataPartWriterCompact::writeDataBlockPrimaryIndexAndSkipIndices(DB::Block const&, std::__1::vector<DB::Granule, std::__1::allocator<DB::Granule> > const&) @ 0x158d682e in /usr/bin/clickhouse
4. DB::MergeTreeDataPartWriterCompact::fillDataChecksums(DB::MergeTreeDataPartChecksums&) @ 0x158d7bc2 in /usr/bin/clickhouse
5. DB::MergeTreeDataPartWriterCompact::fillChecksums(DB::MergeTreeDataPartChecksums&) @ 0x158d847c in /usr/bin/clickhouse
6. DB::MergedBlockOutputStream::finalizePartAsync(std::__1::shared_ptr<DB::IMergeTreeDataPart>&, bool, DB::NamesAndTypesList const*, DB::MergeTreeDataPartChecksums*) @ 0x159c9396 in /usr/bin/clickhouse
7. DB::MutateAllPartColumnsTask::finalize() @ 0x159ee9c5 in /usr/bin/clickhouse
8. ? @ 0x159ecfec in /usr/bin/clickhouse
9. DB::MutatePlainMergeTreeTask::executeStep() @ 0x159d562e in /usr/bin/clickhouse
10. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::routine(std::__1::shared_ptr<DB::TaskRuntimeData>) @ 0xa3b9f1b in /usr/bin/clickhouse
11. DB::MergeTreeBackgroundExecutor<DB::MergeMutateRuntimeQueue>::threadFunction() @ 0xa3b9950 in /usr/bin/clickhouse
12. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xa4b38a6 in /usr/bin/clickhouse
13. void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) @ 0xa4b51f7 in /usr/bin/clickhouse
14. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xa4b11c8 in /usr/bin/clickhouse
15. ? @ 0xa4b43dd in /usr/bin/clickhouse
16. ? @ 0x7fe4fa063609 in ?
17. __clone @ 0x7fe4f9f88133 in ?
 (version 22.8.8.3 (official build))

Version information

• Signoz version: v0.18.3

[nityanandagohain]

This is an issue with clickhouse https://github.com/ClickHouse/ClickHouse/pull/50104

we will be testing the custom build for the PR.

[makeavish]

Is it fixed in latest version @nityanandagohain ?