hashbash
commented
5 months ago The same problem when sending message to Kafka:
2024.06.09 17:54:55.548762 [ 6973 ] {83df6acf-aaa2-4f46-b3f8-f85cd76d0344} <Error> DynamicQueryHandler: std::exception. Code: 1001, type: cppkafka::HandleException, e.what() = Broker: Message size too large, Stack trace (when copying this message, always include the lines below):
0. cppkafka::Producer::produce(cppkafka::MessageBuilder const&) @ 0x00000000159f3ef9 in /usr/bin/clickhouse
1. DB::KafkaProducer::produce(String const&, unsigned long, std::vector<COW<DB::IColumn>::immutable_ptr<DB::IColumn>, std::allocator<COW<DB::IColumn>::immutable_ptr<DB::IColumn>>> const&, unsigned long) @ 0x0000000010e0ebeb in /usr/bin/clickhouse
2. DB::MessageQueueSink::consume(DB::Chunk) @ 0x00000000127bc7b8 in /usr/bin/clickhouse
3. DB::SinkToStorage::onConsume(DB::Chunk) @ 0x000000001369d45a in /usr/bin/clickhouse
4. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::ExceptionKeepingTransform::work()::$_1, void ()>>(std::__function::__policy_storage const*) @ 0x00000000135d4e2e in /usr/bin/clickhouse
5. DB::runStep(std::function<void ()>, DB::ThreadStatus*, std::atomic<unsigned long>*) @ 0x00000000135d4b5c in /usr/bin/clickhouse
6. DB::ExceptionKeepingTransform::work() @ 0x00000000135d4233 in /usr/bin/clickhouse
7. DB::ExecutionThreadContext::executeTask() @ 0x00000000133818ba in /usr/bin/clickhouse
8. DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x0000000013378370 in /usr/bin/clickhouse
9. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<DB::PipelineExecutor::spawnThreads()::$_0, void ()>>(std::__function::__policy_storage const*) @ 0x000000001337948f in /usr/bin/clickhouse
10. ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::worker(std::__list_iterator<ThreadFromGlobalPoolImpl<false>, void*>) @ 0x000000000c828e7f in /usr/bin/clickhouse
11. void std::__function::__policy_invoker<void ()>::__call_impl<std::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<false>::ThreadFromGlobalPoolImpl<void ThreadPoolImpl<ThreadFromGlobalPoolImpl<false>>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>(void&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x000000000c82c99c in /usr/bin/clickhouse
12. void* std::__thread_proxy[abi:v15000]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void ThreadPoolImpl<std::thread>::scheduleImpl<void>(std::function<void ()>, Priority, std::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x000000000c82b1c7 in /usr/bin/clickhouse
13. start_thread @ 0x0000000000007ea7 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
14. ? @ 0x00000000000fba2f in /usr/lib/x86_64-linux-gnu/libc-2.31.so
(version 23.9.2.56 (official build))
anchitj
Description
I got the error, while consuming messages (not large) from compacted topics:
There are some relevant issues already. This one describes my problem exactly:
But it fixed a long time ago and I should no longer get the error for the same reason.
And this one seems a bit relevant:
But I don't use transactions. That's why I'm not sure if the 1.9.0 release will fix the bug
What's the reason of this error in my case?
How to reproduce
I don't know how to reproduce that. It seems that it should reproduses according to the instruction from that issue:
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
group.id=Khajiit.GlobalCatalog.KpcCategoryCalculator.V1,client.id=Khajiit.GlobalCatalog.KpcCategoryCalculator.V1_***,enable.auto.commit=False,auto.commit.interval.ms=0,auto.offset.reset=earliest,fetch.max.bytes=2000000000,message.max.bytes=1000000000,receive.message.max.bytes=2000000512,debug=allCentos 7 x64(broker),Microsoft Windows 10.0.17763 x64(consumer app)debug=..as necessary) from librdkafka: excerpts (limited by GitHub attachments size), full logs (uploaded to drive.google.com)