pingcap / tiflash

The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial
https://docs.pingcap.com/tidb/stable/tiflash-overview
Apache License 2.0
946 stars 409 forks source link

TiFlash crash when using ZSTD and the compression level is -1 #9322

Closed Lloyd-Pottiger closed 2 months ago

Lloyd-Pottiger commented 2 months ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[profiles]
[profiles.default]
## Compression algorithm of DeltaTree Storage Engine
dt_compression_method = zstd

## Compression level of DeltaTree Storage Engine
dt_compression_level = -1

2. What did you expect to see? (Required)

running normally

3. What did you see instead (Required)

[2024/08/13 19:07:23.966 +08:00] [FATAL] [Exception.cpp:106] [\"Code: 49, e.displayText() = DB::Exception: Check codec failed, e.what() = DB::Exception, Stack trace:\\n\\n\\n  0x55f97debf533\\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) [tiflash+33117491]\\n                \\tdbms/src/Common/StackTrace.cpp:23\\n  0x55f9836bd1da\\tDB::CompressionCodecFactory::create(DB::CompressionSettings const&) [tiflash+125383130]\\n                \\tdbms/src/IO/Compression/CompressionCodecFactory.cpp:214\\n  0x55f97dd929b6\\tDB::CompressedWriteBuffer<true>::CompressedWriteBuffer(DB::WriteBuffer&, DB::CompressionSettings, unsigned long) [tiflash+31885750]\\n                \\tdbms/src/IO/Compression/CompressedWriteBuffer.cpp:53\\n  0x55f9834c8afb\\tDB::DM::serializeColumn(DB::WriteBuffer&, DB::IColumn const&, std::__1::shared_ptr<DB::IDataType const> const&, unsigned long, unsigned long, DB::CompressionMethod, long) [tiflash+123333371]\\n                \\tdbms/src/Storages/DeltaMerge/ColumnFile/ColumnFilePersisted.cpp:77\\n  0x55f9834c2803\\tDB::DM::ColumnFileTiny::writeColumnFileData(DB::DM::DMContext const&, DB::Block const&, unsigned long, unsigned long, DB::DM::WriteBatches&) [tiflash+123308035]\\n                \\tdbms/src/Storages/DeltaMerge/ColumnFile/ColumnFileTiny.cpp:310\\n  0x55f9834ef43e\\tDB::DM::ColumnFileFlushTask::prepare(DB::DM::WriteBatches&) [tiflash+123491390]\\n                \\tdbms/src/Storages/DeltaMerge/Delta/ColumnFileFlushTask.cpp:52\\n  0x55f9834de543\\tDB::DM::DeltaValueSpace::flush(DB::DM::DMContext&) [tiflash+123422019]\\n                \\tdbms/src/Storages/DeltaMerge/Delta/DeltaValueSpace.cpp:381\\n  0x55f98334ad0a\\tDB::DM::Segment::flushCache(DB::DM::DMContext&) [tiflash+121769226]\\n                \\tdbms/src/Storages/DeltaMerge/Segment.cpp:2297\\n  0x55f9832ebad9\\tDB::DM::DeltaMergeStore::flushCache(std::__1::shared_ptr<DB::DM::DMContext> const&, DB::DM::RowKeyRange const&, bool) [tiflash+121379545]\\n                \\tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:804\\n  0x55f9832eea5a\\tDB::DM::DeltaMergeStore::flushCache(DB::Context const&, DB::DM::RowKeyRange const&, bool) [tiflash+121391706]\\n                \\tdbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp:777\\n  0x55f984c3c537\\tDB::KVStore::tryFlushRegionCacheInStorage(DB::TMTContext&, DB::Region const&, std::__1::shared_ptr<DB::Logger> const&, bool) [tiflash+147924279]\\n                \\tdbms/src/Storages/StorageDeltaMerge.cpp:977\\n  0x55f984cc59cd\\tDB::KVStore::forceFlushRegionDataImpl(DB::Region&, bool, DB::TMTContext&, DB::RegionTaskLock const&, unsigned long, unsigned long) const [tiflash+148486605]\\n                \\tdbms/src/Storages/KVStore/MultiRaft/Persistence.cpp:255\\n  0x55f984cc4cef\\tDB::KVStore::canFlushRegionDataImpl(std::__1::shared_ptr<DB::Region> const&, unsigned char, bool, DB::TMTContext&, DB::RegionTaskLock const&, unsigned long, unsigned long, unsigned long, unsigned long) [tiflash+148483311]\\n                \\tdbms/src/Storages/KVStore/MultiRaft/Persistence.cpp:230\\n  0x55f984cc529c\\tDB::KVStore::tryFlushRegionData(unsigned long, bool, bool, DB::TMTContext&, unsigned long, unsigned long, unsigned long, unsigned long) [tiflash+148484764]\\n                \\tdbms/src/Storages/KVStore/MultiRaft/Persistence.cpp:123\\n  0x55f984c960e1\\tTryFlushData [tiflash+148291809]\\n                \\tdbms/src/Storages/KVStore/FFI/ProxyFFI.cpp:161\\n  0x7fa85cb165a5\\t_$LT$engine_store_ffi..observer..TiFlashObserver$LT$T$C$ER$GT$$u20$as$u20$raftstore..coprocessor..AdminObserver$GT$::pre_exec_admin::h836587b334032bf7 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+28984741]\\n                \\tcontrib/tiflash-proxy/proxy_components/engine_store_ffi/src/[observer.rs:120](http://observer.rs:120/)\\n  0x7fa85dda097d\\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::apply_raft_cmd::hb8d12067eb11c51c [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+48425341]\\n                \\tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/[apply.rs:1451](http://apply.rs:1451/)\\n  0x7fa85ddb5efe\\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::hf829479500040f95 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+48512766]\\n                \\tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/[apply.rs:1399](http://apply.rs:1399/)\\n  0x7fa85ddb87e9\\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::hee24047b52360b75 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+48523241]\\n                \\tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/[apply.rs:1145](http://apply.rs:1145/)\\n  0x7fa85dd93e3d\\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::hca3398a59a64f6ce [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+48373309]\\n                \\tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/[apply.rs:4068](http://apply.rs:4068/)\\n  0x7fa85dd98101\\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h17829962adcd7513 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+48390401]\\n                \\tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/[apply.rs:4423](http://apply.rs:4423/)\\n  0x7fa85cc8e0f5\\t_$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::hd7705b57f19da499 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+30523637]\\n                \\tcontrib/tiflash-proxy/components/raftstore/src/store/fsm/[apply.rs:4712](http://apply.rs:4712/)\\n  0x7fa85cbb0bd3\\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h24e1fe1cc3f17aa1 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+29617107]\\n                \\tcontrib/tiflash-proxy/components/batch-system/src/[batch.rs:422](http://batch.rs:422/)\\n  0x7fa85ccef25f\\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h0d3b967cdfc91423 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+30921311]\\n                \\t/rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys_common/[backtrace.rs:155](http://backtrace.rs:155/)\\n  0x7fa85cd4d899\\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h544bde9cf8936717 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+31307929]\\n                \\t/rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/core/src/ops/[function.rs:250](http://function.rs:250/)\\n  0x7fa85e6ba3a6\\tstd::sys::unix::thread::Thread::new::thread_start::hd01330a228cdd35d [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+57967526]\\n                \\t/rustc/89e2160c4ca5808657ed55392620ed1dbbce78d1/library/std/src/sys/unix/[thread.rs:108](http://thread.rs:108/)\\n  0x7fa85ac2cc02\\tstart_thread [libc.so.6+564226]\\n  0x7fa85acb1c40\\tclone3 [libc.so.6+1109056]\"] [source=\"uint8_t DB::TryFlushData(EngineStoreServerWrap *, uint64_t, uint8_t, uint64_t, uint64_t, uint64_t, uint64_t)\"] [thread_id=309]

4. What is your TiFlash version? (Required)

master

kennedy8312 commented 2 months ago

/impact crash