search

pingcap / tiflash

The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial
https://docs.pingcap.com/tidb/stable/tiflash-overview
Apache License 2.0
941 stars 410 forks source link

decodeTiDBRowV2Datum Invalid float value length 1 #6354

Closed lilinghai closed 1 month ago

lilinghai commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

tiflash crash for the error

[2022/11/24 08:12:02.978 +08:00] [ERROR] [Exception.cpp:89] ["Code: 49, e.displayText() = DB::Exception: Invalid float value length 1, e.what() = DB::Exception, Stack trace:
       0x160d1de    DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+23122398]
                    dbms/src/Common/Exception.h:46
       0x188d794    DB::ColumnVector<float>::decodeTiDBRowV2Datum(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, bool) [tiflash+25745300]
       0x5bef2f7    DB::ColumnNullable::decodeTiDBRowV2Datum(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, bool) [tiflash+96400119]
                    dbms/src/Columns/ColumnNullable.cpp:172
       0x5f58b1c    bool DB::appendRowV2ToBlockImpl<false>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, DB::Block&, unsigned long, std::__1::vector<TiDB::ColumnInfo, std::__1::allocator<TiDB::ColumnInfo> > const&, long, bool, bool) [tiflash+99978012]
                    dbms/src/Storages/Transaction/RowCodec.cpp:499
       0x5f57324    DB::appendRowToBlock(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, DB::Block&, unsigned long, std::__1::shared_ptr<DB::DecodingStorageSchemaSnapshot const> const&, bool) [tiflash+99971876]
                    dbms/src/Storages/Transaction/RowCodec.cpp:349
       0x5f24b23    bool DB::RegionBlockReader::readImpl<(DB::TMTPKType)0>(DB::Block&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > > const&, bool) [tiflash+99765027]
                    dbms/src/Storages/Transaction/RegionBlockReader.cpp:146
       0x5f008eb    DB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&)::$_0::operator()(bool) const [tiflash+99617003]
                    dbms/src/Storages/Transaction/PartitionStreams.cpp:130
       0x5efba56    DB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+99596886]
                    dbms/src/Storages/Transaction/PartitionStreams.cpp:181
       0x5efb3c9    DB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&, bool) [tiflash+99595209]
                    dbms/src/Storages/Transaction/PartitionStreams.cpp:359
       0x5f1fe2d    DB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&) [tiflash+99745325]
                    dbms/src/Storages/Transaction/Region.cpp:723
       0x5ee566e    DB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+99505774]
                    dbms/src/Storages/Transaction/KVStore.cpp:293
       0x5f04bb5    HandleWriteRaftCmd [tiflash+99634101]
                    dbms/src/Storages/Transaction/ProxyFFI.cpp:95
  0x7f1591e2cf2d    _$LT$engine_store_ffi..observer..TiFlashObserver$u20$as$u20$raftstore..coprocessor..QueryObserver$GT$::post_exec_query::h1a8a961972b8ccf8 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+17366829]
  0x7f1592cfa00d    raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::apply_raft_cmd::h0b08541bf9853547 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+32886797]
  0x7f1592d11545    raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::ha4bde57f9f25a6ab [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+32982341]
  0x7f1592d16a14    raftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h88c3ac142c377f65 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+33004052]
  0x7f1592cec00c    raftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::hf865ae2f9856a0b6 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+32829452]
  0x7f1592cef628    raftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h91ceadd162e4f65b [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+32843304]
  0x7f15923c6f5c    _$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::hed686c0d358455a2 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+23240540]
  0x7f1592366723    batch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h2c84599628ed3b21 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+22845219]
  0x7f1592414ad2    std::sys_common::backtrace::__rust_begin_short_backtrace::h385c7c98d4548244 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+23558866]
  0x7f1592457b4e    core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h626760f3e1255bb8 [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+23833422]
  0x7f159339fc75    std::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [[libtiflash_proxy.so](http://libtiflash_proxy.so/)+39857269]
                    /rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/[thread.rs:108](http://thread.rs:108/)
  0x7f1590b88ea5    start_thread [libpthread.so.0+32421]
  0x7f158ff8d96d    __clone [libc.so.6+1042797]"] [source="DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap *, DB::WriteCmdsView, DB::RaftCmdHeader)"] [thread_id=115]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

master

CalvinNeo commented 1 year ago

| component | bad | good | | tidb | cf49466a1e03a3929f6587b33df8e3d8fe4a4c6a | 028c5eb8359d1f756b99e40f33c8f3f998f7769e | | tiflash | e7be8e7f06af59a0359287b9849bdd436666e3b4 | 7203151a59ac7c44c673fc5f689fef312c660731 |

CalvinNeo commented 1 year ago

I changed this to moderate, since we can't reproduce for now, the test works just ok.

lilinghai commented 1 year ago

another error info

[2022/12/05 14:24:27.406 +08:00] [ERROR] [Exception.cpp:89] ["Code: 69, e.displayText() = DB::Exception: Wrong precision:0, e.what() = DB::Exception, Stack trace:
       0x172406e\tDB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) [tiflash+24264814]
                \tdbms/src/Common/Exception.h:46
       0x18be19a\tDB::createDecimal(unsigned long, unsigned long) [tiflash+25944474]
                \tdbms/src/DataTypes/DataTypeDecimal.h:196
       0x169ad43\tDB::ColumnDecimal<DB::Decimal<int> >::decodeTiDBRowV2Datum(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, bool) [tiflash+23702851]
                \tdbms/src/Columns/ColumnDecimal.cpp:258
       0x6880a37\tDB::ColumnNullable::decodeTiDBRowV2Datum(unsigned long, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, unsigned long, bool) [tiflash+109578807]
                \tdbms/src/Columns/ColumnNullable.cpp:172
       0x6bf64fc\tbool DB::appendRowV2ToBlockImpl<false>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, DB::Block&, unsigned long, std::__1::vector<TiDB::ColumnInfo, std::__1::allocator<TiDB::ColumnInfo> > const&, long, bool, bool) [tiflash+113206524]
                \tdbms/src/Storages/Transaction/RowCodec.cpp:499
       0x6bf4d04\tDB::appendRowToBlock(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, std::__1::__map_const_iterator<std::__1::__tree_const_iterator<std::__1::__value_type<long, unsigned long>, std::__1::__tree_node<std::__1::__value_type<long, unsigned long>, void*>*, long> >, DB::Block&, unsigned long, std::__1::shared_ptr<DB::DecodingStorageSchemaSnapshot const> const&, bool) [tiflash+113200388]
                \tdbms/src/Storages/Transaction/RowCodec.cpp:349
       0x6bc2503\tbool DB::RegionBlockReader::readImpl<(DB::TMTPKType)0>(DB::Block&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > > const&, bool) [tiflash+112993539]
                \tdbms/src/Storages/Transaction/RegionBlockReader.cpp:146
       0x6b9e2cb\tDB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&)::$_0::operator()(bool) const [tiflash+112845515]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:130
       0x6b99436\tDB::writeRegionDataToStorage(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&) [tiflash+112825398]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:181
       0x6b98da9\tDB::RegionTable::writeBlockByRegion(DB::Context&, DB::RegionPtrWithBlock const&, std::__1::vector<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> >, std::__1::allocator<std::__1::tuple<DB::RawTiDBPK, unsigned char, unsigned long, std::__1::shared_ptr<DB::StringObject<false> const> > > >&, std::__1::shared_ptr<DB::Logger> const&, bool) [tiflash+112823721]
                \tdbms/src/Storages/Transaction/PartitionStreams.cpp:359
       0x6bbd80d\tDB::Region::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, DB::TMTContext&) [tiflash+112973837]
                \tdbms/src/Storages/Transaction/Region.cpp:723
       0x6b82ece\tDB::KVStore::handleWriteRaftCmd(DB::WriteCmdsView const&, unsigned long, unsigned long, unsigned long, DB::TMTContext&) [tiflash+112733902]
                \tdbms/src/Storages/Transaction/KVStore.cpp:293
       0x6ba2595\tHandleWriteRaftCmd [tiflash+112862613]
                \tdbms/src/Storages/Transaction/ProxyFFI.cpp:95
  0x7f4844e2620d\t_$LT$engine_store_ffi..observer..TiFlashObserver$u20$as$u20$raftstore..coprocessor..QueryObserver$GT$::post_exec_query::h1a8a961972b8ccf8 [libtiflash_proxy.so+17367565]
  0x7f4845cf796d\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::apply_raft_cmd::h0b08541bf9853547 [libtiflash_proxy.so+32905581]
  0x7f4845d0eea5\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::process_raft_cmd::ha4bde57f9f25a6ab [libtiflash_proxy.so+33001125]
  0x7f4845d14374\traftstore::store::fsm::apply::ApplyDelegate$LT$EK$GT$::handle_raft_committed_entries::h88c3ac142c377f65 [libtiflash_proxy.so+33022836]
  0x7f4845ce996c\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_apply::hf865ae2f9856a0b6 [libtiflash_proxy.so+32848236]
  0x7f4845cecf88\traftstore::store::fsm::apply::ApplyFsm$LT$EK$GT$::handle_tasks::h91ceadd162e4f65b [libtiflash_proxy.so+32862088]
  0x7f48453c023c\t_$LT$raftstore..store..fsm..apply..ApplyPoller$LT$EK$GT$$u20$as$u20$batch_system..batch..PollHandler$LT$raftstore..store..fsm..apply..ApplyFsm$LT$EK$GT$$C$raftstore..store..fsm..apply..ControlFsm$GT$$GT$::handle_normal::hed686c0d358455a2 [libtiflash_proxy.so+23241276]
  0x7f484535fde3\tbatch_system::batch::Poller$LT$N$C$C$C$Handler$GT$::poll::h2c84599628ed3b21 [libtiflash_proxy.so+22846947]
  0x7f484540ddb2\tstd::sys_common::backtrace::__rust_begin_short_backtrace::h385c7c98d4548244 [libtiflash_proxy.so+23559602]
  0x7f4845450e5e\tcore::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h626760f3e1255bb8 [libtiflash_proxy.so+23834206]
  0x7f4846399375\tstd::sys::unix::thread::Thread::new::thread_start::hd2791a9cabec1fda [libtiflash_proxy.so+39859061]
                \t/rustc/96ddd32c4bfb1d78f0cd03eb068b1710a8cebeef/library/std/src/sys/unix/thread.rs:108
  0x7f4843b81ea5\tstart_thread [libpthread.so.0+32421]
  0x7f4842f8696d\t__clone [libc.so.6+1042797]"] [source="DB::EngineStoreApplyRes DB::HandleWriteRaftCmd(const DB::EngineStoreServerWrap *, DB::WriteCmdsView, DB::RaftCmdHeader)"] [thread_id=115]
[2022/12/05 14:24:27.407 +08:00] [DEBUG] [SegmentReader.cpp:45] ["Stop begin"] [source=SegmentReader] [thread_id=115]
JaySon-Huang commented 4 months ago

Note: it could be a duplicate issue to https://github.com/pingcap/tiflash/issues/7024. The root cause should be some invalid rows is generated by tidb. Investigating.

JaySon-Huang commented 1 month ago

This issue is a bug of tidb https://github.com/pingcap/tidb/issues/53634. Have been fixed in the tidb-server side.