search

pingcap / tiflash

The analytical engine for TiDB and TiDB Cloud. Try free: https://tidbcloud.com/free-trial
https://docs.pingcap.com/tidb/stable/tiflash-overview
Apache License 2.0
937 stars 409 forks source link

Query core with `rpad` #6475

Open guo-shaoge opened 1 year ago

guo-shaoge commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. setup cluster with tiflash(In my case, 4 tiflash are setup)
  2. run mysql_test ./mysql-tester -host 127.0.0.1 -port 4000 -user root -params '&allowNativePasswords=true&allowAllFiles=true' -record non_disaggregated_2 non_disaggregated_2.txt
  3. query core

    2. What did you expect to see? (Required)

    succeed

    3. What did you see instead (Required)

    core stack(fine_grained shuffle disabled):

    [2022/12/09 16:02:49.396 +08:00] [ERROR] [BaseDaemon.cpp:570] ["
   0x61d8c01\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+102599681]
            \tlibs/libdaemon/src/BaseDaemon.cpp:221
0x7f6ba8431a00\t<unknown symbol> [libc.so.6+231936]
   0x71afd8a\tmemcpy [tiflash+119209354]
            \tlibs/libmemcpy/memcpy.cpp:23
   0x173c86b\tDB::WriteBuffer::write(char const*, unsigned long) [tiflash+24365163]
            \tdbms/src/IO/WriteBuffer.h:93
   0x625ae73\tDB::DataTypeString::serializeBinaryBulk(DB::IColumn const&, DB::WriteBuffer&, unsigned long, unsigned long) const [tiflash+103132787]
            \tdbms/src/DataTypes/DataTypeString.cpp:115
   0x6256b9a\tDB::DataTypeNullable::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, std::__1::function<DB::WriteBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&,         unsigned long, unsigned long, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const [tiflash+103115674]
            \tdbms/src/DataTypes/DataTypeNullable.cpp:80
   0x6db32f9\tDB::CHBlockChunkCodecStream::encode(DB::Block const&, unsigned long, unsigned long) [tiflash+115028729]
            \tdbms/src/Flash/Coprocessor/CHBlockChunkCodec.cpp:137
   0x16f1aee\tDB::HashPartitionWriter<std::__1::shared_ptr<DB::MPPTunnelSet> >::partitionAndEncodeThenWriteBlocks() [tiflash+24058606]
            \tdbms/src/Flash/Mpp/HashPartitionWriter.cpp:93
   0x6e67312\tDB::ExchangeSenderBlockInputStream::readImpl() [tiflash+115766034]
            \tdbms/src/DataStreams/ExchangeSenderBlockInputStream.cpp:43
   0x6203175\tDB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+102773109]
            \tdbms/src/DataStreams/IProfilingBlockInputStream.cpp:75
   0x6202e65\tDB::IProfilingBlockInputStream::read() [tiflash+102772325]
            \tdbms/src/DataStreams/IProfilingBlockInputStream.cpp:43
   0x6ed8078\tDB::DataStreamExecutor::execute(DB::ResultHandler) [tiflash+116228216]
            \tdbms/src/Flash/Executor/DataStreamExecutor.cpp:28
   0x6eae071\tDB::MPPTask::runImpl() [tiflash+116056177]
            \tdbms/src/Flash/Mpp/MPPTask.cpp:381
   0x18007c8\tauto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+25167816]
            \tdbms/src/Common/wrapInvocable.h:36
   0x1803fc3\tDB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+25182147]
            \tdbms/src/Common/DynamicThreadPool.cpp:101
   0x1803620\tDB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+25179680]
            \tdbms/src/Common/DynamicThreadPool.cpp:115
   0x1804712\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*        )(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::        'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+25184018]
            \t/usr/local/bin/../include/c++/v1/thread:291
0x7f6ba847f8fd\t<unknown symbol> [libc.so.6+551165]"] [source=BaseDaemon] [thread_id=652]

core stack(fine_grained shuffle enable)

[2022/12/09 15:52:27.445 +08:00] [ERROR] [BaseDaemon.cpp:570] ["                                                                                                                                                                                                   0x61d8c01\tfaultSignalHandler(int, siginfo_t*, void*) [tiflash+102599681]
                \tlibs/libdaemon/src/BaseDaemon.cpp:221
  0x7ff570973a00\t<unknown symbol> [libc.so.6+231936]
       0x71afd80\tmemcpy [tiflash+119209344]
                \tlibs/libmemcpy/memcpy.cpp:23
       0x173c86b\tDB::WriteBuffer::write(char const*, unsigned long) [tiflash+24365163]
                \tdbms/src/IO/WriteBuffer.h:93
       0x625ae73\tDB::DataTypeString::serializeBinaryBulk(DB::IColumn const&, DB::WriteBuffer&, unsigned long, unsigned long) const [tiflash+103132787]
                \tdbms/src/DataTypes/DataTypeString.cpp:115
       0x6256b9a\tDB::DataTypeNullable::serializeBinaryBulkWithMultipleStreams(DB::IColumn const&, std::__1::function<DB::WriteBuffer* (std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> > const&)> const&,         unsigned long, unsigned long, bool, std::__1::vector<DB::IDataType::Substream, std::__1::allocator<DB::IDataType::Substream> >&) const [tiflash+103115674]
                \tdbms/src/DataTypes/DataTypeNullable.cpp:80
       0x6db32f9\tDB::CHBlockChunkCodecStream::encode(DB::Block const&, unsigned long, unsigned long) [tiflash+115028729]
                \tdbms/src/Flash/Coprocessor/CHBlockChunkCodec.cpp:137
       0x16f0802\tDB::FineGrainedShuffleWriter<std::__1::shared_ptr<DB::MPPTunnelSet> >::batchWriteFineGrainedShuffle() [tiflash+24053762]
                \tdbms/src/Flash/Mpp/FineGrainedShuffleWriter.cpp:140
       0x6e672f3\tDB::ExchangeSenderBlockInputStream::readImpl() [tiflash+115766003]
                \tdbms/src/DataStreams/ExchangeSenderBlockInputStream.cpp:47
       0x6203175\tDB::IProfilingBlockInputStream::read(DB::PODArray<unsigned char, 4096ul, Allocator<false>, 15ul, 16ul>*&, bool) [tiflash+102773109]
                \tdbms/src/DataStreams/IProfilingBlockInputStream.cpp:75
       0x6202e65\tDB::IProfilingBlockInputStream::read() [tiflash+102772325]
                \tdbms/src/DataStreams/IProfilingBlockInputStream.cpp:43
       0x6ed8078\tDB::DataStreamExecutor::execute(DB::ResultHandler) [tiflash+116228216]
                \tdbms/src/Flash/Executor/DataStreamExecutor.cpp:28
       0x6eae071\tDB::MPPTask::runImpl() [tiflash+116056177]
                \tdbms/src/Flash/Mpp/MPPTask.cpp:381
       0x18007c8\tauto DB::wrapInvocable<std::__1::function<void ()> >(bool, std::__1::function<void ()>&&)::'lambda'()::operator()() [tiflash+25167816]
                \tdbms/src/Common/wrapInvocable.h:36
       0x1803fc3\tDB::DynamicThreadPool::executeTask(std::__1::unique_ptr<DB::IExecutableTask, std::__1::default_delete<DB::IExecutableTask> >&) [tiflash+25182147]
                \tdbms/src/Common/DynamicThreadPool.cpp:101
       0x1803620\tDB::DynamicThreadPool::fixedWork(unsigned long) [tiflash+25179680]
                \tdbms/src/Common/DynamicThreadPool.cpp:115
       0x1804712\tvoid* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, std::__1::thread DB::ThreadFactory::newThread<void (DB::DynamicThreadPool::*        )(unsigned long), DB::DynamicThreadPool*, unsigned long&>(bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, void (DB::DynamicThreadPool::*&&)(unsigned long), DB::DynamicThreadPool*&&, unsigned long&)::        'lambda'(auto&&...), DB::DynamicThreadPool*, unsigned long> >(void*) [tiflash+25184018]
                \t/usr/local/bin/../include/c++/v1/thread:291
  0x7ff5709c18fd\t<unknown symbol> [libc.so.6+551165]"] [source=BaseDaemon] [thread_id=760]

4. What is your TiFlash version? (Required)

v6.5.0-alpha

MySQL [test]> select * from information_schema.cluster_info;
+---------+-----------------+-----------------+-------------+------------------------------------------+---------------------------+------------------+-----------+
| TYPE    | INSTANCE        | STATUS_ADDRESS  | VERSION     | GIT_HASH                                 | START_TIME                | UPTIME           | SERVER_ID |
+---------+-----------------+-----------------+-------------+------------------------------------------+---------------------------+------------------+-----------+
| tidb    | 127.0.0.1:4000  | 127.0.0.1:10080 | 6.4.0-alpha | 631f32d2d563864b56fb6477934354f00aa330ba | 2022-12-09T15:23:03+08:00 | 46m37.595525908s |    615264 |
| pd      | 127.0.0.1:2379  | 127.0.0.1:2379  | 6.5.0-alpha | 8227cdba21205fea57b0aca9644e97c2c53a94a8 | 2022-12-09T15:16:11+08:00 | 53m29.59553174s  |         0 |
| tikv    | 127.0.0.1:20160 | 127.0.0.1:20180 | 6.5.0-alpha | 3e0b8ddc6ab5fdf80afcce5884021c2015204256 | 2022-12-09T15:16:14+08:00 | 53m26.595535992s |         0 |
| tiflash | 127.0.0.1:3932  | 127.0.0.1:20294 | 6.5.0-alpha | f248fac2bffd2416a8c56206ed37783b77d28093 | 2022-12-09T15:52:54+08:00 | 16m46.595538134s |         0 |
| tiflash | 127.0.0.1:3931  | 127.0.0.1:20293 | 6.5.0-alpha | f248fac2bffd2416a8c56206ed37783b77d28093 | 2022-12-09T16:00:08+08:00 | 9m32.595540284s  |         0 |
| tiflash | 127.0.0.1:3933  | 127.0.0.1:20295 | 6.5.0-alpha | f248fac2bffd2416a8c56206ed37783b77d28093 | 2022-12-09T16:03:16+08:00 | 6m24.595542249s  |         0 |
| tiflash | 127.0.0.1:3930  | 127.0.0.1:20292 | 6.5.0-alpha | f248fac2bffd2416a8c56206ed37783b77d28093 | 2022-12-09T15:16:30+08:00 | 53m10.595543953s |         0 |
+---------+-----------------+-----------------+-------------+------------------------------------------+---------------------------+------------------+-----------+
7 rows in set (0.004 sec)
guo-shaoge commented 1 year ago

When rpad is removed, query doesn't core.

SeaRise commented 1 year ago

Is there any relation between crash and window function? Is there a minimal recurrence case?

guo-shaoge commented 1 year ago

Is there any relation between crash and window function? Is there a minimal recurrence case?

Guess it's because rpad didn't limit result length. No related to window function(because When rpad is removed, query doesn't core.).

guo-shaoge commented 1 year ago

Is there any relation between crash and window function? Is there a minimal recurrence case?

Yes, as issue said. I upload the recurrence case. I have environment that can recurrence crash, but no time for now, I can debug this bug later.