search

StarRocks / starrocks

The world's fastest open query engine for sub-second analytics both on and off the data lakehouse. With the flexibility to support nearly any scenario, StarRocks provides best-in-class performance for multi-dimensional analytics, real-time analytics, and ad-hoc queries. A Linux Foundation project.
https://starrocks.io
Apache License 2.0
8.98k stars 1.8k forks source link

query error from HUDI external table #6028

Closed tiannan-sr closed 2 years ago

tiannan-sr commented 2 years ago

Steps to reproduce the behavior (Required)

  1. create hudi external table:

    create external table ex_hudi_tbl8 (
uuid int
,col_boolean boolean
,col_int int
,col_long bigint
,col_float float
,col_double double
,col_decimal decimal(38,18)
,col_date date
,col_string string
,col_binary varchar(200))
ENGINE=hudi
properties (
"resource" = "hudi_emr_tn",
"table" = "hudi_par_parquet_snappy",
"database" = "hudi_db");

  2. select from hudi external table error:

    mysql> select * from ex_hudi_tbl8;
ERROR 1064 (HY000): build chunk meta error
mysql> select * from ex_hudi_tbl8;
+------+-------------+---------+----------+-----------+------------+-----------------------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------+---------------------+----------------------------------+--------------------+------------------------+------------------------------------------------------------------------------+
| uuid | col_boolean | col_int | col_long | col_float | col_double | col_decimal                             | col_date   | col_string                                                                                                                                                                                                                                                                                                                | col_binary             | _hoodie_commit_time | _hoodie_partition_path           | _hoodie_record_key | _hoodie_commit_seqno   | _hoodie_file_name                                                            |
+------+-------------+---------+----------+-----------+------------+-----------------------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------+---------------------+----------------------------------+--------------------+------------------------+------------------------------------------------------------------------------+
|    5 |           1 |       5 |        5 |     5.005 |     5.0005 |          50000000000.000000000500000000 | 2020-05-01 | A total of 4,685 companies have been listed on the A-share market in China as of Dec 31, 2021, with 46 percent of them based in Beijing, Shanghai, Shenzhen, Hangzhou, Suzhou, Guangzhou, Ningbo, Nanjing, Wuxi, and Chengdu, said a report of National Business Daily on Wednesday.                                      | 1110001010101011001001 | 20220328173350281   | col_date=2020-05-01/col_int=5    | uuid:5             | 20220328173350281_0_9  | 96802430-c948-4ae1-8291-1d119ecb237b-0_0-684-648_20220328173350281.parquet   |
|   10 |           0 |      10 |       10 |     10.01 |    10.0001 |         100000000000.000000000100000000 | 2020-10-01 | Volkswagen deliveries hit by chip shortages                                                                                                                                                                                                                                                                               | 1110001010101011001001 | 20220328173451281   | col_date=2020-10-01/col_int=10   | uuid:10            | 20220328173451281_0_21 | 8b000532-aeac-44ec-9eb1-0f8893c3b14f-0_0-979-1206_20220328173451281.parquet  |
|    3 |           1 |       4 |        3 |     3.003 |     3.0003 |          30000000000.000000000300000000 | 2020-03-01 | Nation raises caution on overseas packages                                                                                                                                                                                                                                                                                | 1110001010101011001001 | 20220328173335305   | col_date=2020-03-01/col_int=4    | uuid:3             | 20220328173335305_0_12 | e1396e6d-c381-4840-b63d-b382d537df3c-0_0-597-544_20220328173335305.parquet   |
|    8 |           0 |       9 |        8 |     8.008 |     8.0008 |          80000000000.000000000800000000 | 2020-08-01 | Citi report finds MNC mood in China buoyant                                                                                                                                                                                                                                                                               | 1110001010101011001001 | 20220328173433445   | col_date=2020-08-01/col_int=9    | uuid:8             | 20220328173433445_0_19 | 7ed092b8-cb56-4c8c-8ab3-b98688e42ea1-0_0-892-1009_20220328173433445.parquet  |
|    4 |           0 |       5 |        4 |     4.004 |     4.0004 |          40000000000.000000000400000000 | 2020-04-01 | Shanghai, Shenzhen register most newly listed firms in 2021                                                                                                                                                                                                                                                               | 1110001010101011001001 | 20220328173344917   | col_date=2020-04-01/col_int=5    | uuid:4             | 20220328173344917_0_14 | db63d32b-4d3b-4423-abeb-560ba0ee7a09-0_0-655-612_20220328173344917.parquet   |
|    9 |           1 |       9 |        9 |     9.009 |     9.0009 |          90000000000.000000000900000000 | 2020-09-01 | Xi: China, Russia major champions of multilateralism, global justice                                                                                                                                                                                                                                                      | 1110001010101011001001 | 20220328173438751   | col_date=2020-09-01/col_int=9    | uuid:9             | 20220328173438751_0_20 | 974b6e3d-97e7-45aa-a851-76c41411a75a-0_0-921-1072_20220328173438751.parquet  |
|    4 |           0 |       4 |        4 |     4.004 |     4.0004 |          40000000000.000000000400000000 | 2020-04-01 | Shanghai, Shenzhen register most newly listed firms in 2021                                                                                                                                                                                                                                                               | 1110001010101011001001 | 20220328173339621   | col_date=2020-04-01/col_int=4    | uuid:4             | 20220328173339621_0_13 | 85989879-2aad-41c5-8b32-ae6fbe701bd9-0_0-626-577_20220328173339621.parquet   |
|    9 |           1 |      10 |        9 |     9.009 |     9.0009 |          90000000000.000000000900000000 | 2020-09-01 | Xi: China, Russia major champions of multilateralism, global justice                                                                                                                                                                                                                                                      | 1110001010101011001001 | 20220328173444514   | col_date=2020-09-01/col_int=10   | uuid:9             | 20220328173444514_0_12 | 4c4db434-ba10-4344-9612-ad12288b2aa7-0_0-950-1138_20220328173444514.parquet  |
|    5 |           1 |       6 |        5 |     5.005 |     5.0005 |          50000000000.000000000500000000 | 2020-05-01 | A total of 4,685 companies have been listed on the A-share market in China as of Dec 31, 2021, with 46 percent of them based in Beijing, Shanghai, Shenzhen, Hangzhou, Suzhou, Guangzhou, Ningbo, Nanjing, Wuxi, and Chengdu, said a report of National Business Daily on Wednesday.                                      | 1110001010101011001001 | 20220328173355888   | col_date=2020-05-01/col_int=6    | uuid:5             | 20220328173355888_0_15 | 1835a3f2-ec29-4dfd-9452-0e490cd71032-0_0-713-686_20220328173355888.parquet   |
|   10 |           0 |      11 |       10 |     10.01 |    10.0001 |         100000000000.000000000100000000 | 2020-10-01 | Volkswagen deliveries hit by chip shortages                                                                                                                                                                                                                                                                               | 1110001010101011001001 | 20220328173457892   | col_date=2020-10-01/col_int=11   | uuid:10            | 20220328173457892_0_22 | 6b9bf553-bd02-42f1-b5a4-014b207b587e-0_0-1008-1277_20220328173457892.parquet |
|    3 |           1 |       3 |        3 |     3.003 |     3.0003 |          30000000000.000000000300000000 | 2020-03-01 | Nation raises caution on overseas packages                                                                                                                                                                                                                                                                                | 1110001010101011001001 | 20220328173330230   | col_date=2020-03-01/col_int=3    | uuid:3             | 20220328173330230_0_11 | 99385896-04d6-4825-8fe8-6f11cc565118-0_0-568-512_20220328173330230.parquet   |
|    8 |           0 |       8 |        8 |     8.008 |     8.0008 |          80000000000.000000000800000000 | 2020-08-01 | Citi report finds MNC mood in China buoyant                                                                                                                                                                                                                                                                               | 1110001010101011001001 | 20220328173426990   | col_date=2020-08-01/col_int=8    | uuid:8             | 20220328173426990_0_11 | e999a63c-bf8e-420b-b91e-0c6f5bf5de8e-0_0-863-948_20220328173426990.parquet   |
|   13 |        NULL |    NULL |     NULL |      NULL |       NULL |                                    NULL | NULL       | NULL                                                                                                                                                                                                                                                                                                                      | NULL                   | 20220328173528828   | col_date=default/col_int=default | uuid:13            | 20220328173528828_0_17 | 978dd657-0c3d-47df-a512-136560c0c775-0_0-1158-1673_20220328173528828.parquet |
|   14 |        NULL |    NULL |     NULL |      NULL |       NULL | 12345678901234567890.123456789012345678 | NULL       |                                                                                                                                                                                                                                                                                                                           | NULL                   | 20220328173534438   | col_date=default/col_int=default | uuid:14            | 20220328173534438_0_23 | 978dd657-0c3d-47df-a512-136560c0c775-0_0-1187-1759_20220328173534438.parquet |
|    1 |           1 |       2 |        1 |     1.001 |     1.0001 |          10000000000.000000000100000000 | 2020-01-01 | Top 10 Unsolved Mysteries of Paleontological Dinosaurs, Did You Know?                                                                                                                                                                                                                                                     | 1110001010101011001001 | 20220328173315943   | col_date=2020-01-01/col_int=2    | uuid:1             | 20220328173315943_0_8  | 5736f8e9-c0d9-4d01-8c99-1f72e6dc31ba-0_0-483-428_20220328173315943.parquet   |
|    6 |           0 |       7 |        6 |     6.006 |     6.0006 |          60000000000.000000000600000000 | 2020-06-01 | The development of enterprises, as well as the push of local governments, has boosted the listings, the report said. For example, in early 2018, Central Chinas Hubei province launched a plan to double its listed companies by including the listing of enterprises into the governments annual performance assessment. | 1110001010101011001001 | 20220328173408179   | col_date=2020-06-01/col_int=7    | uuid:6             | 20220328173408179_0_16 | 74828055-72ee-4aac-a574-db9f7dd5bacc-0_0-773-778_20220328173408179.parquet   |
|   11 |           1 |      12 |       11 |    11.011 |   11.00011 |         110000000000.000000000110000000 | 2020-11-01 | Light installations featuring the tiger, the Chinese zodiac animal for the upcoming lunar year, will be set up in the main plaza.                                                                                                                                                                                         | 1110001010101011001001 | 20220328173510429   | col_date=2020-11-01/col_int=12   | uuid:11            | 20220328173510429_0_14 | db560743-ab82-466d-aa1d-4b393a3dc5f4-0_0-1069-1429_20220328173510429.parquet |
|    2 |           0 |       3 |        2 |     2.002 |     2.0002 |          20000000000.000000000200000000 | 2020-02-01 | Xi Jinping, general secretary of the Communist Party of China Central Committee, called on Tuesday for unrelenting efforts in exercising full and rigorous governance over the Party, saying that the CPC will continue to show zero tolerance for corruption.                                                            | 1110001010101011001001 | 20220328173325236   | col_date=2020-02-01/col_int=3    | uuid:2             | 20220328173325236_0_10 | 480dbff0-d609-44e9-b390-7f0f44c9169a-0_0-539-482_20220328173325236.parquet   |
|    7 |           1 |       8 |        7 |     7.007 |     7.0007 |          70000000000.000000000700000000 | 2020-07-01 | Most listed companies in the top 10 cities come from industries such as information technology, electronics, mechanical equipment, medical biology, and electric power equipment, with information technology bearing most listed companies.                                                                              | 1110001010101011001001 | 20220328173421883   | col_date=2020-07-01/col_int=8    | uuid:7             | 20220328173421883_0_18 | 028eb4bd-34ed-412b-9fd0-97fbe2337199-0_0-834-890_20220328173421883.parquet   |
|   12 |           0 |      13 |       12 |    12.012 |   12.00012 |         120000000000.000000000120000000 | 2020-12-01 | The annual lantern show at Yuyuan Garden, a historic tourist site in Shanghai, will kick off on Tuesday and run through Feb 28.                                                                                                                                                                                           | 1110001010101011001001 | 20220328173522636   | col_date=2020-12-01/col_int=13   | uuid:12            | 20220328173522636_0_16 | f46da3af-bc0e-47f1-9c0d-4eaa709f548c-0_0-1129-1590_20220328173522636.parquet |
|    1 |           1 |       1 |        1 |     1.001 |     1.0001 |          10000000000.000000000100000000 | 2020-01-01 | Top 10 Unsolved Mysteries of Paleontological Dinosaurs, Did You Know?                                                                                                                                                                                                                                                     | 1110001010101011001001 | 20220328173307979   | col_date=2020-01-01/col_int=1    | uuid:1             | 20220328173307979_0_8  | 41a60295-8f3d-4510-92a1-cbadf878530e-0_0-456-404_20220328173307979.parquet   |
|    6 |           0 |       6 |        6 |     6.006 |     6.0006 |          60000000000.000000000600000000 | 2020-06-01 | The development of enterprises, as well as the push of local governments, has boosted the listings, the report said. For example, in early 2018, Central Chinas Hubei province launched a plan to double its listed companies by including the listing of enterprises into the governments annual performance assessment. | 1110001010101011001001 | 20220328173402362   | col_date=2020-06-01/col_int=6    | uuid:6             | 20220328173402362_0_10 | 0d63a4b7-aad9-4454-b551-b2de7dc86e3b-0_0-745-728_20220328173402362.parquet   |
|   11 |           1 |      11 |       11 |    11.011 |   11.00011 |         110000000000.000000000110000000 | 2020-11-01 | Light installations featuring the tiger, the Chinese zodiac animal for the upcoming lunar year, will be set up in the main plaza.                                                                                                                                                                                         | 1110001010101011001001 | 20220328173504921   | col_date=2020-11-01/col_int=11   | uuid:11            | 20220328173504921_0_13 | f22d70ad-303d-44a1-8ce7-c669f2c3e97b-0_0-1041-1354_20220328173504921.parquet |
|    2 |           0 |       2 |        2 |     2.002 |     2.0002 |          20000000000.000000000200000000 | 2020-02-01 | Xi Jinping, general secretary of the Communist Party of China Central Committee, called on Tuesday for unrelenting efforts in exercising full and rigorous governance over the Party, saying that the CPC will continue to show zero tolerance for corruption.                                                            | 1110001010101011001001 | 20220328173320684   | col_date=2020-02-01/col_int=2    | uuid:2             | 20220328173320684_0_9  | d843c0bd-335a-434d-be8d-a15637ce52bf-0_0-510-453_20220328173320684.parquet   |
|    7 |           1 |       7 |        7 |     7.007 |     7.0007 |          70000000000.000000000700000000 | 2020-07-01 | Most listed companies in the top 10 cities come from industries such as information technology, electronics, mechanical equipment, medical biology, and electric power equipment, with information technology bearing most listed companies.                                                                              | 1110001010101011001001 | 20220328173416623   | col_date=2020-07-01/col_int=7    | uuid:7             | 20220328173416623_0_17 | 7b0632a8-bef3-4822-a23a-cd7d87115e48-0_0-805-834_20220328173416623.parquet   |
|   12 |           0 |      12 |       12 |    12.012 |   12.00012 |         120000000000.000000000120000000 | 2020-12-01 | The annual lantern show at Yuyuan Garden, a historic tourist site in Shanghai, will kick off on Tuesday and run through Feb 28.                                                                                                                                                                                           | 1110001010101011001001 | 20220328173517108   | col_date=2020-12-01/col_int=12   | uuid:12            | 20220328173517108_0_15 | a6b2b7ff-3a33-439b-b581-acc6aade96f9-0_0-1100-1509_20220328173517108.parquet |
+------+-------------+---------+----------+-----------+------------+-----------------------------------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------+---------------------+----------------------------------+--------------------+------------------------+------------------------------------------------------------------------------+
26 rows in set (0.07 sec)

Expected behavior (Required)

return the right result

Real behavior (Required)

query error

StarRocks version (Required)

miomiocat commented 2 years ago

please assign to me

tiannan-sr commented 2 years ago

BE crash when the query with the error executed several times: Core was generated by `/home/disk1/sr/qa_test_master/be/lib/starrocks_be'. Program terminated with signal SIGABRT, Aborted.

0 0x00007fbfead2d387 in raise () from /lib64/libc.so.6

[Current thread is 1 (Thread 0x7fbeebc04700 (LWP 20258))] (gdb) bt

0 0x00007fbfead2d387 in raise () from /lib64/libc.so.6

1 0x00007fbfead2ea78 in abort () from /lib64/libc.so.6

2 0x0000000001781aea in google::DumpStackTraceAndExit () at src/utilities.cc:160

3 0x0000000003f6b4ad in google::LogMessage::Fail () at src/logging.cc:1677

4 0x0000000003f6d91f in google::LogMessage::SendToLog (this=0x7fbe4d816500) at src/logging.cc:1634

5 0x0000000003f6affe in google::LogMessage::Flush (this=0x7fbe4d816500) at src/logging.cc:1503

6 0x0000000003f6df29 in google::LogMessageFatal::~LogMessageFatal (this=, __in_chrg=) at src/logging.cc:2253

7 0x00000000024bec42 in starrocks::type_dispatch_column<starrocks::vectorized::ColumnBuilder, starrocks::TypeDescriptor, unsigned long> (ptype=starrocks::INVALID_TYPE, fun=...)

at /usr/include/c++/10.3.0/bits/char_traits.h:371

8 0x00000000024bdbd2 in starrocks::vectorized::ColumnHelper::create_column (type_desc=..., nullable=, is_const=, size=size@entry=1)

at /root/starrocks/be/src/column/column_helper.cpp:226

9 0x00000000032c46aa in starrocks::serde::ProtobufChunkDeserializer::deserialize (this=this@entry=0x7fbe4d816830, buff=..., deserialized_bytes=deserialized_bytes@entry=0x0)

at /usr/include/c++/10.3.0/bits/stl_bvector.h:348

10 0x000000000204d3b8 in starrocks::DataStreamRecvr::SenderQueue::_deserialize_chunk (this=, pchunk=..., chunk=0x62f3ba40, uncompressed_buffer=0x7fbe4d816aa0)

at /root/starrocks/be/src/runtime/data_stream_recvr.cc:715

11 0x000000000204e4ab in starrocks::DataStreamRecvr::SenderQueue::add_chunks (this=0x7095d680, request=..., done=0x7fbe4d816c78, is_pipeline=)

at /root/starrocks/be/src/runtime/data_stream_recvr.cc:492

12 0x00000000020503b9 in starrocks::DataStreamRecvr::add_chunks (this=0x7826bdc0, request=..., done=done@entry=0x7fbe4d816c78)

at /root/starrocks/be/src/runtime/data_stream_recvr.cc:927

13 0x000000000202f35d in starrocks::DataStreamMgr::transmit_chunk (this=, request=..., done=done@entry=0x7fbe4d816c78)

at /root/starrocks/be/src/runtime/data_stream_mgr.cpp:162

14 0x000000000332ec8c in starrocks::PInternalServiceImplBase::transmit_chunk (this=0xb49b350, cntl_base=, request=0x76d4e070,

response=0xd3c1320, done=<optimized out>) at /root/starrocks/be/src/runtime/exec_env.h:104

15 0x00000000040a982e in brpc::policy::ProcessRpcRequest (msg_base=) at /var/local/thirdparty/src/incubator-brpc-0.9.7/src/brpc/policy/baidu_rpc_protocol.cpp:496

16 0x00000000040a0297 in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x6767a900) at /var/local/thirdparty/src/incubator-brpc-0.9.7/src/brpc/input_messenger.cpp:135

17 0x00000000040a1143 in brpc::RunLastMessage::operator() (last_msg=0x6767a900, this=)

at /var/local/thirdparty/src/incubator-brpc-0.9.7/src/brpc/input_messenger.cpp:141

18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=)

at /usr/include/c++/10.3.0/bits/unique_ptr.h:361

19 brpc::InputMessenger::OnNewMessages (m=0x726d2000) at /usr/include/c++/10.3.0/bits/unique_ptr.h:355

20 0x0000000004147e0e in brpc::Socket::ProcessEvent (arg=0x726d2000) at /var/local/thirdparty/src/incubator-brpc-0.9.7/src/brpc/socket.cpp:1017

21 0x0000000004055d9f in bthread::TaskGroup::task_runner (skip_remained=) at /var/local/thirdparty/src/incubator-brpc-0.9.7/src/bthread/task_group.cpp:296

22 0x00000000041de581 in bthread_make_fcontext ()

23 0x0000000000000000 in ?? ()

(gdb)

miomiocat commented 2 years ago

set enable_exchange_pass_through=true