sollhui
commented
7 months ago Can the retry be successful？
Open zhaozhihua2008 opened 7 months ago
sollhui
commented
7 months ago Can the retry be successful？
zhaozhihua2008
commented
7 months ago Can the retry be successful？
No，it can not. The data is exported from mysql. Import it to clickhouse has no problem, but import it to doris got exception. Don't know why.
sollhui
commented
7 months ago Can the retry be successful？
No，it can not. The data is exported from mysql. Import it to clickhouse has no problem, but import it to doris got exception. Don't know why.
Sorry for delay, we can not recurrent in 2.0.3, I think may be other reason cause this fail, like replica failed or memory is not enough. To verify it, you can search label "4e5d4cbc-744c-4e32-80f7-8b1a2e02e295" in log to find why load failed.
Search before asking
Version
2.0.3
What's Wrong?
stream load 500w records successfully， but load 1000w of data failed. error log as follows. { "TxnId": 1511, "Label": "4e5d4cbc-744c-4e32-80f7-8b1a2e02e295", "Comment": "", "TwoPhaseCommit": "false", "Status": "Fail", "Message": "[INTERNAL_ERROR]cancelled: closed\n\n\t0# doris::io::StreamLoadPipe::_append(std::shared_ptr const&, unsigned long) at /root/src/doris-2.0/be/src/common/status.h:0\n\t1# doris::io::StreamLoadPipe::append(std::shared_ptr const&) at /root/src/doris-2.0/be/src/io/fs/stream_load_pipe.cpp:162\n\t2# doris::StreamLoadAction::on_chunk_data(doris::HttpRequest) at /root/src/doris-2.0/be/src/common/status.h:442\n\t3# ?\n\t4# bufferevent_runreadcb\n\t5# ?\n\t6# ?\n\t7# event_base_loop\n\t8# std::_Function_handler<void (), doris::EvHttpServer::start()::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98\n\t9# doris::ThreadPool::dispatch_thread() at /root/src/doris-2.0/be/src/util/threadpool.cpp:0\n\t10# doris::Thread::supervise_thread(void) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562\n\t11# start_thread\n\t12# __clone\n",
"NumberTotalRows": 0,
"NumberLoadedRows": 0,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 637441272,
"LoadTimeMs": 12338,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 19,
"ReadDataTimeMs": 7874,
"WriteDataTimeMs": 0,
"CommitAndPublishTimeMs": 0
}
What You Expected?
load 1000w data successfully.
How to Reproduce?
1.create table : CREATE TABLE p_ele_ec_meter_h ( obj_id varchar(64) NOT NULL COMMENT '（1）按对象、时间粒度进行汇总，其中对象、时间粒度必须有效。 （2）对象有可能企业、机构、地址、工艺、系统、单元', time_level tinyint(4) NOT NULL COMMENT '21：15分钟 0：时 1： 日 3：月 4：年', obj_rel_id varchar(64) NOT NULL COMMENT '对象所处的统计关系id', root_org_id varchar(64) NOT NULL COMMENT '企业id', obj_type int(11) NOT NULL COMMENT '对象类型', data_time datetime NOT NULL COMMENT '数据时间', obj_name varchar(255) DEFAULT NULL COMMENT '对象名称', obj_rel_level int(11) DEFAULT NULL COMMENT '在统计关系中，对象所在层级，和c_ec_statistic_relation的node_level匹配', ec_value double DEFAULT NULL COMMENT '当前周期总电量值', gu_value double DEFAULT NULL COMMENT '当前周期谷电量值', feng_value double DEFAULT NULL COMMENT '当前周期峰电量值', jian_value double DEFAULT NULL COMMENT '当前周期尖电量值', ping_value double DEFAULT NULL COMMENT '当前周期平电量值', ec_kgce_value double DEFAULT NULL COMMENT '当前周期折标准煤值', tmp_ec_value double DEFAULT NULL, base_value double DEFAULT NULL, ec_value_min double DEFAULT NULL, ec_value_max double DEFAULT NULL, en_type_id int(11) DEFAULT '1', en_type_level2_id int(11) DEFAULT '1', direct_type int(11) DEFAULT NULL COMMENT '1：输入 2：输出 记录统计关系中输入/出的汇总形式。只有电量汇总会有输出，为了计算损耗。' ) UNIQUE KEY(obj_id,time_level,obj_rel_id,root_org_id,obj_type,data_time) PARTITION BY RANGE(data_time)() DISTRIBUTED BY HASH(obj_id) PROPERTIES( "replication_num" = "2", "dynamic_partition.time_unit" = "MONTH", "dynamic_partition.create_history_partition"="true" "dynamic_partition.start" = "-24", "dynamic_partition.end" = "3", "dynamic_partition.prefix" = "p", "dynamic_partition.buckets" = "4" );
3.import data curl --location-trusted -u root:root -H "column_separator:," -H "exec_mem_limit:8589934592" -H "columns:obj_id,root_org_id,e c_value,gu_value,obj_name,obj_type,data_time,feng_value,jian_value,ping_value,time_level,ec_kgce_value,tmp_ec_value,base_val ue,obj_rel_id,obj_rel_level,ec_value_min,ec_value_max,en_type_id,en_type_level2_id,direct_type" -T /root/zzh/test1.csv http: //localhost:8030/api/example_db/p_ele_ec_meter_h1/_stream_load
Anything Else?
no
Are you willing to submit PR?
Code of Conduct