apache / doris

Apache Doris is an easy-to-use, high performance and unified analytics database.
https://doris.apache.org
Apache License 2.0
12.19k stars 3.19k forks source link

how to load orc file to table using default broker #5353

Open peng-xin opened 3 years ago

peng-xin commented 3 years ago

when i load orc file to doris,it is always error. image

env: hive: 2.3.4 doris: 0.13(build in docker)

doris table :

CREATE TABLE user_test ( key int(11) NULL COMMENT "", name varchar(1024) NULL COMMENT "", age int(11) NULL COMMENT "", birth datetime NULL COMMENT "", money decimal(10,2) NULL COMMENT "" ) ENGINE=OLAP UNIQUE KEY(key, name, age,birth) COMMENT "OLAP" DISTRIBUTED BY HASH(key) BUCKETS 10 PROPERTIES ( "replication_num" = "1", "in_memory" = "false", "storage_format" = "V2" );

hive table :

CREATE TABLE user_test_orc( key int, name string, age int, money double, birth timestamp) ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' LOCATION 'hdfs://hadoop01:9000/user/hive/warehouse/user_test_orc' TBLPROPERTIES ( 'transient_lastDdlTime'='1612337896')

hive table data:

insert into user_test_orc values('1','张三','18','2','1990-12-01 00:01:07'), ('2','张三','18','2','1990-12-01 00:01:07'), ('3','张三','18','2','1990-12-01 00:01:07'), ('4','张三','18','2','1990-12-01 00:01:07'), ('5','张三','18','2','1990-12-01 00:01:07'), ('6','张三','18','2','1990-12-01 00:01:07'), ('7','张三','18','2','1990-12-01 00:01:07'), ('8','张三','18','2','1990-12-01 00:01:07'), ('9','张三','18','2','1990-12-01 00:01:07'), ('10','lisi','11','7','1997-12-01 00:01:07'), ('11','lisi','11','7','1997-12-01 00:01:07'), ('12','lisi','11','7','1997-12-01 00:01:07'), ('13','lisi','11','7','1997-12-01 00:01:07'), ('14','lisi','11','7','1997-12-01 00:01:07'), ('15','lisi','11','7','1997-12-01 00:01:07'), ('16','lisi','11','7','1997-12-01 00:01:07'), ('17','lisi','11','7','1997-12-01 00:01:07'), ('18','lisi','11','7','1997-12-01 00:01:07'), ('19','lisi','11','7','1997-12-01 00:01:07')

load shell:

LOAD LABEL test.user_test ( DATA INFILE("hdfs://172.16.116.102:9000/user/hive/warehouse/user_test_orc/*") INTO TABLE user_test ) WITH BROKER "hdfs_broker" ( "username" = "eagle", "password" = "" ) PROPERTIES ( "timeout" = "10800", "max_filter_ratio" = "0.001" );

morningman commented 3 years ago

You need to specify FORMAT AS 'orc' in your load stmt. execute help broker load to see examples.

peng-xin commented 3 years ago

You need to specify FORMAT AS 'orc' in your load stmt. execute help broker load to see examples.

it is worked.thank you so much.