search

Huawei-Spark / Spark-SQL-on-HBase

Native, optimized access to HBase Data through Spark SQL/Dataframe Interfaces
Apache License 2.0
321 stars 164 forks source link

The LOAD function cannot load data from file #8

Closed Zhangshunyu closed 9 years ago

Zhangshunyu commented 9 years ago

Hello, I have a question about the function of LOAD. I want to load the data which was stored as TEXT file under the path of hdfs, so i did as follows: astro> select * from sql_zsy05; //the table before my data is loaded OK +----+----+----+------+ |key1|key2|key3| value| +----+----+----+------+ | 9| 9| 7|hidfla| | 9| 9| 8|hidfla| | 9| 9| 9|hidfla| +----+----+----+------+

Time taken: 6.559 seconds astro> LOAD DATA INPATH "hdfs://10.67.180.114:9000/TEST" OVERWRITE INTO TABLE sql_zsy05 FIELDS TERMINATED BY '\t'; OK
Time taken: 7.796 seconds astro> select * from sql_zsy05; // after the data is loaded successful, the table has no change
OK +----+----+----+------+ |key1|key2|key3| value| +----+----+----+------+ | 9| 9| 7|hidfla| | 9| 9| 8|hidfla| | 9| 9| 9|hidfla| +----+----+----+------+

So, what should I do? Did I used the method incorrectly？Thank you！

xinyunh commented 9 years ago

@Zhangshunyu Don't worry. We will take a look.

xinyunh commented 9 years ago

Hi @Zhangshunyu, You operate correctly. And it seems to be an unsupported feature.

Hi @yzhou2001, When are you open to a phone call? We might need to talk about some details of this issue.

xinyunh commented 9 years ago

@Zhangshunyu I check the parser. The keyword 'overwrite' is useless. We will move fast to support it soon.

Zhangshunyu commented 9 years ago

@xinyunh Hi xinyun, I find that the LOAD function could load data successfully in YARN-CLUSTER mode, and I must point out that the implementation of the test 3 days ago was in STAND-ALONE mode.

xinyunh commented 9 years ago

Hi @Zhangshunyu ,

We found it always overwrites the original data, no matter we have the key word 'overwrite' or not. We might need your help to figure out the reason of this issue. So, may we know your hbase version number? My WeChat ID is "phoenix_newlife". If you want, we could have a talk directly.

Zhangshunyu commented 9 years ago

@xinyunh My hbase version number is 0.98

xinyunh commented 9 years ago

Hi @Zhangshunyu,

Yeah, we are using the same version. It seems we need more details to reproduce this issue, for example, the test data. And it is better we could have a phone call.

Zhangshunyu commented 9 years ago

@xinyunh OK, I friend you on WeChat and You can contact me anytime. Besides, do you have Espace?