search

br1ghtyang / asterixdb

Automatically exported from code.google.com/p/asterixdb
0 stars 0 forks source link

Unable to load/process external data from HDFS with newer hadoop version #521

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
What steps will reproduce the problem?
1. Following the instructions given over 
(http://asterixdb.ics.uci.edu/documentation/aql/externaldata.html), I tried 
loading the sample dataset:

drop dataverse ExternalFileDemo if exists;
create dataverse ExternalFileDemo;
use dataverse ExternalFileDemo;

create type LineitemType as closed {
      l_orderkey:int32,
      l_partkey: int32,
      l_suppkey: int32,
      l_linenumber: int32,
      l_quantity: double,
      l_extendedprice: double,
      l_discount: double,
      l_tax: double,
      l_returnflag: string,
      l_linestatus: string,
      l_shipdate: string,
      l_commitdate: string,
      l_receiptdate: string,
      l_shipinstruct: string,
      l_shipmode: string,
      l_comment: string
}

create external dataset Lineitem(LineitemType)
using hdfs 
(("hdfs"="hdfs://127.0.0.1:9000/"),("path"="/user/tejas/asterix/lineitem.tbl"),(
"input-format"="text-input-format"),("format"="delimited-text"),("delimiter"="|"
));

for $c in dataset('Lineitem')
    where $c.l_orderkey <= 3
    order by $c.l_orderkey, $c.l_linenumber
    return $c

2. I have verified if the HDFS instance is running and the file path is correct:

hadoop dfs  -cat hdfs://127.0.0.1:9000/user/tejas/asterix/lineitem.tbl

1|156|4|1|17|17954.55|0.04|0.02|N|O|1996-03-13|1996-02-12|1996-03-22|DELIVER IN 
PERSON|TRUCK|egular courts above the|
1|68|9|2|36|34850.16|0.09|0.06|N|O|1996-04-12|1996-02-28|1996-04-20|TAKE BACK 
RETURN|MAIL|ly final dependencies: slyly bold |
1|64|5|3|8|7712.48|0.10|0.02|N|O|1996-01-29|1996-03-05|1996-01-31|TAKE BACK 
RETURN|REG AIR|riously. regular, express dep|
................
................
................

What is the expected output? What do you see instead?
I got this error message over the asterix UI:

Unable to create adapter org.apache.hadoop.ipc.RemoteException: Server IPC 
version 8 cannot communicate with client version 3 [AlgebricksException]

What version of the product are you using? On what operating system?
Asterix 0.8.0 running on Ubuntu 12.10, single node setup.
Hadoop 2.0.3-alpha

Please provide any additional information below.

Original issue reported on code.google.com by tejas.pa...@gmail.com on 8 Jun 2013 at 7:55

GoogleCodeExporter commented 8 years ago 
Asterix still uses hadoop-0.20.2 jars. The hadoop downloads page doesn't 
provide link for v20.2 (http://mirror.tcpdiag.net/apache/hadoop/common/). We 
should move on.

Original comment by tejas.pa...@gmail.com on 8 Jun 2013 at 8:15

GoogleCodeExporter commented 8 years ago 
Raman - please own and update the status on this!  Thx.

Original comment by dtab...@gmail.com on 2 Aug 2013 at 5:53