Closed alphaCoder closed 9 years ago
You are not setting the node of the Elasticsearch cluster correctly - should be es.nodes
not es-nodes
.
Hence why the task executes against the default location (localhost
) where Elasticsearch is not running.
Thank you Costin. My bad I missed that.
Hi,
I am using Horton works hadoop 2.1 sandbox. I downloaded the ES-hadoop 2.0.2 jar files and add them to my hive query.
My query looks like this.
CREATE EXTERNAL TABLE Tweet ( user STRING, message STRING ) STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler' TBLPROPERTIES('es.resource' = 'twitter/tweet', 'es.index.auto.create' = 'false', 'es-nodes' = 'http://alpha-es' )
I am trying connect to the remote elasticsearch cluster. I even tried replacing the http://alpha-es:9200 or alpha-es nothing works. I am getting the below error.
java.io.IOException: org.elasticsearch.hadoop.rest.EsHadoopNoNodesLeftException: Connection error (check network and/or proxy settings)- all nodes failed; tried [[localhost:9200]]
more detailed log:
14/12/04 15:03:11 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr. 14/12/04 15:03:11 INFO log.PerfLogger:
14/12/04 15:03:11 INFO log.PerfLogger:
14/12/04 15:03:11 INFO ql.Driver: Concurrency mode is disabled, not creating a lock manager
14/12/04 15:03:11 INFO log.PerfLogger:
14/12/04 15:03:11 INFO log.PerfLogger:
14/12/04 15:03:11 INFO parse.ParseDriver: Parsing command: use default
14/12/04 15:03:11 INFO parse.ParseDriver: Parse Completed
14/12/04 15:03:11 INFO log.PerfLogger: </PERFLOG method=parse start=1417734191820 end=1417734191822 duration=2 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:11 INFO log.PerfLogger:
14/12/04 15:03:11 INFO ql.Driver: Semantic Analysis Completed
14/12/04 15:03:11 INFO log.PerfLogger: </PERFLOG method=semanticAnalyze start=1417734191823 end=1417734191824 duration=1 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:11 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
14/12/04 15:03:11 INFO log.PerfLogger:
14/12/04 15:03:11 INFO log.PerfLogger: </PERFLOG method=doAuthorization start=1417734191825 end=1417734191826 duration=1 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:11 INFO log.PerfLogger: </PERFLOG method=compile start=1417734191816 end=1417734191826 duration=10 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:11 INFO log.PerfLogger:
14/12/04 15:03:11 INFO ql.Driver: Starting command: use default
14/12/04 15:03:12 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/12/04 15:03:12 INFO hooks.ATSHook: Created ATS Hook
14/12/04 15:03:12 INFO log.PerfLogger:
14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1417734192336 end=1417734192338 duration=2 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1417734191814 end=1417734192338 duration=524 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:12 INFO log.PerfLogger:
14/12/04 15:03:12 INFO log.PerfLogger:
14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=runTasks start=1417734192338 end=1417734192413 duration=75 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:12 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/12/04 15:03:12 INFO hooks.ATSHook: Created ATS Hook
14/12/04 15:03:12 INFO log.PerfLogger:
14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1417734192961 end=1417734192962 duration=1 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1417734191826 end=1417734192964 duration=1138 from=org.apache.hadoop.hive.ql.Driver>
OK
14/12/04 15:03:12 INFO ql.Driver: OK
14/12/04 15:03:12 INFO log.PerfLogger:
14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=releaseLocks start=1417734192964 end=1417734192964 duration=0 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=Driver.run start=1417734191814 end=1417734192964 duration=1150 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:12 INFO log.PerfLogger:
14/12/04 15:03:12 INFO log.PerfLogger:
14/12/04 15:03:12 INFO parse.ParseDriver: Parsing command: SELECT * FROM
14/12/04 15:03:12 INFO parse.SemanticAnalyzer: Starting Semantic Analysis
14/12/04 15:03:12 INFO parse.SemanticAnalyzer: Completed phase 1 of Semantic Analysis
14/12/04 15:03:12 INFO parse.SemanticAnalyzer: Get metadata for source tables
14/12/04 15:03:13 INFO parse.SemanticAnalyzer: Get metadata for subqueries
14/12/04 15:03:13 INFO parse.SemanticAnalyzer: Get metadata for destination tables
14/12/04 15:03:13 INFO ql.Context: New scratch dir is hdfs://sandbox.hortonworks.com:8020/tmp/hive-beeswax-hue/hive_2014-12-04_15-03-12_967_1966756796962932357-1
14/12/04 15:03:13 INFO parse.SemanticAnalyzer: Completed getting MetaData in Semantic Analysis
14/12/04 15:03:13 INFO parse.SemanticAnalyzer: Set stats collection dir : hdfs://sandbox.hortonworks.com:8020/tmp/hive-beeswax-hue/hive_2014-12-04_15-03-12_967_1966756796962932357-1/-ext-10002
14/12/04 15:03:13 INFO ppd.OpProcFactory: Processing for FS(137)
14/12/04 15:03:13 INFO ppd.OpProcFactory: Processing for SEL(136)
14/12/04 15:03:13 INFO ppd.OpProcFactory: Processing for TS(135)
14/12/04 15:03:13 INFO parse.SemanticAnalyzer: Completed plan generation
14/12/04 15:03:13 INFO ql.Driver: Semantic Analysis Completed
14/12/04 15:03:13 INFO log.PerfLogger: </PERFLOG method=semanticAnalyze start=1417734192973 end=1417734193164 duration=191 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:13 INFO exec.TableScanOperator: Initializing Self 135 TS
14/12/04 15:03:13 INFO exec.TableScanOperator: Operator 135 TS initialized
14/12/04 15:03:13 INFO exec.TableScanOperator: Initializing children of 135 TS
14/12/04 15:03:13 INFO exec.SelectOperator: Initializing child 136 SEL
14/12/04 15:03:13 INFO exec.SelectOperator: Initializing Self 136 SEL
14/12/04 15:03:13 INFO exec.SelectOperator: SELECT structuser:string,message:string
14/12/04 15:03:13 INFO exec.SelectOperator: Operator 136 SEL initialized
14/12/04 15:03:13 INFO exec.SelectOperator: Initializing children of 136 SEL
14/12/04 15:03:13 INFO exec.ListSinkOperator: Initializing child 138 OP
14/12/04 15:03:13 INFO exec.ListSinkOperator: Initializing Self 138 OP
14/12/04 15:03:13 INFO exec.ListSinkOperator: Operator 138 OP initialized
14/12/04 15:03:13 INFO exec.ListSinkOperator: Initialization Done 138 OP
14/12/04 15:03:13 INFO exec.SelectOperator: Initialization Done 136 SEL
14/12/04 15:03:13 INFO exec.TableScanOperator: Initialization Done 135 TS
14/12/04 15:03:13 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:default.tweet.user, type:string, comment:null), FieldSchema(name:default.tweet.message, type:string, comment:null)], properties:null)
14/12/04 15:03:13 INFO log.PerfLogger:
14/12/04 15:03:13 INFO log.PerfLogger: </PERFLOG method=doAuthorization start=1417734193169 end=1417734193481 duration=312 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:13 INFO log.PerfLogger: </PERFLOG method=compile start=1417734192965 end=1417734193481 duration=516 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:13 INFO hive.metastore: Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083
14/12/04 15:03:13 INFO hive.metastore: Connected to metastore.
14/12/04 15:03:13 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
14/12/04 15:03:13 INFO log.PerfLogger:
14/12/04 15:03:13 INFO ql.Driver: Starting command: SELECT * FROM
14/12/04 15:03:14 INFO log.PerfLogger: </PERFLOG method=PreHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1417734194312 end=1417734194314 duration=2 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:14 INFO log.PerfLogger: </PERFLOG method=TimeToSubmit start=1417734191814 end=1417734194316 duration=2502 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:14 INFO log.PerfLogger:
14/12/04 15:03:14 INFO log.PerfLogger: </PERFLOG method=runTasks start=1417734194316 end=1417734194316 duration=0 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:14 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/
14/12/04 15:03:14 INFO hooks.ATSHook: Created ATS Hook
14/12/04 15:03:14 INFO log.PerfLogger:
14/12/04 15:03:14 INFO log.PerfLogger: </PERFLOG method=PostHook.org.apache.hadoop.hive.ql.hooks.ATSHook start=1417734194628 end=1417734194628 duration=0 from=org.apache.hadoop.hive.ql.Driver>
14/12/04 15:03:14 INFO log.PerfLogger: </PERFLOG method=Driver.execute start=1417734193528 end=1417734194629 duration=1101 from=org.apache.hadoop.hive.ql.Driver>
OK
14/12/04 15:03:14 INFO ql.Driver: OK
14/12/04 15:03:16 INFO session.SessionState: No Tez session required at this point. hive.execution.engine=mr.
14/12/04 15:03:16 INFO httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
14/12/04 15:03:16 INFO httpclient.HttpMethodDirector: Retrying request
14/12/04 15:03:16 INFO httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
14/12/04 15:03:16 INFO httpclient.HttpMethodDirector: Retrying request
14/12/04 15:03:16 INFO httpclient.HttpMethodDirector: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
14/12/04 15:03:16 INFO httpclient.HttpMethodDirector: Retrying request
14/12/04 15:03:16 ERROR rest.NetworkClient: Node [Connection refused] failed (localhost:9200); no other nodes left - aborting...
default.tweet
14/12/04 15:03:12 INFO parse.ParseDriver: Parse Completed 14/12/04 15:03:12 INFO log.PerfLogger: </PERFLOG method=parse start=1417734192968 end=1417734192972 duration=4 from=org.apache.hadoop.hive.ql.Driver> 14/12/04 15:03:12 INFO log.PerfLogger:default.tweet
14/12/04 15:03:14 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 14/12/04 15:03:14 INFO hooks.ATSHook: Created ATS Hook 14/12/04 15:03:14 INFO log.PerfLogger:Appreciate any help.