search

forward3d / rbhive

Ruby gem for querying Apache Hive
http://www.forward3d.com
MIT License
98 stars 75 forks source link

rbhive broken on cdh3u4 release/update (Apache Hive 0.7.1-cdh3u4) #2

Closed fuentesjr closed 11 years ago

fuentesjr commented 12 years ago

I should mention that it was working fine on the cdh3u3 release.

bash-3.2$ cat test_rbhive.rb require 'rbhive'

results = RBHive.connect('hive.server.address') do |connection|
connection.fetch "select state_code from zip_code" end

results.each do |data| puts "data = #{data.inspect}" end

bash-3.2$ ruby test_rbhive.rb Connecting to hive.server.address on port 10000 Executing Hive Query: select state_code from zip_code /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:26:in recv_execute': Query returned non-zero code: 10, cause: FAILED: Error in semantic analysis: Unable to fetch table zip_code (HiveServerException) from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:17:inexecute' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:140:in execute_unsafe' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:81:inblock in fetch' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in block in safe' from <internal:prelude>:10:insynchronize' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in safe' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:80:infetch' from test_rbhive.rb:4:in block in <main>' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:14:inconnect' from test_rbhive.rb:3:in `

'

forward-bot commented 12 years ago

Hi Sal,

We upgraded one of our Hive servers to CDH3U4 and were able to query it using RBHive. Is there any chance the CH3U4 over-wrote your Hive config files which would mean Hive doesn't know about the existing tables?

The upgrade does ask if it can install new config files on Ubuntu (not sure what OS you're using).

You can check if it's an RBHive issue or a Hive issue by trying out the same query in the Hive CLI on the actual Hive server. If the query works on the Hive CLI then it would be an RBHive issue, otherwise it might be something to do with your Hive installation.

Let me know what you find.

Thanks.

Abs

On 27 Jun 2012, at 23:05, Sal Fuentes wrote:

I should mention that it was working fine on the cdh3u3 release.

bash-3.2$ cat test_rbhive.rb require 'rbhive'

results = RBHive.connect('hive.server.address') do |connection|
connection.fetch "select state_code from zip_code" end

results.each do |data| puts "data = #{data.inspect}" end

bash-3.2$ ruby test_rbhive.rb Connecting to hive.server.address on port 10000 Executing Hive Query: select state_code from zipcode /home/****/.rvm/gems/ruby-1.9.2-p290@_/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:26:in recv_execute': Query returned non-zero code: 10, cause: FAILED: Error in semantic analysis: Unable to fetch table zip_code (HiveServerException) from /home/****_/.rvm/gems/ruby-1.9.2-p290@**_**/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:17:inexecute' from /home/**_/.rvm/gems/ruby-1.9.2-p290@_/gems/rbhive-0.2.94/lib/rbhive/connection.rb:140:in `execute_unsafe' from /home/**/.rvm/gems/ruby-1.9.2-p290@**/gems/rbhive-0.2.94/lib/rbhive/connection.rb:81:in `block in fetch' from /home/**/.rvm/gems/ruby-1.9.2-p290@**/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in block in safe' from <internal:prelude>:10:insynchronize' from /home/**/.rvm/gems/ruby-1.9.2-p290@**/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in `safe' from /home/**/.rvm/gems/ruby-1.9.2-p290@**/gems/rbhive-0.2.94/lib/rbhive/connection.rb:80:in fetch' from test_rbhive.rb:4:inblock in

' from /home/**/.rvm/gems/ruby-1.9.2-p290@****/gems/rbhive-0.2.94/lib/rbhive/connection.rb:14:in connect' from test_rbhive.rb:3:in
'

Reply to this email directly or view it on GitHub: https://github.com/forward/rbhive/issues/2

fuentesjr commented 12 years ago

I tried the same query in the Hive CLI on the same Hive server the test script connects to with the following results:

hive> select state_code from zip_code limit 3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201206270101_16742, Tracking URL = http://jobtracker-prod.server.addr:50030/jobdetails.jsp?jobid=job_201206270101_16742 Kill Command = /home/t/hadoop_current/bin/hadoop job -Dmapred.job.tracker=jobtracker-prod.server.addr:9001 -kill job_201206270101_16742 2012-06-29 17:42:33,427 Stage-1 map = 0%, reduce = 0% 2012-06-29 17:42:39,475 Stage-1 map = 33%, reduce = 0% 2012-06-29 17:42:42,508 Stage-1 map = 100%, reduce = 0% 2012-06-29 17:42:45,531 Stage-1 map = 100%, reduce = 100% Ended Job = job_201206270101_16742 OK NY NY NY Time taken: 20.205 seconds hive> exit;

I also tweaked the script and ran it again:

bash-3.2$ cat test_rbhive.rb require 'rbhive'

results = RBHive.connect('tron.ev1.yellowpages.com') do |connection|

connection.fetch "select state_code from zip_code"

connection.fetch "describe zip_code" end

results.each do |data| puts "data = #{data.inspect}" end

bash-3.2$ ruby test_rbhive.rb Connecting to hive.server.addr on port 10000 Executing Hive Query: describe zip_code/home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:26:in recv_execute': Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (HiveServerException) from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:17:inexecute' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:140:in execute_unsafe' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:81:inblock in fetch' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in block in safe' from <internal:prelude>:10:insynchronize' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in safe' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:80:infetch' from test_rbhive.rb:5:in block in <main>' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:14:inconnect' from test_rbhive.rb:3:in `

'

Other information:

cat /etc/redhat-release CentOS release 5.4 (Final)

Let me know if there is any other information that will be helpful.

forward-bot commented 12 years ago

Hi Sal,

Thanks for the update. We're about to upgrade our cluster to the latest release of CDH in the coming days. If we get the same issue with RBHive then we will release a fix. At the moment we are not able to reproduce the problem therefore are not able to diagnose and fix the bug.

Unfortunately I'm not able to give you any timeframes on when this will happen but will send you an update when it does.

Thanks

Abhinay Mehta

On 29 Jun 2012, at 18:56, Sal Fuentes wrote:

I tried the same query in the Hive CLI on the same Hive server the test script connects to with the following results:

hive> select state_code from zip_code limit 3; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201206270101_16742, Tracking URL = http://jobtracker-prod.server.addr:50030/jobdetails.jsp?jobid=job_201206270101_16742 Kill Command = /home/t/hadoop_current/bin/hadoop job -Dmapred.job.tracker=jobtracker-prod.server.addr:9001 -kill job_201206270101_16742 2012-06-29 17:42:33,427 Stage-1 map = 0%, reduce = 0% 2012-06-29 17:42:39,475 Stage-1 map = 33%, reduce = 0% 2012-06-29 17:42:42,508 Stage-1 map = 100%, reduce = 0% 2012-06-29 17:42:45,531 Stage-1 map = 100%, reduce = 100% Ended Job = job_201206270101_16742 OK NY NY NY Time taken: 20.205 seconds hive> exit;

I also tweaked the script and ran it again:

bash-3.2$ cat test_rbhive.rb require 'rbhive'

results = RBHive.connect('tron.ev1.yellowpages.com') do |connection|

connection.fetch "select state_code from zip_code"

connection.fetch "describe zip_code" end

results.each do |data| puts "data = #{data.inspect}" end

bash-3.2$ ruby test_rbhive.rb Connecting to hive.server.addr on port 10000 Executing Hive Query: describe zip_code/home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:26:in recv_execute': Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask (HiveServerException) from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/thrift/thrift_hive.rb:17:inexecute' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:140:in execute_unsafe' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:81:inblock in fetch' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in block in safe' from <internal:prelude>:10:insynchronize' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:145:in safe' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:80:infetch' from test_rbhive.rb:5:in block in <main>' from /home/someuser/.rvm/gems/ruby-1.9.2-p290@somegemset/gems/rbhive-0.2.94/lib/rbhive/connection.rb:14:inconnect' from test_rbhive.rb:3:in `

'

Other information:

cat /etc/redhat-release CentOS release 5.4 (Final)

Let me know if there is any other information that will be helpful.

Reply to this email directly or view it on GitHub: https://github.com/forward/rbhive/issues/2#issuecomment-6670356

andytinycat commented 11 years ago

Closed as we can't reproduce.