search

RevolutionAnalytics / RHadoop

RHadoop
https://github.com/RevolutionAnalytics/RHadoop/wiki
763 stars 278 forks source link

Streaming command failed - package rmr #122

Closed alugit closed 12 years ago

alugit commented 12 years ago

Hi,

I am new to RHadoop (used plain R before). I have setup a single node Hadoop cluster (v 1.0.3) on Ubuntu (Narwhal) and installed the R (2.15) and RHadoop packages rmr, rhdfs and rhive. When I try to run the first tutorial program I run into errors. A snippet of my program and error is given below. I am running this from Rstudio , but running it via R also gives the same error:

small.ints = to.dfs(1:1000) Warning: $HADOOP_HOME is deprecated.

12/08/07 11:47:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library 12/08/07 11:47:57 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 12/08/07 11:47:57 INFO compress.CodecPool: Got brand-new compressor

mapreduce(input=small.ints, map=function(k,v) keyval(v, v^2)) Warning: $HADOOP_HOME is deprecated.

packageJobJar: [/tmp/RtmphSrUsJ/rmr-local-env, /tmp/RtmphSrUsJ/rmr-global-env, /tmp/RtmphSrUsJ/rhstr.map6a4401ebc24, /tmp/hadoop-chitra/hadoop-unjar247643577887623940/] [] /tmp/streamjob8964961053266911444.jar tmpDir=null 12/08/07 11:48:42 INFO mapred.FileInputFormat: Total input paths to process : 1 12/08/07 11:48:42 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-chitra/mapred/local] 12/08/07 11:48:42 INFO streaming.StreamJob: Running job: job_201208071147_0001 12/08/07 11:48:42 INFO streaming.StreamJob: To kill this job, run: 12/08/07 11:48:42 INFO streaming.StreamJob: /home/chitra/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201208071147_0001 12/08/07 11:48:42 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_201208071147_0001 12/08/07 11:48:43 INFO streaming.StreamJob: map 0% reduce 0% 12/08/07 11:49:20 INFO streaming.StreamJob: map 100% reduce 100% 12/08/07 11:49:20 INFO streaming.StreamJob: To kill this job, run: 12/08/07 11:49:20 INFO streaming.StreamJob: /home/chitra/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201208071147_0001 12/08/07 11:49:20 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_201208071147_0001 12/08/07 11:49:20 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201208071147_0001_m_000000 12/08/07 11:49:20 INFO streaming.StreamJob: killJob... Streaming Command Failed! Error in mr(map = map, reduce = reduce, combine = combine, in.folder = if (is.list(input)) { : hadoop streaming failed with error code 1

When I examine the stderr logs it shows:

Error in library(rmr) : there is no package called ‘rmr’ Execution halted java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249)

The sys log shows:

2012-08-08 15:13:19,720 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2012-08-08 15:13:19,762 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/job.jar <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/job.jar 2012-08-08 15:13:19,765 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rmr-local-env <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rmr-local-env 2012-08-08 15:13:19,767 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/org <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/org 2012-08-08 15:13:19,770 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/META-INF <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/META-INF 2012-08-08 15:13:19,772 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rmr-global-env <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rmr-global-env 2012-08-08 15:13:19,774 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rhstr.map3dc9720e258c <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rhstr.map3dc9720e258c 2012-08-08 15:13:19,777 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/.job.jar.crc <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/.job.jar.crc 2012-08-08 15:13:19,779 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rhstr.reduce3dc943c68b69 <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rhstr.reduce3dc943c68b69 2012-08-08 15:13:19,889 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2012-08-08 15:13:19,960 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2012-08-08 15:13:19,967 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@28db23f1 2012-08-08 15:13:20,056 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2012-08-08 15:13:20,056 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2012-08-08 15:13:20,058 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2012-08-08 15:13:20,063 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100 2012-08-08 15:13:20,102 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720 2012-08-08 15:13:20,102 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2012-08-08 15:13:20,125 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/usr/bin/Rscript, rhstr.map3dc9720e258c] 2012-08-08 15:13:20,158 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s] 2012-08-08 15:13:20,159 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s] 2012-08-08 15:13:20,249 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done 2012-08-08 15:13:20,250 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed! 2012-08-08 15:13:20,270 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2012-08-08 15:13:20,310 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds. 2012-08-08 15:13:20,310 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName chitra for UID 1000 from the native implementation 2012-08-08 15:13:20,313 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 2012-08-08 15:13:20,316 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

What is wrong here? I see that the package rmr is correctly installed under the directory for R 2.15 and has execute permissions for all users. Thanks, CP

piccolbo commented 12 years ago

Given what you observed, you could have two versions of R installed. There is very little that is rmr specific about running R in rmr: it uses Rscript, and it runs as a hadoop specific user (exact name depends on Hadoop version). So I think you performed exactly the right checks, therefore it has to be something a bit more subtle and complicated than, say, having installed rmr under your home.

Antonio

On Wed, Aug 8, 2012 at 1:59 PM, alugit notifications@github.com wrote:

Hi,

I am new to RHadoop (used plain R before). I have setup a single node Hadoop cluster (v 1.0.3) on Ubuntu (Narwhal) and installed the R (2.15) and RHadoop packages rmr, rhdfs and rhive. When I try to run the first tutorial program I run into errors. A snippet of my program and error is given below. I am running this from Rstudio , but running it via R also gives the same error:

small.ints = to.dfs(1:1000) Warning: $HADOOP_HOME is deprecated.

12/08/07 11:47:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library 12/08/07 11:47:57 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 12/08/07 11:47:57 INFO compress.CodecPool: Got brand-new compressor

mapreduce(input=small.ints, map=function(k,v) keyval(v, v^2)) Warning: $HADOOP_HOME is deprecated.

packageJobJar: [/tmp/RtmphSrUsJ/rmr-local-env, /tmp/RtmphSrUsJ/rmr-global-env, /tmp/RtmphSrUsJ/rhstr.map6a4401ebc24, /tmp/hadoop-chitra/hadoop-unjar247643577887623940/] [] /tmp/streamjob8964961053266911444.jar tmpDir=null 12/08/07 11:48:42 INFO mapred.FileInputFormat: Total input paths to process : 1 12/08/07 11:48:42 INFO streaming.StreamJob: getLocalDirs(): [/tmp/hadoop-chitra/mapred/local] 12/08/07 11:48:42 INFO streaming.StreamJob: Running job: job_201208071147_0001 12/08/07 11:48:42 INFO streaming.StreamJob: To kill this job, run: 12/08/07 11:48:42 INFO streaming.StreamJob: /home/chitra/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201208071147_0001 12/08/07 11:48:42 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_201208071147_0001 12/08/07 11:48:43 INFO streaming.StreamJob: map 0% reduce 0% 12/08/07 11:49:20 INFO streaming.StreamJob: map 100% reduce 100% 12/08/07 11:49:20 INFO streaming.StreamJob: To kill this job, run: 12/08/07 11:49:20 INFO streaming.StreamJob: /home/chitra/hadoop-1.0.3/libexec/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201208071147_0001 12/08/07 11:49:20 INFO streaming.StreamJob: Tracking URL: http://localhost.localdomain:50030/jobdetails.jsp?jobid=job_201208071147_0001 12/08/07 11:49:20 ERROR streaming.StreamJob: Job not successful. Error: # of failed Map Tasks exceeded allowed limit. FailedCount: 1. LastFailedTask: task_201208071147_0001_m_000000 12/08/07 11:49:20 INFO streaming.StreamJob: killJob... Streaming Command Failed! Error in mr(map = map, reduce = reduce, combine = combine, in.folder = if (is.list(input)) { : hadoop streaming failed with error code 1

When I examine the stderr logs it shows:

Error in library(rmr) : there is no package called ‘rmr’ Execution halted java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249)

The sys log shows:

2012-08-08 15:13:19,720 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2012-08-08 15:13:19,762 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/job.jar <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/job.jar 2012-08-08 15:13:19,765 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rmr-local-env <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rmr-local-env 2012-08-08 15:13:19,767 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/org <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/org 2012-08-08 15:13:19,770 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/META-INF <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/META-INF 2012-08-08 15:13:19,772 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rmr-global-env <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rmr-global-env 2012-08-08 15:13:19,774 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rhstr.map3dc9720e258c <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rhstr.map3dc9720e258c 2012-08-08 15:13:19,777 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/.job.jar.crc <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/.job.jar.crc 2012-08-08 15:13:19,779 INFO org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating symlink: /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/jars/rhstr.reduce3dc943c68b69 <- /tmp/hadoop-chitra/mapred/local/taskTracker/chitra/jobcache/job_201208071147_0004/attempt_201208071147_0004_m_000000_0/work/rhstr.reduce3dc943c68b69 2012-08-08 15:13:19,889 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2012-08-08 15:13:19,960 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0 2012-08-08 15:13:19,967 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@28db23fhttps://github.com/org.apache.hadoop.util.LinuxResourceCalculatorPlugin/RHadoop/commit/28db23f1 2012-08-08 15:13:20,056 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2012-08-08 15:13:20,056 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor 2012-08-08 15:13:20,058 INFO org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2012-08-08 15:13:20,063 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 100 2012-08-08 15:13:20,102 INFO org.apache.hadoop.mapred.MapTask: data buffer = 79691776/99614720 2012-08-08 15:13:20,102 INFO org.apache.hadoop.mapred.MapTask: record buffer = 262144/327680 2012-08-08 15:13:20,125 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed exec [/usr/bin/Rscript, rhstr.map3dc9720e258c] 2012-08-08 15:13:20,158 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=1/0/0 in:NA [rec/s] out:NA [rec/s] 2012-08-08 15:13:20,159 INFO org.apache.hadoop.streaming.PipeMapRed: R/W/S=10/0/0 in:NA [rec/s] out:NA [rec/s] 2012-08-08 15:13:20,249 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread done 2012-08-08 15:13:20,250 INFO org.apache.hadoop.streaming.PipeMapRed: PipeMapRed failed! 2012-08-08 15:13:20,270 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1 2012-08-08 15:13:20,310 INFO org.apache.hadoop.io.nativeio.NativeIO: Initialized cache for UID to User mapping with a cache timeout of 14400 seconds. 2012-08-08 15:13:20,310 INFO org.apache.hadoop.io.nativeio.NativeIO: Got UserName chitra for UID 1000 from the native implementation 2012-08-08 15:13:20,313 WARN org.apache.hadoop.mapred.Child: Error running child java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 1 at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:362) at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:576) at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:135) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:36) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 2012-08-08 15:13:20,316 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

What is wrong here? I see that the package rmr is correctly installed under the directory for R 2.15 and has execute permissions for all users. Thanks, CP

— Reply to this email directly or view it on GitHubhttps://github.com/RevolutionAnalytics/RHadoop/issues/122.

alugit commented 12 years ago

I had old R versions which I have removed, created a 'hadoop' user (My hadoop version is 1.0.3 and did not find any specific user name for this version so resorted to hadoop) and run the commands as hadoop user but still same result. The only thing I see is that when I install the required rmr package it seems to do so under the home directory for hadoop user. It does find rmr though - else the command would not even execute to this stage. So what is the issue here then?

piccolbo commented 12 years ago

Try to reinstall rmr in a system directory rather than a user specific directory. It could very well be that when R is started as part of Hadoop its libPaths() is more limited than when used interactively. This fixed it for other users, even when we didn't really understand why the other way did not work.

Antonio

On Thu, Aug 9, 2012 at 1:25 PM, alugit notifications@github.com wrote:

I had old R versions which I have removed, created a 'hadoop' user (My hadoop version is 1.0.3 and did not find any specific user name for this version so resorted to hadoop) and run the commands as hadoop user but still same result. The only thing I see is that when I install the required rmr package it seems to do so under the home directory for hadoop user. It does find rmr though - else the command would not even execute to this stage. So what is the issue here then?

— Reply to this email directly or view it on GitHubhttps://github.com/RevolutionAnalytics/RHadoop/issues/122#issuecomment-7625903.

alugit commented 12 years ago

Thank you! Yes, the problem was indeed that the rmr package (and other dependant packages) needed to be installed in the system directory rather than the user specific directory - Now onto some serious R hadoop programming ...

ssuman commented 12 years ago

Hi, I'm new to RHadoop..I installed rmr and its dependents packages and i'm facing similar issue..I went through the entire communication chain but didn't understand what you meant by system directory..how do I install it in system directory? I do see /home/hduser/R

Regards, S Suman Bharadwaj

piccolbo commented 12 years ago

Hi Suman, I think in general it is better if you open your own issue just because issues have a life cycle from open to closed and if you want to refer to an old issue you just write # and the number and that's enough to generate a link to the old issue and a reference back too. That said, if you don't know what a system directory is it may be advisable to get the help of somebody with more system administration experience. A directory that starts with /home is unlikely to be a good candidate in any unix-like system. Things though are not so clear, on the mac /usr is a system directory, /Users is the equivalent of /home on linux, more or less. I guess I would call .libPaths() from the R prompt and make an educated guess or go by trial and error with the directories listed there. I hope this helps

Antonio

On Fri, Oct 12, 2012 at 8:11 AM, ssuman notifications@github.com wrote:

Hi, I'm new to RHadoop..I installed rmr and its dependents packages and i'm facing similar issue..I went through the entire communication chain but didn't understand what you meant by system directory..how do I install it in system directory? I do see /home/hduser/R

Regards, S Suman Bharadwaj

— Reply to this email directly or view it on GitHubhttps://github.com/RevolutionAnalytics/RHadoop/issues/122#issuecomment-9379556.