piccolbo
commented
13 years ago Hi and thanks for your report. Rhipe and rmr are very different in the API and the guts of the implementation, so watch out. Probably better to start mentally from your knowledge of R and Hadoop and not think of Rhipe at all. One problem in both versions of your program is that in rmr the parsing of the input is abstracted away, so that you don't have to repeat the same boilerplate code in every mapper. The goal is to have map(k,v) get two legitimate R objects as arguments. To do that, we use a default format, which is <json object>\t<json object>\n. Of course that doesn't cut it for everyone, so you can specify a parsing function as a parameter to mapreduce, textinputformat. That function takes a line of text and returns a key-value pair. Since you did not specify that function, the input has to be in the default format and that's probably the beginning of your problems -- haven't checked the rest of the code, one thing at a time. The good news is, I added support for csv in dev yesterday. Probably buggy but I'd love some help in getting it in good shape for release and you wouldn't have to invent the wheel again. You need to build the package from src if you go down that route. So you'd just do
mapreduce(your other args here, textinputformat = csvtextinputformat(options describing your specific csv dialect here, as in read.table))Also I would recommend (beg for?) two things: first read the manual. We are trying very hard to keep it up to date and usable and it is an immense help if people read it and help me improve it rather than retyping select parts of the manual as replies here. People think that to help an open source project they have to understand i-node internals, but the fact is just giving feedback on the manual is so important while maybe unglamorous. The other thing, I know you are after your problem, but it seems to me you'd want to build up to that through simpler jobs, like: mapreduce(in, out, textinputformat = csvtextinputformat(...)), default mapper and reducers and default everything else. This will read your input and write it out unchanged, only going from csv to json. Not very useful but you make sure the parsing is right, check, move on to learn how mappers work, check that works, and so on. At least, that's what works for me. If you are familiar with google Hangout we can have a pair programming session (you need to be confortable with screen sharing of course). Contact me as +piccolbo on google plus and we'll take it from there.
kagharpure
RevolutionAnalytics
Hi all,
Firstly, I'd like to apologize for not sticking to gfm conventions: it's almost midnight, my brain is nowhere near the functional level required to figure something new out and I'm posting here for the first time. Now:
I am somewhat familiar with R (~1.5 years), fairly new to Hadoop and very new to RHadoop. I've recently been trying to run a couple of simple linear regression codes written using RHadoop; however, I seem to be running repeatedly into what seems to be a hadoop streaming issue. Before I jump into it, I shall explain the codes and where I'm running them a little.
Both of the codes mentioned above expect an input csv on the HDFS with no headers and only containing values for the predictor variables and the dependent variable, which must be the last variable (counting left to right)
Since I'm not entirely sure yet of the way RHadoop works, I wrote both codes with different assumptions on RHadoop's working. One code is based on the assumption that text files will be processed line-by-line by the mapper and the second one is a logical rip-off from an equivalent Rhipe code, where the assumption is that for each mapper there will exist (magically) a list of values (a la map.values in Rhipe).
First code (assuming mapper processes line-by-line):
Second code (assuming mappers process lists of values):
I've tried running these codes both on a 14-node hadoop cluster, with nodes running hadoop-0.20.2.append on either Ubuntu Lucid or Maverick, and a single node installation, running on Ubuntu Lucid which itself runs on VMPlayer on Windows 7.
Now, upon sourcing either of these two codes in R, I repeatedly keep running into the following error:
Further exploring the trace on the jobtracker url, this is what I find (for each attempt):
While this trace seems somewhat common on Google, I cannot seem to find anything specifically about it in an RHadoop context.
Additionally, I've also been observing some strange behavior from rhdfs.
This set of commands run fine:
However, when I try the same trick with a file already on the HDFS (put there previously using hadoop fs -copyFromLocal), I seem to run into a strange error:
I initially suspected that this might have something to do with permissions and I was partially right; when I tried the first set of commands as root, I got the following error:
The other set of commands also gave the same error as the first when run as root:
Still clinging to the idea that the issue may be due to permissions, I checked the permissions on the files involved in both sets of commands. However, much to my chagrin, they were both the same (discovered using hadoop fs -ls):
I have now come to one of the following four conclusions:
1) This is due to a very stupid error on my part 2) This is due to some incompatibility issues between hadoop and RHadoop 3) I should be specifying some parameters that I am not (a la the mapred list in Rhipe) 4) None of the above, this is just plain beyond me
Therefore, I now find myself posting here what's ultimately quite a long post, in the hope that someone will know exactly (or at least vaguely) what's going wrong and point me in the right direction.
Thanks,
Krish.