Closed aashishg closed 7 years ago
aashishg
commented
7 years ago Update: I was able to run it by WFDriver.java as entry point. But the output which is part-r-00000 comes out as blank. Any ideas on why is it happening? I have put input csv files in the inputfolder and passed it to hadoop command.
AmruthSkanda
commented
7 years ago Hey
Glad you found this project useful! You're not finding build.XML because this is not ant build. It's a maven project, so you'll have pom.xml. and can you provide more info on your hadoop architecture setup? Is it single node or multi node?
aashishg
commented
7 years ago Ok I tried running with eclipse but not sure if there was maven. Hadoop architecture is single node.
aashishg
commented
7 years ago I am sorry I am a noob at this. I think you have not uploaded whole code? it maybe a mistake in uploading? Actually, I can't find the pom.xml file.
AmruthSkanda
commented
7 years ago You can find pom.xml in this link: https://github.com/romainr/hadoop-tutorials-examples/blob/master/hive-udf/pom.xml OR https://github.com/dpino/Hadoop-Word-Count/blob/master/pom.xml (Whichever works). Place this in your eclipse project dir. Make sure to change project name, artifactid, versions, etc., according to your environment setup. Yes it was mistake not to upload pom and its been a while and I cant find it either. Just place this and from your project dir(where there is pom.xml) run maven build commands like "mvn clean install" to generate your .jar file which you have to supply to your hadoop along with input/output file dirs. You might need to install "mvn" from apt-get/yum (assuming you're on Linux environment) first to run maven commands. for hadoop environment setup you can refer:
Let me know how it goes
aashishg
commented
7 years ago Hey, Sorry for a late reply. Would it be ok if I generate jar without maven in eclipse or with jvc? I did it with eclipse and jvc both and I am getting an empty part-r-00000 I checked the code and I found context.write for writing o/p and hardcoded a srting and a doublewritable in it but I am still getting an empty o/p
aashishg
commented
7 years ago Update: Hey amruth, just finished running this with maven but I still got an empty file output file: part-r-00000
Attaching pom.xml and jar file generated(weather-1.0.jar)
Thanks for the help!
AmruthSkanda
commented
7 years ago It is difficult to debug without understanding your issue properly. There are so many reasons why it can be empty. I'm sure something is wrong with your input data. What is your input data?? send me your sample input data. I'm here attaching a sample file input data of the 26gb actual data which I'd used to run this project.
aashishg
commented
7 years ago Hey Amruth, Sorry for late reply again. I really dont know why github wont notify, lol. Actually I used the data in the sliding window algorithm in R directory in this repo, this folder before
I used the data you provided and the o/p is not blank but I am getting ouput as many
...WFPojo@somehex I think its part-r-00000.zip need to convert WFPojo into text? Attached: part-r-00000
aashishg
commented
7 years ago Update: Hey! Thanks for the file. I got it running :) Also, if possible can you upload R code? I am just asking because I needed to compare it with sliding algo
AmruthSkanda
commented
7 years ago R is entirely different approach and is not big data analytics anymore. I was not responsible for it, So I can't really help you with that! Please contact me through linkedin/FB I shall close this issue here!
aashishg
commented
7 years ago No problem Amruth, Thanks for all your help :D
Not an issue but a request :) Hey amrut, I along with my friends are learning hadoop and mapreduce. We think we could learn alot from this project. Can you please write a quick instruction on running this? We are used to the hadoop make jar and then run approach but we dont know how to make jar in this one as we cant find the build.xml file for ant.
It would be great help for us as this project is close to what we are making.
Thanks!