I tried to run a simple MapReduce job locally and it worked fine, but whenever I run it on hadoop I get this error:
Found 2 unexpected arguments on the command line [hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/mrjob.zip#mrjob.zip, hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/setup-wrapper.sh#setup-wrapper.sh]
Try -help for more information
Here is the entire command:
Step 1 of 3 failed: Command '['C:\\hadoop\\bin\\hadoop.CMD', 'jar', 'C:/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.8.0.jar', '-files', 'hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/MRBaseDiversitySolver.py#MRBaseDiversitySolver.py,hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/mrjob.zip#mrjob.zip,hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/setup-wrapper.sh#setup-wrapper.sh', '-input', 'hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/groups_dh_1.csv', '-output', 'hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/step-output/0000', '-mapper', '/bin/sh -ex setup-wrapper.sh python3 MRBaseDiversitySolver.py --step-num=0 --mapper', '-reducer', '/bin/sh -ex setup-wrapper.sh python3 MRBaseDiversitySolver.py --step-num=0 --reducer']' returned non-zero exit status 1.
It seems to me that hadoop won't allow uploading more that one file using the flag -files.
Have you encountered that? If so, have did you solve it?
I would really appreciate your help since I need this for my Final B.Sc project.
Hey guys,
I tried to run a simple MapReduce job locally and it worked fine, but whenever I run it on hadoop I get this error:
Here is the entire command:
Step 1 of 3 failed: Command '['C:\\hadoop\\bin\\hadoop.CMD', 'jar', 'C:/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.8.0.jar', '-files', 'hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/MRBaseDiversitySolver.py#MRBaseDiversitySolver.py,hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/mrjob.zip#mrjob.zip,hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/wd/setup-wrapper.sh#setup-wrapper.sh', '-input', 'hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/files/groups_dh_1.csv', '-output', 'hdfs:///user/User/tmp/mrjob/MRBaseDiversitySolver.User.20200910.140702.323565/step-output/0000', '-mapper', '/bin/sh -ex setup-wrapper.sh python3 MRBaseDiversitySolver.py --step-num=0 --mapper', '-reducer', '/bin/sh -ex setup-wrapper.sh python3 MRBaseDiversitySolver.py --step-num=0 --reducer']' returned non-zero exit status 1.
It seems to me that hadoop won't allow uploading more that one file using the flag -files.
Have you encountered that? If so, have did you solve it?
I would really appreciate your help since I need this for my Final B.Sc project.
Thanks in advance