Yelp / mrjob

Run MapReduce jobs on Hadoop or Amazon Web Services
http://packages.python.org/mrjob/
Other
2.61k stars 587 forks source link

running on hadoop mode error: Unrecognized option -D #2126

Open yxysss opened 4 years ago

yxysss commented 4 years ago
from mrjob.job import MRJob
from mrjob.step import MRStep
import re
import numpy as np

WORD_RE = re.compile(r"[\w']+")

class WordCount(MRJob):

    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
            yield (word.lower(), 1)

    def combiner(self, word, counts):
        yield (word, sum(counts))

    def reducer(self, word, counts):
        yield (word, sum(counts))

Here I use a very simple wordcount mapreduce job to test on hadoop mode, But I got an error:

Unrecognized option: -D

Try -help for more information Streaming Command Failed! Usage: $HADOOP_HOME/bin/hadoop jar hadoop-streaming.jar [options]

I ran this job on Windows10.