Closed dhuy237 closed 3 years ago
I found a workaround solution but I hope there will be a better way of doing this.
I have to define the argument again inside the mrjob
class so it can recognize it:
from datetime import datetime
import json
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("-t", "--time", help = "Output file")
# Use parse_known_args() to ignore all the arguments for mrjob class
args, unknown = parser.parse_known_args()
class Calculate(MRJob):
def configure_args(self):
super(Calculate, self).configure_args()
# Define the argument again to ignore the argparse when running the MR job
self.add_passthru_arg("-t", "--time", help="output folder for time")
start_time = datetime.now()
Calculate.run()
execute_time = (datetime.now() - start_time).total_seconds()
data = {}
data["step1"] = execute_time
with open(args.time+'/time.json', 'w') as outfile:
json.dump(data, outfile)
And run with this command:
python cal.py data/input/input.txt --output data/output --time data/output
My question on Stack Overflow
Normally, if I want to define a command-line option for mrjob, I have to do like this:
When I want to use the argument, I just need to call
self.options.time
but it only works inside that class.I want to track the time of the
mrjob
and write the time into thejson
file like thiscal.py
:When I run with this command:
I got this error:
Then I found an answer about using
parse_known_args()
and I tried it:Now, I got a new error. I believe this is from
mrjob
because I did remove theargparse
and run the same command, it returns exactly like this:How can I define an argument without affecting the
mrjob
class?