Add options for preprocessor

chewisinho commented 8 years ago

Add docopt argument parser for the preprocessor. The final goal is:

run without arguments -> read from MongoDB, set up all databases simultaneously (most efficient, used right after we get scraper data)
run with arguments -> read from MongoDB, create specified databases (i.e. maybe we want modifications to the link database but the text database is good, so we run the preprocessor with options to recreate the link database only)

I can work on this weekend (if this needs to be done faster, please assign yourself and work on it instead).

See BIDS-projects/topic-modeling/topic-model/topic-model.py for example usage of docopt.

alvinwan commented 8 years ago

Maybe it'd be better, for the sake of users in the future, to force users to specify a preprocessor. Just in case the default is run on accident? Running without arguments would prompt --help, for example.

chewisinho commented 8 years ago

No problem, I'll do that.

chewisinho commented 8 years ago

The command line options are finished - running without arguments currently does not write to any databases (but does not display help messages). Good enough for now.

BIDS-projects / etl

Add options for preprocessor #8