user can specify paths of directories and files (previously just directories)
user can also specify paths via a file where each line is a path (previously all paths had to be command-line arguments)
REST POSTing is done in real-time (previously this was done after all metadata collecting finished)
smarter multiprocessing: user can specify the number of processes to utilize. the file indexing is internally load-balanced, ie. each process will make nearly the same number of REST POSTs each
user can define the logging level
user can specify paths to skip via a file containing paths (--blacklist-file)
directory_scanner.py
new script for getting a list of all files in a path
file paths are outputted to stdout
the number of files and any errors are logged to stderr
indexer_make_dag.py
added 'Everything' level to index all of /data/exp/
directory_scanner.py is used
many subprocess.check_call()'s get that list of paths to sorted order and split into files with 10000 lines/paths each, each file to be processed by a condor job
this staging work is done in /data/user/{user}/indexer-data-exp/
the whole list of paths is placed in a file for future reference, see below
using the --previous-data-exp option (with a previously-generated file of paths) will only index any new files
Overview:
indexer.py
directory_scanner.py
indexer_make_dag.py