increase indexer performance for indexing all of /data/exp/

[ric-evans]

Overview:

indexer.py

• user can specify paths of directories and files (previously just directories)
• user can also specify paths via a file where each line is a path (previously all paths had to be command-line arguments)
• REST POSTing is done in real-time (previously this was done after all metadata collecting finished)
• smarter multiprocessing: user can specify the number of processes to utilize. the file indexing is internally load-balanced, ie. each process will make nearly the same number of REST POSTs each
• user can define the logging level
• user can specify paths to skip via a file containing paths (--blacklist-file)

directory_scanner.py

• new script for getting a list of all files in a path
• file paths are outputted to stdout
• the number of files and any errors are logged to stderr

indexer_make_dag.py

• added 'Everything' level to index all of /data/exp/
  • directory_scanner.py is used
  • many subprocess.check_call()'s get that list of paths to sorted order and split into files with 10000 lines/paths each, each file to be processed by a condor job
  • this staging work is done in /data/user/{user}/indexer-data-exp/
  • the whole list of paths is placed in a file for future reference, see below
  • using the --previous-data-exp option (with a previously-generated file of paths) will only index any new files
• added dry-run argument for testing

[ric-evans]

It mostly looks good. Just some small suggestions.

As for the future, this is starting to get large enough we may want to place it into the actual file catalog code, maybe as a client.

Noted.

[dsschult]

looks good to me