runMe has a data member ableToRun that starts out False and, in the runMe constructor, is set to True if the process can run, and it is not up-to-date. It doesn't distinguish between "can't run" and "could run, but it's up to date, so don't bother." This has two problems. One is that with normal ProcessQueue running, the logs can be misleading--it's hard to tell why the file didn't run. The other is that DBRunner basically ignores ableToRun unless using the -u option. This results in it trying to fire off scripts that just plain can't run.
So this probably needs to be updated to ableToRun and needsUpdate or something similar, with nothing running unless ableToRun is True, DBRunner either checking or ignoring needsUpdate based on its needs (-u or not), and ProcessQueue always also needing needsUpdate.
Minimal example to reproduce issue:
This is pretty tough to nail down and a unit test will be part of solving the issue. @balarsen ran into it when trying to use DBRunner when there was no active code for the desired process (active_code was False). That gave the traceback below. It would also potentially manifest in using ProcessQueue and getting the wrong reason for not running a new file.
Error message/Traceback:
When using DBRunner with no active code:
$ ~/dbUtils/DBRunner.py -m ~/PROCESSING_DB/mageis_rbspa_2017.sqlite -s 20130101 -e 20130101 1
Traceback (most recent call last):
File "/home/ectsoc/dbUtils/DBRunner.py", line 176, in <module>
rundir=None if options.ingest else '.')
File "/home/ectsoc/.local/lib/python2.7/site-packages/dbprocessing/runMe.py",
line 163, in runner
runme.make_command_line(force = force, rundir=rundir)
File "/home/ectsoc/.local/lib/python2.7/site-packages/dbprocessing/runMe.py",
line 657, in make_command_line
cmdline = [self.codepath]
AttributeError: 'runMe' object has no attribute 'codepath'
OS, Python version, and dependency version information:
Commit merged with tests for distinguishing between unable-to-run and up-to-date, in both ProcessQueue and DBRunner contexts, with the appropriate action and error message defined. And said tests passing.
runMehas a data memberableToRunthat starts outFalseand, in therunMeconstructor, is set toTrueif the process can run, and it is not up-to-date. It doesn't distinguish between "can't run" and "could run, but it's up to date, so don't bother." This has two problems. One is that with normalProcessQueuerunning, the logs can be misleading--it's hard to tell why the file didn't run. The other is thatDBRunnerbasically ignoresableToRununless using the-uoption. This results in it trying to fire off scripts that just plain can't run.So this probably needs to be updated to
ableToRunandneedsUpdateor something similar, with nothing running unlessableToRunisTrue,DBRunnereither checking or ignoringneedsUpdatebased on its needs (-uor not), andProcessQueuealways also needingneedsUpdate.Minimal example to reproduce issue:
This is pretty tough to nail down and a unit test will be part of solving the issue. @balarsen ran into it when trying to use
DBRunnerwhen there was no active code for the desired process (active_codewasFalse). That gave the traceback below. It would also potentially manifest in usingProcessQueueand getting the wrong reason for not running a new file.Error message/Traceback:
When using
DBRunnerwith no active code:OS, Python version, and dependency version information:
Version of dbprocessing
git master (142a4976f66d28d227e95ae6acfc73376adc660f)
Closure condition
Commit merged with tests for distinguishing between unable-to-run and up-to-date, in both ProcessQueue and DBRunner contexts, with the appropriate action and error message defined. And said tests passing.