We have in notes and old code that ProcessQueue.py -p should first call ProcessqueueClean before looping over the process queue entries and calling buildChildren. ProcessQueueClean is supposed to remove duplicate entries in the process queue and also make sure it's sorted by date and level to minimize overlap.
That call was commented out by @balarsen in 2017 with the comment that buildChildren will clean the queue. I'm not seeing this...the only cleaning it does is skipping files which aren't the newest version.
This isn't going to break anything, but might slow down the calculation a bit. It's also a bit confusing, since we've got some contradictory notes.
OS, Python version, and dependency version information:
There's not one obvious path forward here. Should probably restore the clean, or update buildChildren to do it, or decide we don't care, and double-check the docs once that's done. Also should remove the commented code once this is resolved.
We have in notes and old code that
ProcessQueue.py -p
should first callProcessqueueClean
before looping over the process queue entries and callingbuildChildren
.ProcessQueueClean
is supposed to remove duplicate entries in the process queue and also make sure it's sorted by date and level to minimize overlap.That call was commented out by @balarsen in 2017 with the comment that
buildChildren
will clean the queue. I'm not seeing this...the only cleaning it does is skipping files which aren't the newest version.This isn't going to break anything, but might slow down the calculation a bit. It's also a bit confusing, since we've got some contradictory notes.
OS, Python version, and dependency version information:
Version of dbprocessing
Current master
Closure condition
There's not one obvious path forward here. Should probably restore the clean, or update
buildChildren
to do it, or decide we don't care, and double-check the docs once that's done. Also should remove the commented code once this is resolved.