Dynamic dispatch of a list of command-line tasks, locally or on a cluster. Supports retrying failed tasks, and adding/removing compute resources on-the-fly.
This PR would move all of the disBatch.py logic into disbatch/disBatch.py. The actual script is then just
from disbatch import disBatch
disBatch.main()
I have also moved the kvsstcp submodule into the disbatch directory, which simplifies path finding. In something like the module system, we were setting PYTHONPATH to have both the disBatch path and the kvsstcp path, while this keeps everything tidy in one place. It causes no issues with the strategy of "clone and run" (you actually don't even have to set DISBATCH_ROOT anymore), nor does it cause issues with pip.
Given the simplified pathing, a lot of the path searching logic can probably be cleaned up.
We also kept the name disBatch.py in the root, which means this doesn't look like a file move, so the git diff is massive, which might be undesirable. It's hard to see what I actually changed. I made a main() function. I had to make the logger and DbUtilPath variables global (since they were being scoped out as local variables when moving the logic to a main() function). No other changes other than a bit of import logic.
Also, this is more of a proof of concept PR, and I'm happy to change it as needed.
blackwer
commented
2 years ago
This PR would move all of the
disBatch.pylogic intodisbatch/disBatch.py. The actual script is then justI have also moved the
kvsstcpsubmodule into thedisbatchdirectory, which simplifies path finding. In something like the module system, we were settingPYTHONPATHto have both thedisBatchpath and thekvsstcppath, while this keeps everything tidy in one place. It causes no issues with the strategy of "clone and run" (you actually don't even have to setDISBATCH_ROOTanymore), nor does it cause issues with pip.Given the simplified pathing, a lot of the path searching logic can probably be cleaned up.
We also kept the name
disBatch.pyin the root, which means this doesn't look like a file move, so the git diff is massive, which might be undesirable. It's hard to see what I actually changed. I made amain()function. I had to make theloggerandDbUtilPathvariables global (since they were being scoped out as local variables when moving the logic to amain()function). No other changes other than a bit of import logic.Also, this is more of a proof of concept PR, and I'm happy to change it as needed.