kentdlee
commented
1 month ago i am away through next week. Can we talk when i get back? if not we can have you talk to another team member. sounds like we should talk
On Fri, Aug 2, 2024 at 12:33 PM Tom Nicholas @.***> wrote:
I would like to use dragon as within the context of a larger piece of python software (Cubed https://github.com/cubed-dev/cubed). In particular I want to write an equivalent to Cubed's ProcessesExecutor https://github.com/cubed-dev/cubed/blob/19f844e2c1ea799ae4bb0cb754f7aa92f917e598/cubed/runtime/executors/local.py#L266 (or ThreadsExecutor https://github.com/cubed-dev/cubed/blob/19f844e2c1ea799ae4bb0cb754f7aa92f917e598/cubed/runtime/executors/local.py#L227) but which uses Dragon as the concurrent_executor https://github.com/cubed-dev/cubed/blob/19f844e2c1ea799ae4bb0cb754f7aa92f917e598/cubed/runtime/executors/local.py#L176 .
All this executor needs to do is execute a series of stages each made up of a number of embarrasingly parallel tasks (each of which are python functions). I just want Dragon to launch the tasks in parallel for me across a whole HPC allocation.
I'm looking through the docs and I have two main questions:
1) Should I use the dragon.workflows.parsl_executor.DragonPoolExecutor https://dragonhpc.github.io/dragon/doc/_build/html/ref/workflows/dragon.workflows.parsl_executor.html#dragon.workflows.parsl_executor.DragonPoolExecutor?
That seems like a drop-in replacement, but if that's actually using Parsl https://parsl-project.org/ (which I noticed got built when I built the dragon executable) and that's all I want to use then would I be better off not bothering with Dragon and just using the parsl.executors.ThreadPoolExecutor https://parsl.readthedocs.io/en/stable/stubs/parsl.executors.ThreadPoolExecutor.html#parsl-executors-threadpoolexecutor instead? What's the difference?
2) How do I launch dragon from within the context of another python program?
All the docs examples seem to say that you use dragon to launch another python program from the command line like this
dragon my_python_script.py
, and dragon works by
replacing all standard Multiprocessing classes with Dragon equivalent classes before CPython resolves the inheritance tree.
(from Inheritance and Multiple Start Methods https://dragonhpc.github.io/dragon/doc/_build/html/pguide/dragon_multiprocessing.html#inheritance-and-multiple-start-methods .)
But this is inconvenient if I can't represent my workload as a single standalone python script. Instead I ideally want to be able to call the Executor from inside a running python process on an interactive job (e.g. from within a jupyter notebook cell) and have it execute across a whole allocation.
Do I need to somehow auto-generate this script and make a subprocess.call to the dragon executable?
Or if I try omitting the dragon executable (mentioned on this page https://dragonhpc.github.io/dragon/doc/_build/html/pguide/dragon_multiprocessing.html#multiprocessing-and-dragon-without-patching) then I'm not sure what this implies:
The Dragon core library can still be imported via e.g. from dragon.managed_memory import MemoryPool and used. In this case, the "dragon" start method must not be set. The infrastructure will not be started.
This part:
Note that all other parts of the Dragon stack, in particular the Dragon Native https://dragonhpc.github.io/dragon/doc/_build/html/ref/native/index.html#dragon-native API require the running Dragon infrastructure and are thus not supported without patching Multiprocessing.
seems to be saying that I can still use Dragon Core but not Dragon Native from within a python program that I didn't launch using the dragon executable. Is the dragon.workflows.parsl_executor.DragonPoolExecutor https://dragonhpc.github.io/dragon/doc/_build/html/ref/workflows/dragon.workflows.parsl_executor.html#dragon.workflows.parsl_executor.DragonPoolExecutor in Dragon Core or Dragon Native?
@applio https://github.com/applio you said you
got dragon to run the add-asarray.py example https://github.com/cubed-dev/cubed/blob/main/examples/add-asarray.py single node as the executor already
so I'm curious what your approach was?
cc @tomwhite https://github.com/tomwhite
— Reply to this email directly, view it on GitHub https://github.com/DragonHPC/dragon/issues/20, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM7WSZDFWEPS53VNO6YBPTZPPGIXAVCNFSM6AAAAABL5AUUCGVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQ2DKNJUGY2TQNA . You are receiving this because you are subscribed to this thread.Message ID: @.***>
applio
TomNicholas
I would like to use dragon as within the context of a larger piece of python software (Cubed - see https://github.com/cubed-dev/cubed/issues/467). In particular I want to write an equivalent to Cubed's
ProcessesExecutor(orThreadsExecutor) but which uses Dragon as theconcurrent_executor.All this executor needs to do is execute a series of stages each made up of a number of embarrasingly parallel tasks (each of which are python functions). I just want Dragon to launch the tasks in parallel for me across a whole HPC allocation.
I'm looking through the docs and I have two main questions:
1) Should I use the
dragon.workflows.parsl_executor.DragonPoolExecutor?That seems like a drop-in replacement, but if that's actually using Parsl (which I noticed got built when I built the
dragonexecutable) and that's all I want to use then would I be better off not bothering with Dragon and just using theparsl.executors.ThreadPoolExecutorinstead? What's the difference?2) How do I launch dragon from within the context of another python program?
All the docs examples seem to say that you use dragon to launch another python program from the command line like this
, and dragon works by
(from Inheritance and Multiple Start Methods.)
But this is inconvenient if I can't represent my workload as a single standalone python script. Instead I ideally want to be able to call the Executor from inside a running python process on an interactive job (e.g. from within a jupyter notebook cell) and have it execute across a whole allocation.
Do I need to somehow auto-generate this script and make a
subprocess.callto thedragonexecutable?Or if I try omitting the
dragonexecutable (mentioned on this page) then I'm not sure what this implies:This part:
seems to be saying that I can still use Dragon Core but not Dragon Native from within a python program that I didn't launch using the
dragonexecutable. Is thedragon.workflows.parsl_executor.DragonPoolExecutorin Dragon Core or Dragon Native?@applio you said you
so I'm curious what your approach was?
cc @tomwhite