search

NERSC / slurm-magic

IPython magic for SLURM.
Other
67 stars 15 forks source link

Non-blocking calls #7

Open muammar opened 4 years ago

muammar commented 4 years ago

Although Popen is being used to execute commands:

https://github.com/NERSC/slurm-magic/blob/4c708cc137cb9f4bd5b44cf26837b466d9bf7b65/slurm_magic.py#L164

.communicate() is a blocking process:

https://github.com/NERSC/slurm-magic/blob/4c708cc137cb9f4bd5b44cf26837b466d9bf7b65/slurm_magic.py#L165

Is it possible to change this so that capturing subprocess.PIPE does not block?

muammar commented 4 years ago

https://github.com/muammar/slurm-magic/commit/afaa2218b4cd710044dd46cbfe55196ded11d29d represents a small hack for this. A better way would be something like what's described here:

https://stackoverflow.com/questions/375427/non-blocking-read-on-a-subprocess-pipe-in-python

I will keep working on that.

rcthomas commented 4 years ago

Hi @muammar, thanks for your interest. I've been wondering if anyone uses this package, because it's barely more functionality than the IPython kernel provides by default. You can background processes with the built in %%script magic. As you can see in the code you've already looked at it's really not doing anything "magic" here, just passing Slurm commands through to the command line.