search

roryk / ipython-cluster-helper

Tool to easily start up an IPython cluster on different schedulers.
148 stars 23 forks source link

Error in cluster startup for PBSPRO #60

Closed TimothyThomas closed 6 years ago

TimothyThomas commented 6 years ago

When I use scheduler='pbspro' the engines start up fine and do their work, but the ipcluster dies and fails to shutdown the engines (see ipcluster log output below). It seems to be an issue with parsing the job_id returned by subprocess.check_output() inside of BcbioPBSPROEngineSetLauncher (lines 737-739). I believe subprocess.check_output() returns bytes by default, so the output.strip() operation raises an exception since it expects a str. Passing the argument universal_newlines=True will make subprocess.check_output() return a str and should prevent this error.

Anyone else experience this issue? I'm wondering if it could be something else on my end, since it's surprising no one else has reported this.

The ipcluster log file has the following:

2018-04-16 11:35:51.031 [IPClusterStart] ERROR | Engine start failed
Traceback (most recent call last):
  File "/uf5a/trthomas/local/miniconda3/envs/pyra/lib/python3.6/site-packages/ipyparallel/apps/ipclusterapp.py", line 334, in start_engines
    self.engine_launcher.start(self.n)
  File "/uf5a/trthomas/local/miniconda3/envs/pyra/lib/python3.6/site-packages/cluster_helper/cluster.py", line 740, in start
    job_id = ";".join(job_ids)
TypeError: sequence item 0: expected str instance, bytes found
2018-04-16 11:36:16.335 [IPClusterStart] SIGINT received, stopping launchers...
2018-04-16 11:36:16.406 [IPClusterStart] ERROR | IPython cluster: stopping
roryk commented 6 years ago

Thanks Timothy for the awesome bug report. I think this is a python2/3 difference:

[rory:~/tmp/foo] $ python3 -c 'from __future__ import print_function; import subprocess; print(subprocess.check_output(["ls"]));'
b'EXAMPLE\n'
[rory:~/tmp/foo] $ python2 -c 'from __future__ import print_function; import subprocess; print(subprocess.check_output(["ls"]));'
EXAMPLE

I pushed a fix for this that passes the universal_newlines option. Thanks so much!

TimothyThomas commented 6 years ago

Aha! I didn't consider possible differences between Python 2/3. Thanks so much for the fix and your work on this project as a whole.