problem running with plugin PBS on cluster

[venpopov]

I'm following this tutorial to learn nipype: http://miykael.github.io/nipype-beginner-s-guide/firstSteps.html

The preprocessing script works fine on my personal computer: https://github.com/miykael/nipype-beginner-s-guide/blob/master/scripts/tutorial_3_first_steps.py

However I get the following error on a CentOS 6.2 remote cluster:

/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)

Traceback (most recent call last):
  File "/home/vpopov/nipype_tutorial/nipype_tutorial.py", line 125, in <module>
    plugin_args=dict(qsub_args='-l nodes=4:ppn=5'))
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 265, in run
    graph=graph)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 402, in _send_procs_to_workers
    updatehash=updatehash)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 564, in _submit_job
    return self._submit_batchtask(batchscriptfile, node)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/pbs.py", line 100, in _submit_batchtask
    str(e))))

RuntimeError: Could not submit pbs task for node selectfiles.a04
decode() argument 1 must be string, not None
Interface CommandLine failed to run. 

I've changed the workflow run command to execute with the server's PBS scheduling system:

preprocess.run(plugin='PBS', 
               plugin_args=dict(qsub_args='-l nodes=4:ppn=5'))

Also, I've removed the write_graph command and the import of pygraphviz, because I couldn't install graphviz and pygraphviz on the server (I have no root access, maybe it's possible to circumvent this somehow?)

Any help is appreciated.

[satra]

on a cluster you can install an entire nipype+python environment using miniconda (http://conda.pydata.org/miniconda.html) without root access.

thanks for reporting this, hopefully this is fixed in master and you can test it using the local install.

[venpopov]

This looks like a stripped down version of Anaconda, which I already installed, and it doesn't include the pygraphviz or graphviz packages, which I can't install without root access. Can you please elaborate how that will solve my problem?

Let me reiterate that I've install almost everything just fine, with the exception of those two packages for which it says that they are highly recommended, but not required. And that my problem is the following error:

RuntimeError: Could not submit pbs task for node selectfiles.a04
decode() argument 1 must be string, not None
Interface CommandLine failed to run. 

[satra]

@venpopov

re: graphviz: https://anaconda.org/search?q=graphviz

re: pbs error, is this with current master?

[venpopov]

This was after following this installation tutorial: http://miykael.github.io/nipype-beginner-s-guide/installation.html, namely the following commands are related to the nipype installation:

pip install nipype pip install nibabel pip install rdflib pip install nipy pip install dipy pip install --upgrade nipype pip install --upgrade nibabel nipype rdflib nipy dipy

[satra]

that would not install the current master. could you please try:

pip install --upgrade https://github.com/nipy/nipype/archive/master.zip

[venpopov]

Actually yes, it was the current master with this command: pip uninstall nipype; pip install https://github.com/nipy/nipype/archive/master.zip

as suggested here due to another bug: https://github.com/nipy/nipype/issues/1395#issuecomment-193118772

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 10:20 AM, Satrajit Ghosh notifications@github.com wrote:

@venpopov https://github.com/venpopov

re: graphviz: https://anaconda.org/search?q=graphviz

re: pbs error, is this with current master?

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193295581.

[satra]

could you please provide the output from this?

from nipype import get_info
get_info()

[venpopov]

{'nibabel_version': '2.0.2', 'networkx_version': '1.11', 'numpy_version': '1.10.4', 'sys_platform': 'linux2', 'sys_version': '2.7.11 |Anaconda 2.5.0 (64-bit)| (default, Dec 6 2015, 18:08:32) \n[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]', 'commit_source': 'archive substitution', 'commit_hash': '%h', 'pkg_path': '/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype', 'sys_executable': '/data2/rederlab/software/anaconda2/bin/python', 'traits_version': '4.5.0', 'scipy_version': '0.17.0'}

[satra]

oops the one thing that i was looking for was commit_hash and i'm not sure why that didn't show up.

in that case, would you mind doing the pip install --upgrade https://github.com/nipy/nipype/archive/master.zip

and ensure that the error still happens, and post it? i did look at the code and the place where i see the error, the one thing that could possibly help would be to use text_type.

in the pbs.py file, one can try:

diff --git a/nipype/pipeline/plugins/pbs.py b/nipype/pipeline/plugins/pbs.py
index 5265468..0c29867 100644
--- a/nipype/pipeline/plugins/pbs.py
+++ b/nipype/pipeline/plugins/pbs.py
@@ -7,7 +7,7 @@ import subprocess

 from .base import (SGELikeBatchManagerBase, logger, iflogger, logging)

-from nipype.interfaces.base import CommandLine
+from ...interfaces.base import CommandLine, text_type

 class PBSPlugin(SGELikeBatchManagerBase):
@@ -97,7 +97,7 @@ class PBSPlugin(SGELikeBatchManagerBase):
                     iflogger.setLevel(oldlevel)
                     raise RuntimeError('\n'.join((('Could not submit pbs task'
                                                    ' for node %s') % node._id,
-                                                  str(e))))
+                                                  text_type(e))))
             else:
                 break
         iflogger.setLevel(oldlevel)

[venpopov]

I tried to upgrade nipype with this command and I get exactly the same error as above.

Can you explain what I should do with the code you gave me? Am I supposed to run it in the terminal, supplying the path to pbs.py?

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 4:11 PM, Satrajit Ghosh notifications@github.com wrote:

oops the one thing that i was looking for was commit_hash and i'm not sure why that didn't show up.

in that case, would you mind doing the pip install --upgrade https://github.com/nipy/nipype/archive/master.zip

and ensure that the error still happens, and post it? i did look at the code and the place where i see the error, the one thing that could possibly help would be to use text_type.

in the pbs.py file, one can try:

diff --git a/nipype/pipeline/plugins/pbs.py b/nipype/pipeline/plugins/pbs.py index 5265468..0c29867 100644 --- a/nipype/pipeline/plugins/pbs.py +++ b/nipype/pipeline/plugins/pbs.py @@ -7,7 +7,7 @@ import subprocess

from .base import (SGELikeBatchManagerBase, logger, iflogger, logging)

-from nipype.interfaces.base import CommandLine +from ...interfaces.base import CommandLine, text_type

class PBSPlugin(SGELikeBatchManagerBase): @@ -97,7 +97,7 @@ class PBSPlugin(SGELikeBatchManagerBase): iflogger.setLevel(oldlevel) raise RuntimeError('\n'.join((('Could not submit pbs task' ' for node %s') % node._id,

• str(e))))
• text_type(e)))) else: break iflogger.setLevel(oldlevel)

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193451283.

[venpopov]

I realized what I had to do with the code you provided. But even after modifying the pbs.py files like that, I still get the same error, though with the slight differentce that is says "text_type(e)" now instead:

  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/pbs.py", line 100, in _submit_batchtask
    text_type(e))))
RuntimeError: Could not submit pbs task for node selectfiles.a16
decode() argument 1 must be string, not None
Interface CommandLine failed to run. 

[satra]

on your system could you please do:

import locale
print(locale.getdefaultlocale())

[venpopov]

('en_US', 'UTF-8')

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 5:09 PM, Satrajit Ghosh notifications@github.com wrote:

on your system could you please do:

import locale print(locale.getdefaultlocale())

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193478045.

[satra]

ok here is the likely culprit - let me know if the change below helps. if so i'll update the codebase.

diff --git a/nipype/interfaces/base.py b/nipype/interfaces/base.py
index 742ccd3..e1b854c 100644
--- a/nipype/interfaces/base.py
+++ b/nipype/interfaces/base.py
@@ -1164,7 +1164,7 @@ class Stream(object):
     def _read(self, drain):
         "Read from the file descriptor"
         fd = self.fileno()
-        buf = os.read(fd, 4096).decode()
+        buf = os.read(fd, 4096).decode(locale.getdefaultlocale()[1])
         if not buf and not self._buf:
             return None
         if '\n' not in buf:

[venpopov]

I just tried that, I get the same error as before.

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 5:33 PM, Satrajit Ghosh notifications@github.com wrote:

ok here is the likely culprit - let me know if the change below helps. if so i'll update the codebase.

diff --git a/nipype/interfaces/base.py b/nipype/interfaces/base.py index 742ccd3..e1b854c 100644 --- a/nipype/interfaces/base.py +++ b/nipype/interfaces/base.py @@ -1164,7 +1164,7 @@ class Stream(object): def _read(self, drain): "Read from the file descriptor" fd = self.fileno()

• buf = os.read(fd, 4096).decode()
• buf = os.read(fd, 4096).decode(locale.getdefaultlocale()[1]) if not buf and not self._buf: return None if '\n' not in buf:

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193486242.

[satra]

ok this is totally stumping me and without access to pbs at this moment is not helping. while i setup a pbs environment, let's try the following:

pip uninstall nipype # do this a few times till it says no nipype 
pip install https://github.com/satra/nipype/archive/fix/jobexecute_log.zip

this has a hacked pbs script which should error immediately. post the stack trace here. in the meantime i'll try to get to a pbs environment.

[venpopov]

Here's the error I get, it is similar:

/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/IPython/kernel/__init__.py:13: ShimWarning: The `IPython.kernel` package has been deprecated. You should import from ipykernel or jupyter_client instead.
  "You should import from ipykernel or jupyter_client instead.", ShimWarning)
Traceback (most recent call last):
  File "/home/vpopov/nipype_tutorial/nipype_tutorial.py", line 125, in <module>
    plugin_args=dict(qsub_args='-l nodes=1:ppn=4'))
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.py", line 595, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 265, in run
    graph=graph)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 402, in _send_procs_to_workers
    updatehash=updatehash)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/base.py", line 564, in _submit_job
    return self._submit_batchtask(batchscriptfile, node)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/pipeline/plugins/pbs.py", line 90, in _submit_batchtask
    result = cmd.run()
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1032, in run
    runtime = self._run_wrapper(runtime)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1460, in _run_wrapper
    runtime = self._run_interface(runtime)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1491, in _run_interface
    redirect_x=self._redirect_x)
  File "/data2/rederlab/software/anaconda2/lib/python2.7/site-packages/nipype/interfaces/base.py", line 1260, in run_command
    stdout = stdout.decode(locale.getdefaultlocale()[1])
TypeError: decode() argument 1 must be string, not None
Interface CommandLine failed to run. 

[satra]

but now i know exactly where it is happening. it seems that within this submission process locale.getdefaultlocale()[1] is returning None and not the UTF-8 that you posted earlier. now i need to figure out why!

[venpopov]

Thanks for the help. I hope you figure it out soon, because in the mean time I can't work! I can always go back to using my own software specific scripts but I've been convinced how much better nipype can be and I would really like to switch, if I manage to get it working.

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 6:22 PM, Satrajit Ghosh notifications@github.com wrote:

but now i know exactly where it is happening. it seems that within this submission process locale.getdefaultlocale()[1] is returning None and not the UTF-8 that you posted earlier. now i need to figure out why!

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193501114.

[satra]

@venpopov - it seems this is quite specific to how you are running python (the default encoding is not being passed to the shell).

i have attempted a patch by forcing an encoding - still at the same location. so do:

pip install --upgrade https://github.com/satra/nipype/archive/fix/jobexecute_log.zip

[venpopov]

It seems to be working fine now! Thank you so much. The pipeline is currently running, but I can see the different nodes get submitted to the pbs scheduler and then exectued, files are appearing in the working_dir. There seems to be a significant delay between the finish of one set of nodes and the next, but maybe this is due to the pbs scheduler itself and not nipype? Anyhow, I won't if everything works until the analysis is done, but the issue described here is gone.

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 8:36 PM, Satrajit Ghosh notifications@github.com wrote:

@venpopov https://github.com/venpopov - it seems this is quite specific to how you are running python (the default encoding is not being passed to the shell).

i have attempted a patch by forcing an encoding - still at the same location. so do:

pip install --upgrade https://github.com/satra/nipype/archive/fix/jobexecute_log.zip

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193547895.

[satra]

there is a config option that sets the polling duration. if your pipeline isn't intensive you can reduce this from the default 60s to something smaller.

workflow_variable.config['execution']['poll_sleep_duration'] = 10

i will close this issue as we merge the changes. i still want to verify that these were the proper changes.

[venpopov]

Thanks for the info! Do I put this within my pipeline .py file?

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 9:56 PM, Satrajit Ghosh notifications@github.com wrote:

there is a config option that sets the polling duration. if your pipeline isn't intensive you can reduce this from the default 60s to something smaller.

workflow_variable.config['execution']['poll_sleep_duration'] = 10

i will close this issue as we merge the changes. i still want to verify that these were the proper changes.

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193577025.

[satra]

yes, right before you call workflow_variable.run('PBS', ...)

[venpopov]

Cool, thanks!

Ven Popov Ph.D. Student Department of Psychology Center for the Neural Basis of Cognition Carnegie Mellon University web: venpopov.com

On Mon, Mar 7, 2016 at 10:02 PM, Satrajit Ghosh notifications@github.com wrote:

yes, right before you call workflow_variable.run('PBS', ...)

— Reply to this email directly or view it on GitHub https://github.com/nipy/nipype/issues/1396#issuecomment-193578712.

[venpopov]

Btw, if I set my metaflow with metaflow.config['execution']['poll_sleep_duration'] = 10, is this property inherited by the subworkflows, or do I have to set it for each/some of them as well?

[satra]

it should be inherited by sub workflows, unless the subworkflow is inside a function node.