search

FCP-INDI / C-PAC

Configurable Pipeline for the Analysis of Connectomes
https://fcp-indi.github.io/
GNU Lesser General Public License v3.0
63 stars 40 forks source link

Problem Loading CPAC #755

Closed as7a5 closed 6 years ago

as7a5 commented 6 years ago

Hi We recentely started to get this error message as we try to load up CPAC: I apreciate your help. We are running it on CentOS 6.8 and python-2.7.3

[siavoa01@node033 ~$ ipython Python 2.7.3 (default, Mar 29 2013, 16:50:34) Type "copyright", "credits" or "license" for more information.

IPython 4.2.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details.

In [1]: import CPAC

ImportError Traceback (most recent call last)

in () ----> 1 import CPAC /local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/__init__.py in () 19 test = _NoseTester().test 20 ---> 21 import anat_preproc, \ 22 func_preproc, \ 23 reho, \ /local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/__init__.py in () 1 from cpac_runner import run 2 from cpac_group_runner import run ----> 3 from cpac_pipeline import run 4 from cpac_group_analysis_pipeline import run 5 from cpac_basc_pipeline import run /local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py in () 14 import nipype.interfaces.io as nio 15 from nipype.interfaces.afni import preprocess ---> 16 from nipype.pipeline.utils import format_dot 17 import nipype.interfaces.ants as ants 18 import nipype.interfaces.c3 as c3 ImportError: No module named utils In [2]: I should mention that all my effort to update/install "utils" does not resolve the problem. The following packages are loaded as we try to load CPAC: - afni/2011_12_21_1014/linux_xorg7_64 - c3d/1.0.0 - cpac/0.3.9 - wxpython/3.0.0 - fsl/5.0.9 - ants/2.2.0
sgiavasis commented 6 years ago

Hello @as7a5,

What version of Nipype do you have?

as7a5 commented 6 years ago

This is what I get: -e git+https://github.com/nipy/nipype@8e1a86a8a562f62b3baf27bcfe7cfe0e075fd2d0#egg=nipype

On Feb 2, 2018, at 1:00 PM, Steve Giavasis notifications@github.com wrote:

Hello @as7a5 https://github.com/as7a5,

What version of Nipype do you have?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FCP-INDI/C-PAC/issues/755#issuecomment-362658648, or mute the thread https://github.com/notifications/unsubscribe-auth/AMlvenq4la29KQK-0UrFzfmef5ycmcDzks5tQ004gaJpZM4R0wPa.

sgiavasis commented 6 years ago

Looks like this is version 1.0 of Nipype. The latest few versions of CPAC rely on Nipype version 0.13- try changing it (can try pip install nipype==0.13) and see if this works. I'll take a note to make this clearer in the docs.

Which version of CPAC are you using, by the way?

as7a5 commented 6 years ago

Hi Steve, The version is 0.3.9 updating nipype==0.13 did not help (tried it again and here is the result): Appreciate your help in advance.

[siavoa01@node001 ~$ module add cpac/0.3.9 [siavoa01@node001 ~$ sudo -s

We trust you have received the usual lecture from the local System Administrator. It usually boils down to these three things:

#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.

Enter AD Password: [root@node001 siavoa01]# module add local shared cpac/0.3.9 [root@node001 siavoa01]# pip install nipype==0.13 Collecting nipype==0.13 /local/apps/python/2.7.3/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning. SNIMissingWarning /local/apps/python/2.7.3/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning Downloading nipype-0.13.0-py2.py3-none-any.whl (3.1MB) 100% |████████████████████████████████| 3.1MB 367kB/s Requirement already satisfied: traits>=4.6 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: funcsigs in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: networkx>=1.7 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: numpy>=1.6.2 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: pytest>=3.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: future>=0.16.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: scipy>=0.11 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: prov>=1.5.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: mock in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: configparser in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: click>=6.6.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: python-dateutil>=1.5 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: simplejson>=3.8.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: pydotplus in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: nibabel>=2.1.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from nipype==0.13) Requirement already satisfied: decorator>=4.1.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from networkx>=1.7->nipype==0.13) Requirement already satisfied: attrs>=17.2.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from pytest>=3.0->nipype==0.13) Requirement already satisfied: setuptools in /local/apps/python/2.7.3/lib/python2.7/site-packages (from pytest>=3.0->nipype==0.13) Requirement already satisfied: py>=1.5.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from pytest>=3.0->nipype==0.13) Requirement already satisfied: six>=1.10.0 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from pytest>=3.0->nipype==0.13) Requirement already satisfied: pluggy<0.7,>=0.5 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from pytest>=3.0->nipype==0.13) Requirement already satisfied: lxml in /local/apps/python/2.7.3/lib/python2.7/site-packages (from prov>=1.5.0->nipype==0.13) Requirement already satisfied: rdflib>=4.2.1 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from prov>=1.5.0->nipype==0.13) Requirement already satisfied: pbr>=0.11 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from mock->nipype==0.13) Requirement already satisfied: pyparsing>=2.0.1 in /local/apps/python/2.7.3/lib/python2.7/site-packages (from pydotplus->nipype==0.13) Requirement already satisfied: isodate in /local/apps/python/2.7.3/lib/python2.7/site-packages (from rdflib>=4.2.1->prov>=1.5.0->nipype==0.13) Installing collected packages: nipype Found existing installation: nipype 1.0.0.dev0+g8e1a86a Uninstalling nipype-1.0.0.dev0+g8e1a86a: Successfully uninstalled nipype-1.0.0.dev0+g8e1a86a Successfully installed nipype-0.13.0 /local/apps/python/2.7.3/lib/python2.7/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning [root@node001 siavoa01]# exit exit [siavoa01@node001 ~$ module list Currently Loaded Modulefiles: 1) sge/2011.11p1 3) default-environment 5) python/2.7 7) afni/2011_12_21_1014/linux_xorg7_64 9) c3d/1.0.0 11) cpac/0.3.9 2) local 4) gcc/4.4 6) wxpython/3.0.0 8) fsl/5.0.9 10) ants/2.2.0 [[siavoa01@node001 ~$ pip freeze |grep nipype nipype==0.13.0 siavoa01@node001 ~$ ipython Python 2.7.3 (default, Mar 29 2013, 16:50:34) Type "copyright", "credits" or "license" for more information.

IPython 4.2.0 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details.

In [1]: import CPAC

ImportError Traceback (most recent call last)

in () ----> 1 import CPAC /local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/__init__.py in () 19 test = _NoseTester().test 20 ---> 21 import anat_preproc, \ 22 func_preproc, \ 23 reho, \ /local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/__init__.py in () 1 from cpac_runner import run 2 from cpac_group_runner import run ----> 3 from cpac_pipeline import run 4 from cpac_group_analysis_pipeline import run 5 from cpac_basc_pipeline import run /local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py in () 14 import nipype.interfaces.io as nio 15 from nipype.interfaces.afni import preprocess ---> 16 from nipype.pipeline.utils import format_dot 17 import nipype.interfaces.ants as ants 18 import nipype.interfaces.c3 as c3 ImportError: No module named utils In [2]: > On Feb 5, 2018, at 12:36 PM, Steve Giavasis wrote: > > Looks like this is version 1.0 of Nipype. The latest few versions of CPAC rely on Nipype version 0.13- try upgrading (can try pip install nipype==0.13) and see if this works. I'll take a note to make this clearer in the docs. > > Which version of CPAC are you using, by the way? > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub , or mute the thread . >
sgiavasis commented 6 years ago

Hi @as7a5,

Sorry, didn't realize you were running CPAC 0.3.9 - try changing your Nipype version to 0.10.0 and see if that helps?

Steve

as7a5 commented 6 years ago

Hi steve, Thanks for your help. CPAC now cods with no error but after a while the user gets the following error. Are yo familiar with it? Is it related?

++ 3dcalc: AFNI version=AFNI_2011_12_21_1014 (Aug 5 2013) [64-bit] ++ Authored by: A cast of thousands ++ 3dcalc: AFNI version=AFNI_2011_12_21_1014 (Aug 5 2013) [64-bit] ++ Authored by: A cast of thousands ++ 3dcalc: AFNI version=AFNI_2011_12_21_1014 (Aug 5 2013) [64-bit] ++ Authored by: A cast of thousands Process Process-564: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 380, in prep_workflow flow = create_anat_datasource() File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/datasource.py", line 61, in create_anat_datasource wf.connect(inputnode, 'anat', outputnode, 'anat') File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 342, in connect for edge in self._graph.in_edges_iter(destnode): AttributeError: 'DiGraph' object has no attribute 'in_edges_iter' Process Process-565: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, *self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 380, in prep_workflow flow = create_anat_datasource() File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/datasource.py", line 61, in create_anat_datasource wf.connect(inputnode, 'anat', outputnode, 'anat') File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 342, in connect for edge in self._graph.in_edges_iter(destnode): AttributeError: 'DiGraph' object has no attribute 'in_edges_iter' Process Process-566: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(self._args, self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 380, in prep_workflow flow = create_anat_datasource() File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/datasource.py", line 61, in create_anat_datasource wf.connect(inputnode, 'anat', outputnode, 'anat') File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 342, in connect for edge in self._graph.in_edges_iter(destnode): AttributeError: 'DiGraph' object has no attribute 'in_edges_iter' ++ 3dcalc: AFNI version=AFNI_2011_12_21_1014 (Aug 5 2013) [64-bit] ++ Authored by: A cast of thousands ++ 3dcalc: AFNI version=AFNI_2011_12_21_1014 (Aug 5 2013) [64-bit] ++ Authored by: A cast of thousands ++ 3dcalc: AFNI version=AFNI_2011_12_21_1014 (Aug 5 2013) [64-bit] ++ Authored by: A cast of thousands Process Process-567: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 380, in prep_workflow flow = create_anat_datasource() File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/datasource.py", line 61, in create_anat_datasource wf.connect(inputnode, 'anat', outputnode, 'anat') File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 342, in connect for edge in self._graph.in_edges_iter(destnode): AttributeError: 'DiGraph' object has no attribute 'in_edges_iter' Process Process-568: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, *self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 380, in prep_workflow flow = create_anat_datasource() File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/datasource.py", line 61, in create_anat_datasource wf.connect(inputnode, 'anat', outputnode, 'anat') File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 342, in connect for edge in self._graph.in_edges_iter(destnode): AttributeError: 'DiGraph' object has no attribute 'in_edges_iter' Process Process-569: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(self._args, self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 380, in prep_workflow flow = create_anat_datasource() File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/datasource.py", line 61, in create_anat_datasource wf.connect(inputnode, 'anat', outputnode, 'anat') File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 342, in connect for edge in self._graph.in_edges_iter(destnode): AttributeError: 'DiGraph' object has no attribute 'in_edges_iter'

Thanks

On Feb 6, 2018, at 12:42 PM, Steve Giavasis notifications@github.com wrote:

Hi @as7a5 https://github.com/as7a5,

Sorry, didn't realize you were running CPAC 0.3.9 - try changing your Nipype version to 0.10.0 and see if that helps?

Steve

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FCP-INDI/C-PAC/issues/755#issuecomment-363504299, or mute the thread https://github.com/notifications/unsubscribe-auth/AMlveuj1wrHsSizogdCJgErBR0y29gMJks5tSI8AgaJpZM4R0wPa.

nrajamani3 commented 6 years ago

Hey @as7a5 This might occur because of the version of NetworkX, we were able to reproduce this while using NetworkX version 2.0. Please re-install and use version 1.11 (as recommended by C-PAC), and see if that helps. Nanditha

as7a5 commented 6 years ago

Hi Nonditha, The next step is the following error: User says It created directories but they are empty:

Process Process-569: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 5841, in prep_workflow create_log_template(pip_ids, wf_names, scan_ids, subject_id, log_dir) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/utils.py", line 1854, in create_log_template template = Template(raw_text) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 945, in new return env.from_string(source, template_class=cls) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 880, in from_string return cls.from_code(self, self.compile(source), globals, None) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 591, in compile self.handle_exception(exc_info, source_hint=source_hint) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "", line 127, in template TemplateSyntaxError: expected token 'end of statement block', got '.'

Thanks for the help.

On Feb 9, 2018, at 4:44 PM, nrajamani3 notifications@github.com wrote:

Hey @as7a5 https://github.com/as7a5 This might occur because of the version of NetworkX, we were able to reproduce this while using NetworkX version 2.0. Please re-install and use version 1.11 (as recommended by C-PAC), and see if that helps. Nanditha

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FCP-INDI/C-PAC/issues/755#issuecomment-364577663, or mute the thread https://github.com/notifications/unsubscribe-auth/AMlveuWEel3wiZfKODIA1kId7UqqvVoCks5tTLwvgaJpZM4R0wPa.

as7a5 commented 6 years ago

I thank in advance. The next step is the following error: User says It created directories but they are empty:

Process Process-569: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(*self._args, **self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 5841, in prep_workflow create_log_template(pip_ids, wf_names, scan_ids, subject_id, log_dir) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/utils.py", line 1854, in create_log_template template = Template(raw_text) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 945, in new return env.from_string(source, template_class=cls) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 880, in from_string return cls.from_code(self, self.compile(source), globals, None) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 591, in compile self.handle_exception(exc_info, source_hint=source_hint) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 780, in handle_exception reraise(exc_type, exc_value, tb) File "", line 127, in template TemplateSyntaxError: expected token 'end of statement block', got '.'

Thanks for the help. User is in urgent situation.

sgiavasis commented 6 years ago

Hi @as7a5,

Try turning off the QA pages interface (in "Output Settings" in the pipeline config), and see if it works.

If so, this is a known error in older versions of CPAC. Note that the QA pages will be returned (and that error also resolved) in the next release (v1.0.4).

Best, Steve

as7a5 commented 6 years ago

I apologize for too many back and forth. I made sure bioth networkx and Jinja2 have correct version. Now the last lines of output reads:

180215-16:24:21,703 workflow INFO: sink index: 52 180215-16:24:21,706 workflow INFO: node, out_file, key: resting_preproc_50608_session_1.func_slice_timing_correction_0, out_file, slice_time_corrected 180215-16:24:21,710 workflow INFO: sink index: 53 180215-16:24:21,720 workflow INFO:

WARNING: Datasink - Cannot Create the strategy and pipeline graph, dot or/and pygraphviz is not installed

180215-16:24:21,722 workflow INFO: /ifs/data/li_subtypes/Females/out/pipeline_ABIDE_males_females_DC/50608_session_1 Process Process-569: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/local/apps/python/2.7.3/lib/python2.7/multiprocessing/process.py", line 114, in run self._target(self._args, **self._kwargs) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/pipeline/cpac_pipeline.py", line 5841, in prep_workflow create_log_template(pip_ids, wf_names, scan_ids, subject_id, log_dir) File "/local/apps/cpac/0.3.9/lib/python2.7/site-packages/CPAC/utils/utils.py", line 1854, in create_log_template template = Template(raw_text) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 945, in new t._debug_info = namespace['debug_info'] File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 880, in from_string

File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 591, in compile None, None) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/environment.py", line 773, in handle_exception method ask the loader for the template and returns a :class:Template. File "/local/apps/python/2.7.3/lib/python2.7/site-packages/jinja2/debug.py", line 18, in from jinja2._compat import iteritems, reraise, code_type ImportError: cannot import name code_type

sgiavasis commented 6 years ago

Hi @as7a5,

What are your networkx and jinja2 versions, exactly?

as7a5 commented 6 years ago

Ok seems the problem is solved. BUT can you make a conda (miniconda,…) environment for this? This will be very helpful since there is a very restricted dependencies here. Thanks

On Feb 19, 2018, at 12:34 PM, Steve Giavasis notifications@github.com wrote:

Hi @as7a5 https://github.com/as7a5,

What are your networkx and jinja2 versions, exactly?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FCP-INDI/C-PAC/issues/755#issuecomment-366760276, or mute the thread https://github.com/notifications/unsubscribe-auth/AMlvenf7DU5aA_Fg38BrSZOYunDwY5v8ks5tWbDDgaJpZM4R0wPa.

sgiavasis commented 6 years ago

Great!

Also the prepackaged Miniconda environment is a great idea. I'll look into swinging this- created issue #769

Thanks for your feedback!

Best, Steve

as7a5 commented 6 years ago

Sorry I was too quick. The user almost at the end of process gets the following:

1. File: crash/crash-20180221-104854-florid01-tissue_masks.c0.c0.c0.c0.pklz Node: resting_preproc_50815_session_1.nuisance_0.tissue_masks.c0.c0.c0.c0 Working directory: /ifs/data/li_subtypes/Females/work/resting_preproc_50815_session_1/nuisance_0/_scan_rest_1_rest/_csf_threshold_0.96/_gm_threshold_0.7/_wm_threshold_0.96/tissue_masks

Node inputs:

csf_seg_file = /ifs/data/li_subtypes/Females/work/resting_preproc_50815_session_1/nuisance_0/_csf_threshold_0.96/csf_anat_to_2mm_flirt_applyxfm/segment_prob_0_maths_maths_maths_flirt.nii.gz csf_threshold = data_file = /ifs/data/li_subtypes/Females/work/resting_preproc_50815_session_1/nuisance_0/_scan_rest_1_rest/_csf_threshold_0.96/func_to_2mm_flirt_applyxfm/rest_calc_tshift_resample_volreg_calc_maths_flirt.nii.gz function_str = S"def extract_tissue_data(data_file,\n ventricles_mask_file,\n wm_seg_file, csf_seg_file, gm_seg_file,\n wm_threshold=0.0, csf_threshold=0.0, gm_threshold=0.0):\n import numpy as np\n import nibabel as nb\n import os \n from CPAC.nuisance import erode_mask\n from CPAC.utils import safe_shape\n\n print 'Tissues extraction thresholds wm %d, csf %d, gm %d' % (wm_threshold,\n csf_threshold,\n gm_threshold)\n\n try:\n data = nb.load(data_file).get_data().astype('float64')\n except:\n raise MemoryError('Unable to load %s' % data_file)\n\n\n try:\n lat_ventricles_mask = nb.load(ventricles_mask_file).get_data().astype('float64')\n except:\n raise MemoryError('Unable to load %s' % lat_ventricles_mask)\n\n\n if not safe_shape(data, lat_ventricles_mask):\n raise ValueError('Spatial dimensions for data and the lateral ventricles mask do not match')\n\n try:\n wm_seg = nb.load(wm_seg_file).get_data().astype('float64')\n except:\n raise MemoryError('Unable to load %s' % wm_seg)\n\n\n if not safe_shape(data, wm_seg):\n raise ValueError('Spatial dimensions for data, white matter segment do not match')\n\n wm_mask = erode_mask(wm_seg > wm_threshold)\n wm_sigs = data[wm_mask]\n file_wm = os.path.join(os.getcwd(), 'wm_signals.npy')\n np.save(file_wm, wm_sigs)\n del wm_sigs\n\n try:\n csf_seg = nb.load(csf_seg_file).get_data().astype('float64')\n except:\n raise MemoryError('Unable to load %s' % csf_seg)\n\n\n if not safe_shape(data, csf_seg):\n raise ValueError('Spatial dimensions for data, cerebral spinal fluid segment do not match')\n\n # Only take the CSF at the lateral ventricles as labeled in the Harvard\n # Oxford parcellation regions 4 and 43\n csf_mask = (csf_seg > csf_threshold)*(lat_ventricles_mask==1)\n csf_sigs = data[csf_mask]\n file_csf = os.path.join(os.getcwd(), 'csf_signals.npy')\n np.save(file_csf, csf_sigs)\n del csf_sigs\n\n\n try:\n gm_seg = nb.load(gm_seg_file).get_data().astype('float64')\n except:\n raise MemoryError('Unable to load %s' % gm_seg)\n\n\n if not safe_shape(data, gm_seg):\n raise ValueError('Spatial dimensions for data, gray matter segment do not match')\n\n\n gm_mask = erode_mask(gm_seg > gm_threshold)\n gm_sigs = data[gm_mask]\n file_gm = os.path.join(os.getcwd(), 'gm_signals.npy')\n np.save(file_gm, gm_sigs)\n del gm_sigs\n\n\n\n nii = nb.load(wm_seg_file)\n wm_mask_file = os.path.join(os.getcwd(), 'wm_mask.nii.gz')\n csf_mask_file = os.path.join(os.getcwd(), 'csf_mask.nii.gz')\n gm_mask_file = os.path.join(os.getcwd(), 'gm_mask.nii.gz')\n nb.Nifti1Image(wm_mask, header=nii.get_header(), affine=nii.get_affine()).to_filename(wm_mask_file)\n nb.Nifti1Image(csf_mask, header=nii.get_header(), affine=nii.get_affine()).to_filename(csf_mask_file)\n nb.Nifti1Image(gm_mask, header=nii.get_header(), affine=nii.get_affine()).to_filename(gm_mask_file)\n\n return file_wm, file_csf, file_gm\n" . gm_seg_file = /ifs/data/li_subtypes/Females/work/resting_preproc_50815_session_1/nuisance_0/_gm_threshold_0.7/gm_anat_to_2mm_flirt_applyxfm/segment_prob_1_maths_maths_maths_flirt.nii.gz gm_threshold = ignore_exception = False ventricles_mask_file = /ifs/data/li_subtypes/Females/work/resting_preproc_50815_session_1/nuisance_0/_csf_threshold_0.96/ho_mni_to_2mm_ants_applyxfm/HarvardOxford-lateral-ventricles-thr25-2mm_trans.nii.gz wm_seg_file = /ifs/data/li_subtypes/Females/work/resting_preproc_50815_session_1/nuisance_0/_wm_threshold_0.96/wm_anat_to_2mm_flirt_applyxfm/segment_prob_2_maths_maths_maths_flirt.nii.gz wm_threshold =

Traceback: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 18, in run_node result['result'] = node.run(updatehash=updatehash) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1424, in run self._run_interface() File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1534, in _run_interface self._result = self._run_command(execute) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1660, in _run_command result = self._interface.run() File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/interfaces/base.py", line 998, in run runtime = self._run_interface(runtime) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/interfaces/utility.py", line 436, in _run_interface out = function_handle(**args) File "", line 42, in extract_tissue_data File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/npyio.py", line 511, in save pickle_kwargs=pickle_kwargs) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/format.py", line 562, in write_array version) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/format.py", line 308, in _write_array_header header = asbytes(_filter_header(header)) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/format.py", line 467, in _filter_header return tokenize.untokenize(tokens) File "/local/apps/python/2.7.3/lib/python2.7/tokenize.py", line 262, in untokenize return ut.untokenize(iterable) File "/local/apps/python/2.7.3/lib/python2.7/tokenize.py", line 198, in untokenize self.add_whitespace(start) File "/local/apps/python/2.7.3/lib/python2.7/tokenize.py", line 187, in add_whitespace assert row <= self.prev_row AssertionError: Interface Function failed to run.

2.

File: crash/crash-20180220-131451-florid01-cal_DVARS.c0.pklz Node: resting_preproc_50797_session_1.gen_motion_stats_0.cal_DVARS.c0 Working directory: /ifs/data/li_subtypes/Females/work/resting_preproc_50797_session_1/gen_motion_stats_0/_scan_rest_1_rest/cal_DVARS

Node inputs:

function_str = S'def calculate_DVARS(rest, mask):\n """\n Method to calculate DVARS as per\n power\'s method\n\n Parameters\n ----------\n rest : string (nifti file)\n path to motion correct functional data\n mask : string (nifti file)\n path to brain only mask for functional data\n\n Returns\n -------\n out_file : string (numpy mat file)\n path to file containing array of DVARS \n calculation for each voxel\n """\n\n import numpy as np\n import nibabel as nib\n import os\n\n out_file = os.path.join(os.getcwd(), \'DVARS.npy\')\n\n rest_data = nib.load(rest).get_data().astype(np.float32)\n mask_data = nib.load(mask).get_data().astype(\'bool\')\n\n #square of relative intensity value for each voxel across\n #every timepoint \n data = np.square(np.diff(rest_data, axis = 3))\n #applying mask, getting the data in the brain only\n data = data[mask_data]\n #square root and mean across all timepoints inside mask\n DVARS = np.sqrt(np.mean(data, axis=0))\n\n\n np.save(out_file, DVARS)\n\n return out_file\n' . ignore_exception = False mask = /ifs/data/li_subtypes/Females/work/resting_preproc_50797_session_1/func_preproc_bet_0/_scan_rest_1_rest/erode_one_voxel/rest_calc_tshift_resample_volreg_brain_mask_ero.nii.gz rest = /ifs/data/li_subtypes/Females/work/resting_preproc_50797_session_1/func_preproc_bet_0/_scan_rest_1_rest/func_motion_correct_A/rest_calc_tshift_resample_volreg.nii.gz

Traceback: Traceback (most recent call last): File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 18, in run_node result['result'] = node.run(updatehash=updatehash) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1424, in run self._run_interface() File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1534, in _run_interface self._result = self._run_command(execute) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/pipeline/engine.py", line 1660, in _run_command result = self._interface.run() File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/interfaces/base.py", line 998, in run runtime = self._run_interface(runtime) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/nipype/interfaces/utility.py", line 436, in _run_interface out = function_handle(**args) File "", line 38, in calculate_DVARS File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/npyio.py", line 511, in save pickle_kwargs=pickle_kwargs) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/format.py", line 562, in write_array version) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/format.py", line 308, in _write_array_header header = asbytes(_filter_header(header)) File "/local/apps/python/2.7.3/lib/python2.7/site-packages/numpy/lib/format.py", line 467, in _filter_header return tokenize.untokenize(tokens) File "/local/apps/python/2.7.3/lib/python2.7/tokenize.py", line 262, in untokenize return ut.untokenize(iterable) File "/local/apps/python/2.7.3/lib/python2.7/tokenize.py", line 198, in untokenize self.add_whitespace(start) File "/local/apps/python/2.7.3/lib/python2.7/tokenize.py", line 187, in add_whitespace assert row <= self.prev_row AssertionError: Interface Function failed to run.

Just to mention: Jinja2==2.7.2 networkx==1.11 nipype==0.10.0

I apreciate your help.

sgiavasis commented 6 years ago

Do these files look/open okay?

/ifs/data/li_subtypes/Females/work/resting_preproc_50797_session_1/func_preproc_bet_0/_scan_rest_1_rest/erode_one_voxel/rest_calc_tshift_resample_volreg_brain_mask_ero.nii.gz

/ifs/data/li_subtypes/Females/work/resting_preproc_50797_session_1/func_preproc_bet_0/_scan_rest_1_rest/func_motion_correct_A/rest_calc_tshift_resample_volreg.nii.gz

as7a5 commented 6 years ago

User says:

"Yes, they can be opened and looking fine.

D"

On Feb 27, 2018, at 4:34 PM, Steve Giavasis notifications@github.com wrote:

Do these files look/open okay?

/ifs/data/li_subtypes/Females/work/resting_preproc_50797_session_1/func_preproc_bet_0/_scan_rest_1_rest/erode_one_voxel/rest_calc_tshift_resample_volreg_brain_mask_ero.nii.gz

/ifs/data/li_subtypes/Females/work/resting_preproc_50797_session_1/func_preproc_bet_0/_scan_rest_1_rest/func_motion_correct_A/rest_calc_tshift_resample_volreg.nii.gz

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/FCP-INDI/C-PAC/issues/755#issuecomment-369034133, or mute the thread https://github.com/notifications/unsubscribe-auth/AMlveoV9VEdXsWh_I_KSj1_4HCy8o0yDks5tZHT9gaJpZM4R0wPa.

as7a5 commented 6 years ago

Hi Ali and Steve,

thanks for the help with running CPAC on the cluster in the past. I've been running it in the last couple of days and it seemed to be working, but looking at the outputs it looks like it only generated half of the outputs that are specified. Is it possible that there are still some dependencies missing? For example only fALFF is computed, however there is no output for SCA / ReHo/ VMHC.

Thanks,

Dori

Dorothea L. Floris, PhD PostDoctoral Fellow

Hassenfeld Children’s Hospital at NYU Langone Health

Department of Child and Adolescent Psychiatry

Child Study Center

One Park Avenue

8th Floor

New York, NY 10016 T 646-754-7245 Dorothea.Floris@nyumc.org| www.AboutOurKids.orghttps://mail.nyumc.org/owa/redir.aspx?C=AP8eWij240yi2EazdMUHzobcxC5zgtMIFBwagnio3tQr-zu8Yx0Y03_TWm0MBDoY6-29E9t9kK0.&URL=http%3a%2f%2fwww.aboutourkids.org%2f

Connect with us on Facebook for articles, workshops, tips, and more: www.facebook.com/NYUChildStudyCenterhttps://mail.nyumc.org/owa/redir.aspx?C=AP8eWij240yi2EazdMUHzobcxC5zgtMIFBwagnio3tQr-zu8Yx0Y03_TWm0MBDoY6-29E9t9kK0.&URL=https%3a%2f%2fmail.nyumc.org%2fowa%2fredir.aspx%3fC%3daEKHC0jTCE6MZZ0hEBidv1dAWC6PbdEIaRBRgGJ61bInkFvBTfiSzzDOrMc18CPwDWgcl9sY4-M.%26URL%3dhttp%253a%252f%252fwww.facebook.com%252fNYUChildStudyCenter

From: Ali Siavosh-Haghighi ali.siavosh@gmail.com Sent: Wednesday, February 21, 2018 10:43:52 AM To: FCP-INDI/C-PAC Cc: FCP-INDI/C-PAC; Mention; Floris, Dorothea Subject: Re: [FCP-INDI/C-PAC] Problem Loading CPAC (#755)

Ok seems the problem is solved. BUT can you make a conda (miniconda,…) environment for this? This will be very helpful since there is a very restricted dependencies here. Thanks

On Feb 19, 2018, at 12:34 PM, Steve Giavasis notifications@github.com<mailto:notifications@github.com> wrote:

Hi @as7a5https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_as7a5&d=DwMFaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=taTtOEkBqnS4aE9QHtpsO7oN3sOBh0o2BQdu5t2UdgQ&m=XfH2JVe2-m3j9ZuDVb-m-ajOE4MgytLg9TEggFcw-u8&s=4ujWWKUA2Sq7biXuvx4HBi45XiNxUNYShFu3u-SvRsA&e=,

What are your networkx and jinja2 versions, exactly?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_FCP-2DINDI_C-2DPAC_issues_755-23issuecomment-2D366760276&d=DwMFaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=taTtOEkBqnS4aE9QHtpsO7oN3sOBh0o2BQdu5t2UdgQ&m=XfH2JVe2-m3j9ZuDVb-m-ajOE4MgytLg9TEggFcw-u8&s=g2MPCGM3dNrWWKPq8z9GxFwGygOndj5T-JC6YCy8_Tc&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AMlvenf7DU5aA-5FFg38BrSZOYunDwY5v8ks5tWbDDgaJpZM4R0wPa&d=DwMFaQ&c=j5oPpO0eBH1iio48DtsedbOBGmuw5jHLjgvtN2r4ehE&r=taTtOEkBqnS4aE9QHtpsO7oN3sOBh0o2BQdu5t2UdgQ&m=XfH2JVe2-m3j9ZuDVb-m-ajOE4MgytLg9TEggFcw-u8&s=0QJh24-GzNoyih3fB05IR5m1J10_1WLt7NMmeEkG8u8&e=.

This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain information that is proprietary, confidential, and exempt from disclosure under applicable law. Any unauthorized review, use, disclosure, or distribution is prohibited. If you have received this email in error please notify the sender by return email and delete the original message. Please note, the recipient should check this email and any attachments for the presence of viruses. The organization accepts no liability for any damage caused by any virus transmitted by this email.

nrajamani3 commented 6 years ago

Hi Dori, Which version of C-PAC are you using?We just released a new beta version - 1.1.0, and that might have already resolved this error! Please let us know if you've already updated and still getting this error.

sgiavasis commented 6 years ago

Didn't realize this was related- opened another issue for this here: https://github.com/FCP-INDI/C-PAC/issues/796

This is an error that was resolved in v1.0.1. If users don't want to upgrade their C-PAC install, they can get around this problem by making sure their Numpy install is not past v1.11 (can do pip install numpy==1.11).

sgiavasis commented 6 years ago

User is now running smoothly.