search

populse / capsul

Collaborative Analysis Platform : Simple, Unifying, Lean
Other
7 stars 14 forks source link

Configure nipype with SPM12 standalone #279

Closed rbregand closed 1 year ago

rbregand commented 1 year ago

Hi,

I built a new casa (5.3.19), everything went fine for the steps : sources, configure and build. To configure nipype with SPM12 standalone, the function capsul.capsul.in_context.nipype.configure_spm() is execute and this errors occurs : 

Traceback (most recent call last):
  File "/casa/home/.local/lib/python3.10/site-packages/nipype/utils/spm_docs.py", line 49, in _strip_header
    index = doc.index(hdr)
ValueError: substring not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/casa/home/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/casa/home/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/casa/home/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/casa/host/src/qualicati/bin/qualicati", line 39, in <module>
    qualiCATIToolBox, batchMode, qapp = qualiCATI.createQualiCATI(
  File "/casa/host/src/qualicati/python/qualiCATI/__init__.py", line 162, in createQualiCATI
    activate_configuration(conf)
  File "/casa/host/src/capsul/capsul/engine/__init__.py", line 572, in activate_configuration
    activate_module(m)
  File "/casa/host/src/capsul/capsul/engine/__init__.py", line 597, in activate_module
    activate_configurations()
  File "/casa/host/src/capsul/capsul/engine/module/nipype.py", line 55, in activate_configurations
    nipype.configure_all()
  File "/casa/host/src/capsul/capsul/in_context/nipype.py", line 16, in configure_all
    configure_spm()
  File "/casa/host/src/capsul/capsul/in_context/nipype.py", line 41, in configure_spm
    spm.SPMCommand.set_mlab_paths(
  File "/casa/home/.local/lib/python3.10/site-packages/nipype/interfaces/spm/base.py", line 323, in set_mlab_paths
    info_dict = Info.getinfo(matlab_cmd=matlab_cmd, paths=paths, use_mcr=use_mcr)
  File "/casa/home/.local/lib/python3.10/site-packages/nipype/interfaces/spm/base.py", line 239, in getinfo
    out = sd._strip_header(out.runtime.stdout)
  File "/casa/home/.local/lib/python3.10/site-packages/nipype/utils/spm_docs.py", line 51, in _strip_header
    raise IOError("This docstring was not generated by Nipype!\n") from e
OSError: This docstring was not generated by Nipype!

The same error happens to a colleague depending on the version of SPM12 used, you will find the results of our tests here : image

This is how we tested each case: 

Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nipype.interfaces import spm
>>> spm.SPMCommand.set_mlab_paths(matlab_cmd='/path/to/spm12-standalone-7771/run_spm12.sh /path/to/spm12-standalone-7771/mcr/v713 script', use_mcr=True)
>>> print(spm.SPMCommand().version)
12.7771

We think the error is linked to the OS of the user. Have you already noticed this weird interaction ? Do you know how to solve it ?

Thanks in advance for your help, Rémy

denisri commented 1 year ago

Hi,

I am not sure to understand how to trigger the problem. Is it the 3 lines in the snippet at the end of your message ? But this one does not make use of capsul and does not match the error message at the beginning. Anyway when I run these lines on my side (Ubuntu 18.04 host, container casa-dev-5.3-19, casa-dev-5.4.4, nipype 1.8.5, spm12 standalone 7771 + MCR v97) I get not error.

Note: in the error message, you seem to have a local install of nipype (in /casa/home/.local), which is not the one coming with casa-distro images. I don't know if this can make any difference.

The error, nipype/utils/spm_docs.py, line 49, is in a function which seems to parse a doc wrapped in nipype, but we don't have a full traceback, so I don't know where it exactly comes from. The docstring of this function says Strip Matlab header and splash info off doc. Searches for the tag 'NIPYPE' in the doc and returns everyting after that., so I suspect it is either within nipype, or liked to the matlab MCR version ?

Could you please remove your local install of nipype and try again please to see if it still happens using the one in casa-distro images ?

rbregand commented 1 year ago

After removing local nipype package, the error still occurs : 

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/nipype/utils/spm_docs.py", line 49, in _strip_header
    index = doc.index(hdr)
ValueError: substring not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/casa/home/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/__main__.py", line 45, in <module>
    cli.main()
  File "/casa/home/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 444, in main
    run()
  File "/casa/home/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/../debugpy/server/cli.py", line 285, in run_file
    runpy.run_path(target_as_str, run_name=compat.force_str("__main__"))
  File "/usr/lib/python3.10/runpy.py", line 289, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/casa/host/src/qualicati/bin/qualicati", line 39, in <module>
    qualiCATIToolBox, batchMode, qapp = qualiCATI.createQualiCATI(
  File "/casa/host/src/qualicati/python/qualiCATI/__init__.py", line 162, in createQualiCATI
    activate_configuration(conf)
  File "/casa/host/src/capsul/capsul/engine/__init__.py", line 572, in activate_configuration
    activate_module(m)
  File "/casa/host/src/capsul/capsul/engine/__init__.py", line 597, in activate_module
    activate_configurations()
  File "/casa/host/src/capsul/capsul/engine/module/nipype.py", line 55, in activate_configurations
    nipype.configure_all()
  File "/casa/host/src/capsul/capsul/in_context/nipype.py", line 16, in configure_all
    configure_spm()
  File "/casa/host/src/capsul/capsul/in_context/nipype.py", line 41, in configure_spm
    spm.SPMCommand.set_mlab_paths(
  File "/usr/local/lib/python3.10/dist-packages/nipype/interfaces/spm/base.py", line 323, in set_mlab_paths
    info_dict = Info.getinfo(matlab_cmd=matlab_cmd, paths=paths, use_mcr=use_mcr)
  File "/usr/local/lib/python3.10/dist-packages/nipype/interfaces/spm/base.py", line 239, in getinfo
    out = sd._strip_header(out.runtime.stdout)
  File "/usr/local/lib/python3.10/dist-packages/nipype/utils/spm_docs.py", line 51, in _strip_header
    raise IOError("This docstring was not generated by Nipype!\n") from e
OSError: This docstring was not generated by Nipype!

Versions: OS : Linux Mint 20.1 casa : 3.5.19, SPM12standalone : 7771 mcr : 713

The errors doesn't seem easily to triggered especially for Ubuntu user. According to the previous table, we also notice during our tests no issue with your current setup (Ubuntu 18.04 host, container casa-dev-5.3-19, casa-dev-5.4.4, nipype 1.8.5, spm12 standalone 7771). The error might occurs if you're only changing your SPM version from 7771 to 6472.

denisri commented 1 year ago

I don't have 6472 at hand and don't really want to take time to install it... In the traceback you show, you are not running the little code you have provided, but you are running qualicati. I assume it doesn't change anything but can you check that the problem still happens using your little code ? It would make sure that it's not a configuration/path problem from values entered in qualicati / capsul. If it still hapens, then the problem is not related to capsul at all, but to nipype/spm/matlab and the host system (?). Anyway we will try to find out... According to the traceback, the problem occurs when parsing the output (stdout) of a matlab/mcr command, and looking for the string NIPYPE in it. It doesn't find it, which is the problem. Can you try to get the output which causes the problem ? Using this code, replacing matlab_cmd with your path:

from nipype.interfaces import matlab
from traits.api import Undefined
matlab_cmd='/host/usr/local/spm12-standalone/run_spm12.sh /host/usr/local/spm12-standalone/mcr/v97 script'
mlab = matlab.MatlabCommand(matlab_cmd=matlab_cmd)
mlab.inputs.mfile = False
mlab.inputs.nodesktop = Undefined
mlab.inputs.nosplash = Undefined
mlab.inputs.single_comp_thread = Undefined
mlab.inputs.mfile = True
mlab.inputs.uses_mcr = True
mlab.inputs.script = """
if isempty(which('spm')),
throw(MException('SPMCheck:NotFound','SPM not in matlab path'));
end;
spm_path = spm('dir');
[name, version] = spm('ver');
fprintf(1, 'NIPYPE path:%s|name:%s|release:%s', spm_path, name, version);
exit;
"""
out = mlab.run()
print(out.runtime.stdout)

This code (copied from nipype source code) produces for me this output:

------------------------------------------
Setting up environment variables
---
LD_LIBRARY_PATH is .:/host/usr/local/spm12-standalone/mcr/v97/runtime/glnxa64:/host/usr/local/spm12-standalone/mcr/v97/bin/glnxa64:/host/usr/local/spm12-standalone/mcr/v97/sys/os/glnxa64:/host/usr/local/spm12-standalone/mcr/v97/sys/opengl/lib/glnxa64
SPM12, version 7771 (standalone)
MATLAB, version 9.7.0.1737446 (R2019b) Update 9
 ___  ____  __  __                                            
/ __)(  _ \(  \/  )                                           
\__ \ )___/ )    (   Statistical Parametric Mapping           
(___/(__)  (_/\/\_)  SPM12 - https://www.fil.ion.ucl.ac.uk/spm/

Executing /home/dr144257/casa_distro/pyscript.m at 27-Apr-2023 19:34:56:
Executing /home/dr144257/casa_distro/pyscript.m at 27-Apr-2023 19:34:56:
-----------------------------------------------------------------------------------------------------
MATLAB Version: 9.7.0.1737446 (R2019b) Update 9
MATLAB License Number: unknown
Operating System: Linux 5.4.0-146-generic #163~18.04.1-Ubuntu SMP Mon Mar 20 15:02:59 UTC 2023 x86_64
Java Version: Java 1.8.0_202-b08 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
-----------------------------------------------------------------------------------------------------
MATLAB                                                Version 9.7         (R2019b)
MATLAB Compiler                                       Version 7.1         (R2019b)
NIPYPE path:/volatile/local/spm12-standalone-7771/spm12_mcr/spm12/spm12|name:SPM12|release:7771

The last line (NIPYPE path: ...) is what is expected. However, if I use a wrong path for the MCR for instance, using:

matlab_cmd='/host/usr/local/spm12-standalone/run_spm12.sh /host/usr/local/spm12-standalone/mcr/v978 script'

(note the v978 instead of v97 at the end of the MCR path, which thus is a wrong, non-existing path on my computer) Then the output is:

------------------------------------------
Setting up environment variables
---
LD_LIBRARY_PATH is .:/host/usr/local/spm12-standalone/mcr/v978/runtime/glnxa64:/host/usr/local/spm12-standalone/mcr/v978/bin/glnxa64:/host/usr/local/spm12-standalone/mcr/v978/sys/os/glnxa64:/host/usr/local/spm12-standalone/mcr/v978/sys/opengl/lib/glnxa64

and the expected line starting with NIPYPE is not here. So maybe it's just a wrong path in the config ?

rbregand commented 1 year ago

The script provides the same error as qualicati's :

(cati_platform-master-5.3)remy.bregand@ICM-CATI-LF004:/casa/host/src/qualicati$ python
Python 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from nipype.interfaces import spm
>>> spm.SPMCommand.set_mlab_paths(matlab_cmd='/path/to/spm12-standalone-7771/run_spm12.sh /path/to/spm12-standalone-7771/mcr/v713 script', use_mcr=True)
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/nipype/utils/spm_docs.py", line 49, in _strip_header
    index = doc.index(hdr)
ValueError: substring not found

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/nipype/interfaces/spm/base.py", line 323, in set_mlab_paths
    info_dict = Info.getinfo(matlab_cmd=matlab_cmd, paths=paths, use_mcr=use_mcr)
  File "/usr/local/lib/python3.10/dist-packages/nipype/interfaces/spm/base.py", line 239, in getinfo
    out = sd._strip_header(out.runtime.stdout)
  File "/usr/local/lib/python3.10/dist-packages/nipype/utils/spm_docs.py", line 51, in _strip_header
    raise IOError("This docstring was not generated by Nipype!\n") from e
OSError: This docstring was not generated by Nipype!

I agree, I don't think the issue is related to capsul. It's more about the link between SPM and nipype, I really appreciate your help thanks.

After running your code I got an error totally different from yours : 

cd /casa/host/src/qualicati ; /usr/bin/env /bin/python /casa/home/.vscode/extensions/ms-python.python-2022.8.1/pythonFiles/lib/python/debugpy/launcher 34603 -- /casa/home/nipipe_test.py 
My Own Exception: Fatal error loading library /path/to/spm12-standalone-7771/mcr/v713/bin/glnxa64/libmwmclmcr.so Error: libXp.so.6: cannot open shared object file: No such file or directory

It seems libXp.so.6 package is missing, it's a known trouble-shoothing as you can see here : https://en.wikibooks.org/wiki/SPM/Standalone

I also found a ticket speaking about this package and the version of the MCR (https://github.com/brainvisa/casa-distro/issues/268 starting from Nov 5, 2021 comment)

denisri commented 1 year ago

Oh yes, that's it ! It's the MCR version and libXp.so. Thanks for finding it out...

libXp.so is completely obsolete, it is not needed by recent versions of the MCR, and it was impossible (or, at least, too difficult) to install it in Ubuntu 22.04 images (casa-run-5.3 images): see https://github.com/brainvisa/casa-distro/commit/1cdb656890445d88ffb3478249b5e7f9406df675 So to use SPM in Ubuntu 22.04 images, you will need to use a standalone SPM built for more recent MCR. In Neurospin we use v97 and it works. We have no better choice...

rbregand commented 1 year ago

I tried with the MCR v97 as you recommend, now everything is working perfectly. Thanks again for your help