Provide better documentation for SPM standalone setup

ghisvail commented 1 year ago

Discussed in https://github.com/aramis-lab/clinica/discussions/819

^{Originally posted by **AudreyDuran** November 28, 2022} Hi, I am trying to run `t1-volume` pipeline with SPM12 standalone and clinica v0.7.2, on a custom dataset. Here are some environment variables values defined in my `~/.bashrc`: ``` SPM_HOME="/opt/spm12" SPM_VERSION=12 SPM_REVISION=r7771 MATLAB_VERSION=R2019b MCR_VERSION=v97 MCR_UPDATE=9 MCR_INHIBIT_CTF_LOCK=1 MATLABCMD="${SPM_HOME}/run_spm12.sh /opt/mcr/${MCR_VERSION} script" ``` When I run the pipeline, I get the following output: ``` 2022-11-28 10:17:48,336:INFO:The t1-volume pipeline is divided into 4 parts: t1-volume-tissue-segmentation pipeline: Tissue segmentation, bias correction and spatial normalization to MNI space t1-volume-create-dartel pipeline: Inter-subject registration with the creation of a new DARTEL template t1-volume-dartel2mni pipeline: DARTEL template to MNI t1-volume-parcellation pipeline: Atlas statistics 2022-11-28 10:17:55,639:INFO:Part 1/4: Running t1-volume-segmentation pipeline. 2022-11-28 10:25:57,830:INFO:The pipeline will be run on the following 1282 image(s): 2022-11-28 10:25:57,830:INFO: sub-00098E7F | ses-w96, ses-w0, 2022-11-28 10:25:57,830:INFO: sub-00331D3D | ses-w0, ses-w96, 2022-11-28 10:25:57,830:INFO: sub-003D930B | ses-w0, ses-w96, 2022-11-28 10:25:57,830:INFO: sub-00438273 | ses-w96, ses-w0, 2022-11-28 10:25:57,830:INFO: sub-006ACF86 | ses-w96, ses-w0, 2022-11-28 10:25:57,830:INFO: sub-0095A525 | ses-w0, ses-w96, 2022-11-28 10:25:57,830:INFO: sub-00DBEADC | ses-w0, ses-w96, 2022-11-28 10:25:57,830:INFO: sub-01BA0A03 | ses-w96, ses-w0, 2022-11-28 10:25:57,830:INFO: sub-01D4AE69 | ses-w0, ses-w96, 2022-11-28 10:25:57,830:INFO: sub-0271F164 | ses-w0, ses-w96, 2022-11-28 10:25:57,830:INFO: sub-02D7D496 | ses-w96, ses-w0, 2022-11-28 10:25:57,831:INFO: sub-0391403C | ses-w96, ses-w0, 2022-11-28 10:25:57,831:INFO: sub-03BCF734 | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-04550C9F | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-047D2FA8 | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-04CB17EC | ses-w96, ses-w0, 2022-11-28 10:25:57,831:INFO: sub-05A8B309 | ses-w96, ses-w0, 2022-11-28 10:25:57,831:INFO: sub-06152DED | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-0693D50B | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-069B4AC2 | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-07AB59AD | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-07E02FCB | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-07E24E6C | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: sub-0844A958 | ses-w96, ses-w0, 2022-11-28 10:25:57,831:INFO: sub-08BB1347 | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO: ... 2022-11-28 10:25:57,831:INFO: sub-FFCE393E | ses-w0, ses-w96, 2022-11-28 10:25:57,831:INFO:The pipeline will last approximately 10 minutes per image. Traceback (most recent call last): File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/utils/spm_docs.py", line 49, in _strip_header index = doc.index(hdr) ValueError: substring not found The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/workspace/thera/clinicaEnv/bin/clinica", line 8, in sys.exit(main()) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/cmdline.py", line 89, in main cli() File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1130, in __call__ return self.main(*args, **kwargs) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1055, in main rv = self.invoke(ctx) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1657, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke return ctx.invoke(self.callback, **ctx.params) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/t1_volume/t1_volume_cli.py", line 91, in cli ctx.invoke( File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/click/core.py", line 760, in invoke return __callback(*args, **kwargs) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/t1_volume_tissue_segmentation/t1_volume_tissue_segmentation_cli.py", line 68, in cli else pipeline.run() File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/engine.py", line 261, in run self.build() File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/engine.py", line 25, in func_wrapper res = func(self, *args, **kwargs) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/engine.py", line 232, in build self.build_core_nodes() File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/clinica/pipelines/t1_volume_tissue_segmentation/t1_volume_tissue_segmentation_pipeline.py", line 163, in build_core_nodes new_segment = npe.Node(spm.NewSegment(), name="2-SpmSegmentation") File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/preprocess.py", line 1873, in __init__ _local_version = SPMCommand().version File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/base.py", line 310, in __init__ super(SPMCommand, self).__init__(**inputs) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 197, in __init__ unavailable_traits = self._check_version_requirements( File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 295, in _check_version_requirements if names and self.version: File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/base.py", line 358, in version info_dict = Info.getinfo( File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/interfaces/spm/base.py", line 239, in getinfo out = sd._strip_header(out.runtime.stdout) File "/workspace/thera/clinicaEnv/lib/python3.8/site-packages/nipype/utils/spm_docs.py", line 51, in _strip_header raise IOError("This docstring was not generated by Nipype!\n") from e OSError: This docstring was not generated by Nipype! ``` Do you have any idea about what the problem is ? I also get the same `ValueError` when running ``` from nipype.interfaces import spm matlab_cmd = ''/opt/spm12/run_spm12.sh /opt/mcr/v97 script'' spm.SPMCommand.set_mlab_paths(matlab_cmd=matlab_cmd, use_mcr=True) spm.SPMCommand().version ``` You can find bellow some platform details: > {‘nibabel_version’: ‘2.5.2’, > ‘nipype_version’: ‘1.8.5’, > ‘numpy_version’: ‘1.23.3’, > ‘pkg_path’: ‘/clinicaEnv/lib/python3.8/site-packages/nipype’, > ‘scipy_version’: ‘1.9.2’, > ‘sys_executable’: ‘/clinicaEnv/bin/python’, > ‘sys_platform’: ‘linux’, > ‘sys_version’: ‘3.8.5 (default, Jan 27 2021, 15:41:15) \n[GCC 9.3.0]’, > ‘traits_version’: ‘6.3.2’} Thanks a lot for your help !!

ghisvail commented 1 year ago

The solution was to set FORCE_SPMMCR which is used as a fallback to setting use_mcr to True.

This raises a wider discussion regarding our support for both MATLAB and standalone distributions of SPM. Support for both can either be controlled explicitly with the use_mcr and matlab_cmd arguments, or implicitly with the FORCE_SPMMCR and MATLABCMD envvars. See Nipype's documentation.

Both have their pros and cons as always.

NicolasGensollen commented 1 year ago

I haven't looked into it, so apologies if this is a stupid question, but isn't this function supposed to handle the SPM standalone config ?

https://github.com/aramis-lab/clinica/blob/3a4981bfb3e8d4f3d4af6467887098c6fd29d07b/clinica/utils/spm.py#L101-L129

ghisvail commented 1 year ago

so apologies if this is a stupid question, but isn't this function supposed to handle the SPM standalone config ?

The question is absolutely valid. Indeed this function is supposed to handle the SPM config. But there are multiple flaws I think:

It is always paired with another utility function called spm_standalone_is_available within which there is some duplicated logic wrt the detection of the envvars. A path to improvement would be to refactor them both into a use_spm_standalone_if_available function.
As stated in the Nipype's documentation: "In order to use the standalone MCR version of spm, you need to ensure that the following commands are executed at the beginning of your script". Which makes sense since the proposed solution monkey-patch (yikes) the base SPMCommand from which all SPM interfaces inherit from. But due to the complicated logic in our pipelines, it's not impossible that some SPM interface has already been imported before calling use_spm_standalone.
The answered solution works because Nipype's envvars take precedence over base class setup.

ghisvail commented 1 year ago

In my opinion, instead of depending on monkey-patching, we should just let Nipype logic handle it:

matlab_cmd (str): Sets the default matlab command. If None, the value of the environment variable SPMMCRCMD will be used if set and use_mcr is True or the environment variable FORCE_SPMMCR is set. If one of FORCE_SPMMCR or SPMMCRCMD is not set, the existence of the environment variable MATLABCMD is checked and its value is used as the matlab command if possible. If none of the above was successful, the fallback value of ‘matlab -nodesktop -nosplash’ will be used.

i.e.:

To support SPM standalone only, set SPMMCRCMD and FORCE_SPMMCR
To support SPM MATLAB only, set MATLABCMD and do not set FORCE_SPMMCR
To support both with a fallback or a switch, set MATLABCMD and set or unset FORCE_SPMMCR

NicolasGensollen commented 1 year ago

https://github.com/aramis-lab/clinica/blob/444e4a4e711d6a373fae07412fa378da1e356572/clinica/utils/spm.py#L80-L98