Open jAchtzehn opened 5 years ago
I've actually just created a workaround for this issue by changing lines 167-179 in model.py to:
if isdefined(self.inputs.mask_image):
# SPM doesn't handle explicit masking properly, especially
# when you want to use the entire mask image
postscript = "load "+ self.inputs.spm_mat_dir + "/SPM.mat;\n"
postscript += ("SPM.xM.VM = spm_vol('%s');\n" % simplify_list(
self.inputs.mask_image))
postscript += "SPM.xM.I = 0;\n"
postscript += "SPM.xM.T = [];\n"
postscript += ("SPM.xM.TH = ones(size(SPM.xM.TH))*(%s);\n" %
self.inputs.mask_threshold)
postscript += ("SPM.xM.xs = struct('Masking', "
"'explicit masking only');\n")
postscript += "save " + os.getcwd() + "/SPM.mat SPM;\n"
But is that OK? The SPM.mat file's size is now drastically changed...
@jAchtzehn - are you using this within a Nipype workflow? with nodes? if so, the SPM.mat file is passed along and copied by the underlying nipype mechanisms.
Of course! That is the reason why I am so confused that there is a temporary SPM.mat file created in the default MATLAB folder (I have a cd command specified in my startup.m). If I delete that cd command the SPM.mat file is created in the folder I last opened with MATLAB.
To summarize the issue and make it a bit cleared (I hope):
When you add an explicit mask to the level1design with "mask image" input, line 167-179 are inserted in the .m file of level1design.m. These lines try to load an SPM.mat file created either (i) in the last opened MATLAB folder or (ii) in the folder specified in the startup.m file of MATLAB, I will refer to this folder as the "default MATLAB folder". It is then saved at the same position.
How this SPM.mat file is created in the first place and how this SPM.mat file makes its way back into the folder of the level1design node (or the following estimatemodel node) is not clear to me.
The issues this creates:
Now if we change the "spm_mat_dir" input of level1design, the SPM.mat file is created in that directory correctly, but the line "postscript = "load SPM;\n"" still means that the script will try to load an SPM.mat file that is in the default MATLAB folder, resulting in an error.
I have to run a trial-wise beta estimation, so for each trial I have a separate GLM. This means I run several trial analyses in parallel (up to 8). At some point, 8 different level1design nodes are running simultaneously and are trying to load/save the same SPM.mat file because it is created in the default MATLAB folder! After x amounts of analyses, this results in a corrupted SPM.mat file and all other trials afterwards crash.
I've done some further digging today: It seems that lines lines 167-179 in interfaces/spm/model.py are intended to load the SPM.mat inside the level1design work folder and change its contents so explicit masking is used properly by SPM (as it also says in the comments just above these lines). However, this does currently not work! After the level1design node is completed, the SPM.mat file in this folder (and also the consecutive SPM.mat file from the estimatemodel node) do not have these updated contents and explicit masking is not applied. This has caused me some trouble as I was wondering why my analyses showed up with NaN voxel (because they didn't pass the variance threshold) even though I set mask_threshold to '-Inf'!
@jAchtzehn - i'm not sure i can follow what's going wrong here.
here is an example of how a mask file is passed on to level1design:
could you please post your code somewhere where we can take a look? that may be better my comprehension.
I'm getting errors if I set explicit mask at level 1 design as well. Code fails at the subsequent estimation step (see below). Mask seems to be specified correctly in SPM.xM, but SPM.xVol is not being set (probably in the design step?), which causes a failure in the estimation step.
190225-17:07:02,495 nipype.workflow WARNING:
[Node] Error on "repsup_susan6_ydiffbinsame0_l1_l1pipeline.analysis.estimate" (/media/hstojic/dataneuro/fnclearning_fmri/dProcessed/nipype_work/repsup_susan6_ydiffbinsame0_l1_l1pipeline/analysis/_subject_s057/estimate)
[Node] Error on "repsup_susan6_ydiffbinsame0_l1_l1pipeline.analysis.estimate" (/media/hstojic/dataneuro/fnclearning_fmri/dProcessed/nipype_work/repsup_susan6_ydiffbinsame0_l1_l1pipeline/analysis/_subject_s057/estimate)
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-14-337b031fb0f4> in <module>()
1 l1pipeline.run(
2 'MultiProc',
----> 3 plugin_args = {'n_procs': pars['resources']['n_cores']}
4 )
/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/workflows.pyc in run(self, plugin, plugin_args, updatehash)
593 if str2bool(self.config['execution']['create_report']):
594 self._write_report_info(self.base_dir, self.name, execgraph)
--> 595 runner.run(execgraph, updatehash=updatehash, config=self.config)
596 datestr = datetime.utcnow().strftime('%Y%m%dT%H%M%S')
597 if str2bool(self.config['execution']['write_provenance']):
/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/plugins/base.pyc in run(self, graph, config, updatehash)
160 if result['traceback']:
161 notrun.append(
--> 162 self._clean_queue(jobid, graph, result=result))
163 else:
164 self._task_finished_cb(jobid)
/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/plugins/base.pyc in _clean_queue(self, jobid, graph, result)
222
223 if str2bool(self._config['execution']['stop_on_first_crash']):
--> 224 raise RuntimeError("".join(result['traceback']))
225 crashfile = self._report_crash(self.procs[jobid], result=result)
226 if jobid in self.mapnodesubids:
RuntimeError: Traceback (most recent call last):
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
result['result'] = node.run(updatehash=updatehash)
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
result = self._run_interface(execute=True)
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
return self._run_command(execute)
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
result = self._interface.run(cwd=outdir)
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 521, in run
runtime = self._run_interface(runtime)
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/spm/base.py", line 377, in _run_interface
results = self.mlab.run()
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 521, in run
runtime = self._run_interface(runtime)
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/matlab.py", line 170, in _run_interface
self.raise_exception(runtime)
File "/home/hstojic/.pyenv/nipy/local/lib/python2.7/site-packages/nipype/interfaces/base/core.py", line 970, in raise_exception
).format(**runtime.dictcopy()))
RuntimeError: Command:
/home/hstojic/matlab/bin/matlab -nodesktop -nosplash -singleCompThread -r "addpath('/media/hstojic/dataneuro/fnclearning_fmri/dProcessed/nipype_work/repsup_susan6_ydiffbinsame0_l1_l1pipeline/analysis/_subject_s057/estimate');pyscript_estimatemodel;exit"
Standard output:
< M A T L A B (R) >
Copyright 1984-2016 The MathWorks, Inc.
R2016b (9.1.0.441655) 64-bit (glnxa64)
September 7, 2016
To get started, type one of these: helpwin, helpdesk, or demo.
For product information, visit www.mathworks.com.
executing gpml startup script...
Executing pyscript_estimatemodel at 25-Feb-2019 17:06:43:
----------------------------------------------------------------------------------------------------
MATLAB Version: 9.1.0.441655 (R2016b)
MATLAB License Number: 649021
Operating System: Linux 4.11.0-14-generic #20~16.04.1-Ubuntu SMP Wed Aug 9 09:06:22 UTC 2017 x86_64
Java Version: Java 1.7.0_60-b19 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
----------------------------------------------------------------------------------------------------
MATLAB Version 9.1 (R2016b)
Simulink Version 8.8 (R2016b)
Bioinformatics Toolbox Version 4.7 (R2016b)
Communications System Toolbox Version 6.3 (R2016b)
Computer Vision System Toolbox Version 7.2 (R2016b)
Control System Toolbox Version 10.1 (R2016b)
Curve Fitting Toolbox Version 3.5.4 (R2016b)
DSP System Toolbox Version 9.3 (R2016b)
Database Toolbox Version 7.0 (R2016b)
Datafeed Toolbox Version 5.4 (R2016b)
Econometrics Toolbox Version 3.5 (R2016b)
Financial Instruments Toolbox Version 2.4 (R2016b)
Financial Toolbox Version 5.8 (R2016b)
Fixed-Point Designer Version 5.3 (R2016b)
Fuzzy Logic Toolbox Version 2.2.24 (R2016b)
Global Optimization Toolbox Version 3.4.1 (R2016b)
HDL Coder Version 3.9 (R2016b)
Image Acquisition Toolbox Version 5.1 (R2016b)
Image Processing Toolbox Version 9.5 (R2016b)
Instrument Control Toolbox Version 3.10 (R2016b)
LTE System Toolbox Version 2.3 (R2016b)
MATLAB Coder Version 3.2 (R2016b)
MATLAB Compiler Version 6.3 (R2016b)
MATLAB Compiler SDK Version 6.3 (R2016b)
MATLAB Report Generator Version 5.1 (R2016b)
Mapping Toolbox Version 4.4 (R2016b)
Model Predictive Control Toolbox Version 5.2.1 (R2016b)
Neural Network Toolbox Version 9.1 (R2016b)
Optimization Toolbox Version 7.5 (R2016b)
Parallel Computing Toolbox Version 6.9 (R2016b)
Partial Differential Equation Toolbox Version 2.3 (R2016b)
Phased Array System Toolbox Version 3.3 (R2016b)
Robotics System Toolbox Version 1.3 (R2016b)
Robust Control Toolbox Version 6.2 (R2016b)
Signal Processing Toolbox Version 7.3 (R2016b)
SimBiology Version 5.5 (R2016b)
Simscape Version 4.1 (R2016b)
Simscape Multibody Version 4.9 (R2016b)
Simscape Power Systems Version 6.6 (R2016b)
Simulink 3D Animation Version 7.6 (R2016b)
Simulink Coder Version 8.11 (R2016b)
Simulink Control Design Version 4.4 (R2016b)
Simulink Design Optimization Version 3.1 (R2016b)
Simulink Report Generator Version 5.1 (R2016b)
Simulink Verification and Validation Version 3.12 (R2016b)
Stateflow Version 8.8 (R2016b)
Statistical Parametric Mapping Version 6906 (SPM12)
Statistics and Machine Learning Toolbox Version 11.0 (R2016b)
Symbolic Math Toolbox Version 7.1 (R2016b)
System Identification Toolbox Version 9.5 (R2016b)
Trading Toolbox Version 3.1 (R2016b)
Wavelet Toolbox Version 4.17 (R2016b)
SPM version: SPM12 Release: 6906
SPM path: /home/hstojic/matlab2018a/toolbox/spm12/spm.m
------------------------------------------------------------------------
Running job #1
------------------------------------------------------------------------
Running 'Model estimation'
SPM12: spm_spm (v6842) 17:06:52 - 25/02/2019
========================================================================
SPM12: spm_est_non_sphericity (v6827) 17:06:58 - 25/02/2019
========================================================================
Failed 'Model estimation'
Reference to non-existent field 'xVol'.
In file "/home/hstojic/matlab2018a/toolbox/spm12/spm_est_non_sphericity.m" (v6827), function "spm_est_non_sphericity" at line 105.
In file "/home/hstojic/matlab2018a/toolbox/spm12/spm_spm.m" (v6842), function "spm_spm" at line 431.
In file "/home/hstojic/matlab2018a/toolbox/spm12/config/spm_run_fmri_est.m" (v5809), function "spm_run_fmri_est" at line 33.
The following modules did not run:
Failed: Model estimation
Standard error:
MATLAB code threw an exception:
Job execution failed. The full log of this run can be found in MATLAB command window, starting with the lines (look for the line showing the exact #job as displayed in this error message)
------------------
Running job #1
------------------
File:
Name:MATLABbatch system
Line:0
Return code: 0
I updated to the latest version of SPM12 (last update was in November 2018) and the errors disappear.
In fact it seems that SPM now does exactly what code in model.py lines 167 - 179 does, so this part will become unnecessary in the future.
I can verify that. Just manually commented out lines lines 167-179 in interfaces/spm/model.py and the SPM.mat files are identically.
@satra: please review these lines of code. They now seem to be redundant (with the latest SPM version 7487) and seem to be causing trouble on macOS systems such as mine. As I said, the extra steps coded in these lines will create a temporary SPM.mat file in the startup folder of matlab. This is not a problem unless running multiple analyses in parallel. It seems that in this case multiple MATLAB processes want to read/write the same SPM.mat file and this causes level1design to fail!
@jAchtzehn - there seems to be two issues here:
if this is indeed fixed in SPM12, we can change this line:
https://github.com/nipy/nipype/blob/master/nipype/interfaces/spm/model.py#L175
to
if isdefined(self.inputs.mask_image) and '12' not in self.version.split('.')[0]
note: if this is true only of a specific revision of SPM 12, we would need to check for that more precise version.
this appears to be a function of the python script, and will be true of any script that tries to operate on the same file. this is exactly why we encourage using workflows that try to isolate directories. the other possible caveat is code that exists in "startup.m" in MATLAB. this will be executed every time matlab runs, so if this code is changing directories then you should remove those lines.
the nipype interface does make the assumption that it loads the SPM.mat file from whichever path is available to MATLAB. so if there is an SPM.mat file somewhere in MATLAB's path, that would get loaded first.
@jAchtzehn - there seems to be two issues here:
- the redundancy of setting the mask image in certain recent versions
if this is indeed fixed in SPM12, we can change this line:
https://github.com/nipy/nipype/blob/master/nipype/interfaces/spm/model.py#L175
to
if isdefined(self.inputs.mask_image) and '12' not in self.version.split('.')[0]
note: if this is true only of a specific revision of SPM 12, we would need to check for that more precise version.
Agreed and should be checked with the SPM developers I suppose?
- reading and writing the SPM.mat file in the same location
this appears to be a function of the python script, and will be true of any script that tries to operate on the same file. this is exactly why we encourage using workflows that try to isolate directories. the other possible caveat is code that exists in "startup.m" in MATLAB. this will be executed every time matlab runs, so if this code is changing directories then you should remove those lines.
I have indeed removed my code (which is just a cd ... command) from my startup.m file. The SPM.mat file then gets created at the last opened folder when MATLAB was closed. Of course, the problem of multiple instances of MATLAB trying to read/write the file persists even in that case. Please keep in mind that I have not created any special workflows, I am very close to the many examples of firstlevel analysis found in nipype's documentation. The only real difference is that I include a mask image as an input to the level1design node.
the nipype interface does make the assumption that it loads the SPM.mat file from whichever path is available to MATLAB. so if there is an SPM.mat file somewhere in MATLAB's path, that would get loaded first.
As I understand it, the .m file that is created to execute the level1design command sets the current folder to the appropriate node folder inside nipype's working directory, right? However, the extra lines added when an explicit mask is used to not seem to honour this, instead creating the temporary SPM.mat file in another location, thus creating the problem of multiple MATLAB instances accessing the same file.
Summary
Actually there are two issues caused by the same problem:
issue: When using a mask image for SPM level1design and also changing the input "spm_mat_dir" the Level1Design node fails.
issue: Even without changing the spm_mat_dir input, a problem arises when running different analyses in parallel: because each analysis is loading/writing the same SPM.mat file (in the default MATLAB folder) a file I/O error is happening at random (after ~50 runs).
Actual behavior
During level1design with an explicit mask as an input, the script is trying to load a (temporary?!) SPM.mat file in the folder (i) specified by "spm_mat_dir" or (ii) if that is left undefined in the default MATLAB folder (defined by startup.m or the last folder when MATLAB was last closed). However this SPM.mat file is always loaded at the default MATLAB folder, regardless of the "spm_mat_dir" input.
"/Users/jachtzehn/matlab/" is my default MATLAB startup folder.
Expected behavior
The SPM.mat file should be saved and loaded in the folder specified by "spm_mat_dir" input or at least in a node specific folder.
How to replicate the behavior
Run a level1design on any functional data included an explicit mask (defined by "mask_image" input) + change the "spm_mat_dir" folder to anything other than default.
Script/Workflow details
When "mask_image" input if defined, model.py of the spm interface adds a couple of lines to the .m file that load a SPM.mat file, modify its contents and save it again. The respective code in model.py are: line 167 - 179:
As you can see, this only adds "load SPM" to the script, so during runtime the script loads the SPM.mat file from the default Matlab folder, not the level1design node folder.
Platform details: