Open gladomat opened 7 years ago
@gladomat these -v 7.3 files are not implemented in scipy
's reader yet - however we should be able to read them in with h5py
. The spm offenders within nipype seem to be EstimateModel
and EstimateContrast
.
we would still want to attempt to open the mat file with scipy.io.loadmat
first, but if that raises an NotImplementedError
we can opening with h5py.File
. Do you want to give this PR a crack?
Do you want to give this PR a crack?
You mean do I want to add the implementation myself? I can try. It would be an error catcher I suppose. I'll look into it and report how far I've gotten.
yes, essentially two try/except blocks (one for importing h5py
, one for using h5py.File()
) - if you have any problems I can help out/implement it down the line
I'll do it but it may take a little while. Since I'm the only one using it, I guess it can wait.
I also ran in to this issue after needing to use v7.3 .mat files (first level analysis, but there was a large number of very long runs). Adding try/except blocks with h5py
seems simple enough, and if this is still open I wouldn't mind trying to contribute.
Is there anything else I should be looking to modify besides the EstimateModel
and EstimateContrast
from spm/model.py?
@gladomat @mgxd checking in about the status of this. If it'd be useful, I'm still happy to submit a possible modification to spm/model.py that would allow reading of v7.3 .mat files.
@psadil - these are the occurrences i found.
$ grep -rn loadmat nipype
nipype/algorithms/misc.py:358: in_dict = sio.loadmat(op.abspath(self.inputs.in_file))
nipype/algorithms/misc.py:397: in_dict = sio.loadmat(op.abspath(self.inputs.in_file))
nipype/algorithms/rapidart.py:756: spmmat = sio.loadmat(self.inputs.spm_mat_file, struct_as_record=False)
nipype/interfaces/spm/model.py:287: spm = sio.loadmat(self.inputs.spm_mat_file, struct_as_record=False)
nipype/interfaces/spm/model.py:489: spm = sio.loadmat(self.inputs.spm_mat_file, struct_as_record=False)
it may be worthwhile to create a loadmat
function in utils and reuse it in these places:
def loadmat(filename, **kwargs):
try:
return sio.loadmat(filename, **kwargs)
except NotImplementedError:
import h5py as h5
mat = None
with h5.File(filename, 'r') as f:
...
return mat
for specific mappings of MATLAB to HDF5 see this: https://github.com/frejanordsiek/hdf5storage
i think for the purpose of these interfaces h5py
should be sufficient without going to hdf5storage
.
we also use sio.savemat at different places, but i don't think most of those would end up needing to switch at this point.
I've implemented a library that can read the mat 7.3
files, have a look here:
https://github.com/skjerns/mat7.3
or install via pip pip install mat73
and load with
import mat73
data_dict = mat73.loadmat('data.mat')
I've encountered the same problem, my model is too large, Matlab saves SPM.mat in v7.3 and then nipype cannot handle it. I tried using mat73 but it throws "not a MATLAB datatype" errors for me when trying to read a v7.3 SPM.mat file.
If I'm not mistaken only the Vbeta
, Vcon
and Vspm
variables are used from SPM.mat in model.py
, so there is no need for a general v7.3 loadmat function, it is enough to read these variables.
So I made a fix by creating a load_spm_mat()
function similarly to @satra's suggestion, it reads Vbeta
, Vcon
and Vspm
in the except branch and formats them the same as scipy.io.loadmat, therefore it requires no changes in model.py
, except for replacing the loadmat()
function.
Maybe this is not the most elegant solution, but I can make a PR if you think this is a useful addition.
Summary
I'm running a Bayesian analysis using Per Síden's code using spatial priors and the resulting SPM.mat files are massive for ultra high field data (upwards of 2 GB). I had to change the SPM code to make the saves possible, for example in
spm_contrasts.m
line 313:save('SPM.mat', 'SPM', '-v7.3')
.In nipype, however, I get an error:
Actual behavior
Nipype crashes as it can't open or save -v 7.3 .mat files.
Expected behavior
Nipype should save and open -v 7.3 files natively.
How to replicate the behavior
I changed said line in
spm_contrasts.m
to save -v 7.3 .mat files and ran a second level analysis. At the calculate contrasts step I encounter the above error.Script/Workflow details
Sorry, it's kinda long. I'll shorten the subject list to 1 subject.
Platform details:
Linux 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
nipype version: 0.12.1
Execution environment
Native linux with conda installer python 2.7