search

NeuralAnalysis / PyalData

Repository for the Python implementation of the TrialData analysis library.
GNU General Public License v3.0
7 stars 9 forks source link

Missing support for v7.3 mat files #70

Closed raeedcho closed 3 years ago

raeedcho commented 3 years ago

Great work on this conversion so far all!

One thing I've noticed while going through the code is that in mat2dataframe, scipy.io.loadmat won't work with .mat files saved using '-v7.3', which is necessary for TrialData files >2GB in MATLAB. The v7.3 format is actually HDF5, which makes loading it a bit complicated, but this Python module seems to work pretty well at loading that sort of data into a python dict: mat73

I would submit a pull request with a proposed change, but I'm not super familiar with Python yet myself.

bagibence commented 3 years ago

Thank you! :)

If mat73.loadmat returns a dict with the same structure as scipy.io.loadmat, then I guess this should be an easy fix. The problem is I don't have MATLAB on my computer right now to save files in this format.

I would submit a pull request with a proposed change, but I'm not super familiar with Python yet myself.

No worries, I'll submit a fix that might work. Could you test if it does?

raeedcho commented 3 years ago

Sure, I can test it whenever you get around to it--no rush :slightly_smiling_face:

bagibence commented 3 years ago

Looks like there's already a discussion about including this in scipy https://github.com/scipy/scipy/issues/11351

In the meantime I added an optional dependency for mat73 and use its loadmat if scipy fails.

@raeedcho let me know if it works!

bagibence commented 3 years ago

Moving the conversation back to the issue instead of the pull request.

@catiafortunato I found this in mat73's readme:

Proprietary MATLAB types (e.g datetime, duriation, etc) are not supported. If someone tells me how to convert them, I'll implement that

I guess your data has some session date or something similar stored as a datetime. What's the field that gives you this error?

catiafortunato commented 3 years ago

I get these two errors: ERROR:root:ERROR: MATLAB type not supported: datetime, (uint32) and ERROR:root:ERROR: not a MATLAB datatype: <HDF5 dataset "data": shape (3022,), type "<f8">, (float64) ERROR:root:ERROR: not a MATLAB datatype: <HDF5 dataset "ir": shape (3022,), type "<u8">, (uint64) ERROR:root:ERROR: not a MATLAB datatype: <HDF5 dataset "jc": shape (151,), type "<u8">, (uint64)

catiafortunato commented 3 years ago

The first error seems to go away when the mat file contains only one struct inside. 🧐

catiafortunato commented 3 years ago

My dataset had data stored in sparse matrices. If re-stored in full matrices the code in #71 works

bagibence commented 3 years ago

Great! I'll just merge it then.

gnlup commented 1 year ago

I got the same error ERROR:root:ERROR: MATLAB type not supported: datetime, (uint32). My data has nested struct, what can I do to get rid of these errors?

AndSalaMove commented 1 year ago

I got the same error ERROR:root:ERROR: MATLAB type not supported: datetime, (uint32). My data has nested struct, what can I do to get rid of these errors?

+1