Error allocating huge Memory

[ijekt]

Greetings,

i am exploring different packages and methods for preprocessing of meg/eeg data and i am really thankful to have found this translation of matlab, because i only use python so far. Although i following the documentation, i run into an error that doesn't make much sense to me. At this point i am not sure if its because of a wrong format of the data i put into the _dssline() and i can only think about MNE causing this problem.

Background of the data:

• raw data has about 3gb size
• contains 30eeg, 102 meg, 204 grad channels
• read in the .fif file with python MNE

this is the error i get after running the function:

_numpy.core._exceptions.ArrayMemoryError: Unable to allocate 32.4 TiB for an array with shape (2111000, 2111000) and data type float64

32,4TiB seems to be so huge, that i can't really understand how the data is transformed to such a big array. i allready tried to use only the 30 eeg channels and allso changing parameters nkeep, blocksize or using the _dss_line_iter()_ does not change the amount of memory shown in the error.

i appreciate some advice what i am missing and some information if may MNE is causing the problem.

Thank you !

[nbara]

Hey @ijekt, thanks for your message.

I would need the following info to help you:

• exact code you used to get this error
• the exact shape of your data before you feed it into dss_line()

The fact that the error mentions a square matrix of shape (2111000, 2111000) leads me to think that you have transposed the times and channels dimensions (therefore dss_line is trying to compute covariance matrices of shape (n_times, n_times) instead of (n_channels, n_channels).

Could that be the case? Either way, let me know :)

The reason I am asking is because meegkit uses a slightly different shape convention (n_times, n_channels, n_trials) than MNE (n_trials, n_channels, n_times)

[ijekt]

Hello nbara,

that makes totally sense and after transposing the datashape from (343, 2111000) to (2111000, 343) it runs. Thank you very much for pointing out the different shape convention ! That was the information is was missing.

Greetings

[ijekt]

is it a problem if the data has no n_trials ?

[nbara]

Thank you very much for pointing out the different shape convention ! That was the information is was missing.

Hey, glad I could help

is it a problem if the data has no n_trials ?

It shouldn't be. Let me know if you experience a bug