Is formatWindows.m necessary?

[jackgoffinet]

From what I can tell formatWindows.m does a few things:

• It breaks up the LFP recordings into smaller chunks.
• It extracts some information from the '*_LFP.mat' filenames and the Excel file and associates each chunk with this info.
• It fills inactive channels from the CHANACTIVE vector into NaNs.
• Then it saves everything in a new file.

From what I can tell, this is mostly a big reshaping and re-saving of the LFP arrays. Is this a necessary step in the pipeline? It's a big memory cost and there's very little computation happening -- it seems like it's not worth saving this intermediate step.

Are there other reasons it's good to re-save the LFPs in this format? @neil-gallagher and @mhunterklein , do you have any thoughts on this?

[neil-gallagher]

We could avoid saving the reshaped LFPs by just combining the reshaping code into one of the other parts of the pipeline. Historically we haven't done this because formatWindows is a recent addition to the pipeline and not all experiments have used it to parse the data into the 3-D array format expected by preprocessData. I'm still not sure if combining these two pieces of code would be a good idea, because some experiments will still not be well handled by formatWindows and will need to be reshaped manually.

The re-shaping part is necessary for the way we handle the data moving forward. We could re-write the preprocessing part of the pipeline code to handle the data saved in a way that is not reshaped into windows (it's something we've considered in the past because it would also make the filtering portion of the preprocessing code more efficient). The major problem is that this would be a big hassle to re-write and the gains are not that substantial (unless we want to move to overlapping windows) because we will still need to reshape the data before computing features associated with each sliding window.

[jackgoffinet]

Cool, so it sounds like it's probably not worth changing this right now.

If it does get changed sometime, though, it might be good to pass parameters of the reshaping operation to preprocessData instead of data itself. Then there could be a default function that makes the usual windows like in the current formatWindows. Otherwise you would have to make the parameters yourself to pass in.