search

Nirstorm / nirstorm

Brainstorm plugin for fNIRS data analysis
GNU General Public License v3.0
34 stars 12 forks source link

[enh] Make all process compatible with raw files #219

Closed Edouard2laire closed 1 year ago

Edouard2laire commented 1 year ago

This PAr aims at improving the way files are handled. The objective is to keep the type of the files. So if the input is raw, the output is raw. If the input is epoched data, so is the output.

@rcassani, could you check what i am doing wrong? the output file is not registered to the database when creating a new raw file?

it would be nice if it was simpler from the API point of view to save/load either epoch data or raw files as from the process point of view, we don't really care if the file we are processing is raw or epoch. Currently, it leads to a lot of code duplication and case handling to make sure each process can work on either raw or epoched data...

rcassani commented 1 year ago

@rcassani, could you check what i am doing wrong? the output file is not registered to the database when creating a new raw file?

Done 👍

it would be nice if it was simpler from the API point of view to save/load either epoch data or raw files as from the process point of view, we don't really care if the file we are processing is raw or epoch. Currently, it leads to a lot of code duplication and case handling to make sure each process can work on either raw or epoched data...

If there are cases where only processing happens, the process could be written as Category Filter, the the data will be already loaded as the field A in sInput for data and raw. Only when run with data it will offer the option of over writing the input file

Edouard2laire commented 1 year ago

Thank you for the comment. The file are now registered in the database. However, i am not able to open it 


***************************************************************************
** Error: Line 83:  in_bst_data (line 83)
** Cannot load recordings file: "/Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat".
**  load
** Unable to read MAT-file /Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat. File might be corrupt.
** 
** Call stack:
** >in_bst_data.m at 83
** >bst_memory.m>LoadDataFile at 689
** >bst_memory.m at 73
** >view_timeseries.m at 107
** >tree_callbacks.m at 261
** >bst_call.m at 28
** >panel_protocols.m>CreatePanel/protocolTreeClicked_Callback at 125
** >bst_call.m at 28
** >panel_protocols.m>@(h,ev)bst_call(@protocolTreeClicked_Callback,h,ev) at 75
** 
***************************************************************************

it seems that the file is created but matlab cannot open it

Edouard2laire commented 1 year ago

@rcassani When using the filter type of process, is there an easy way to have access to the events (so we could remove those lines https://github.com/Nirstorm/nirstorm/pull/219/files#diff-1ab9716e183e1a953da0f256a555bfac4adad214f253515ee0e3c0534b9ad0d9L106-L113) and not to have to load the entire file a second time?

rcassani commented 1 year ago

Unfortunately, it is necessary to read the events from the file.

It can be faster by reading only that field for imported data:

sDataIn = in_bst_data(sInputs.FileName, 'Events');
events = sDataIn.Events;

For raw files it should be fast to read F (like in the pointed lines), as it is only metadata of the raw file, not the recordings.

Edouard2laire commented 1 year ago

Unfortunately, it is necessary to read the events from the file.

It can be faster by reading only that field for imported data:

sDataIn = in_bst_data(sInputs.FileName, 'Events');
events = sDataIn.Events;

For raw files it should be fast to read F (like in the pointed lines), as it is only metadata of the raw file, not the recordings.

Thank you. it seems that we can make minimal change to bst_process to have the event. would that be ok for you ? (see https://github.com/brainstorm-tools/brainstorm3/pull/655)

Thank you for the comment. The file are now registered in the database. However, i am not able to open it


***************************************************************************
** Error: Line 83:  in_bst_data (line 83)
** Cannot load recordings file: "/Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat".
**  load
** Unable to read MAT-file /Users/edelaire1/Documents/bst_db/TutorielNIRSTORM2023/data/Subject01/@rawsub-01_task-tapping_run-01_dOD/data_0raw_OD_231102_1717.mat. File might be corrupt.
** 
** Call stack:
** >in_bst_data.m at 83
** >bst_memory.m>LoadDataFile at 689
** >bst_memory.m at 73
** >view_timeseries.m at 107
** >tree_callbacks.m at 261
** >bst_call.m at 28
** >panel_protocols.m>CreatePanel/protocolTreeClicked_Callback at 125
** >bst_call.m at 28
** >panel_protocols.m>@(h,ev)bst_call(@protocolTreeClicked_Callback,h,ev) at 75
** 
***************************************************************************

it seems that the file is created but matlab cannot open it

Do you have any idea about that issue? Edit: it seems to be working :)

Edouard2laire commented 1 year ago

List of processes to adapt for continuous recording:

Preprocessing

MBLL

Global

rcassani commented 1 year ago

Hi @Edouard2laire, is my input already required for this PR?

Edouard2laire commented 1 year ago

Hello,

I think this is ready to be merged. Let me know if you want to review it or if I should merge it as is.

One issue I would like us to check is https://github.com/brainstorm-tools/brainstorm3/issues/656 as it is often important in nirs to check what the preprocessing steps are doing to the data so we would open the raw data and after motion correction to be sure the motion correction step worked well and it makes it necessary to have two continuous files open at the same time.

Edouard2laire commented 1 year ago

Accidentally closed the PR. Reopening here https://github.com/Nirstorm/nirstorm/pull/223