PSOM fails but when deactivated it works fine

[arnodelorme]

This is the error I get. When I deactivate PSOM in limo_settings_script, the pipeline runs fine. Weird

**********************************************
09-Sep-2022 23:18:53 : The job has FAILED
Total time used to process the job : 0.59 sec.
**********************************************

ans =

    '
     ******************************
     Log of the (matlab) job : design
     Started on 09-Sep-2022 23:18:52
     User: arno
     host : arnauds-mbp.lan
     system : unix
     ******************************
     The job starts now !
     ********************
     reloading data ..
     pop_loadset(): loading file /System/Volumes/Data/data/matlab/compare_eeg_pipelines/ds003061/derivatives/eeglab/sub-005/eeg/sub-005_task-P300_run-2_eeg_ses-1_design1.set ...
     pop_loadset(): Incompatible with new format, trying old format and converting...

     ********************
     Something went bad ... the job has FAILED !
     The last error message occured was :
     Unrecognized field name "cellArray".
     File /System/Volumes/Data/data/matlab/eeglab/functions/popfunc/pop_loadset.m at line 284
     File /System/Volumes/Data/data/matlab/eeglab/functions/popfunc/pop_loadset.m at line 152
     File /System/Volumes/Data/data/matlab/eeglab/plugins/limo_tools/limo_batch_design_matrix.m at line 24
     File /System/Volumes/Data/data/matlab/eeglab/plugins/limo_tools/external/psom/psom_run_job.m at line 214
     File /System/Volumes/Data/data/matlab/eeglab/plugins/limo_tools/external/psom/psom_run_job.m at line 128
     File /System/Volumes/Data/data/matlab/eeglab/plugins/limo_tools/external/psom/psom_run_script.m at line 437
     File /System/Volumes/Data/data/matlab/eeglab/plugins/limo_tools/external/psom/psom_run_script.m at line 323
     File /System/Volumes/Data/data/matlab/eeglab/plugins/limo_tools/external/psom/psom_pipeline_process.m at line 591
     File /System/Volumes/Data/data/matlab/eeglab/plugins/limo_tools/external/psom/psom_run_pipeline.m at line 417
     File /Applications/MATLAB_R2021b.app/toolbox/matlab/lang/parallel_function.m at line 837
     File /Applications/MATLAB_R2021b.app/toolbox/parallel/remoteParallelFunction.m at line 67

     ****************
     Checking outputs
     ****************
     The output file or directory /System/Volumes/Data/data/matlab/compare_eeg_pipelines/ds003061/derivatives/eeglab/sub-005/eeg/ses-1/ds003061_design1_GLM_Channels_Time_WLS/Yr.mat has not been generated!

     **********************************************
     09-Sep-2022 23:18:53 : The job has FAILED
     Total time used to process the job : 0.59 sec.
     **********************************************
     '

[arnodelorme]

This is reproducible

[arnodelorme]

This is without using PSOM (even if it mentions it below)

[CPernet]

well even with your added debugging section (not needed IMO, because I debug just fine by calling the pipeline) it still relies on psom, the difference being that your version doesn't catch errors and actually stops computing and doesn't parallelise (hence in other thread my cst change to have it on by default)

[CPernet]

all my test passed, what data are you using and which version of eeglab (dev?)

[arnodelorme]

I am using EEGLAB dev with this dataset

https://openneuro.org/datasets/ds003061

1. Import with BIDS (run 2 only, I deleted all other runs)
2. Remove channels 65:74
3. Highpass at 0.5
4. Extract epochs standard, oddball, and noise

[STUDY, ALLEEG] = pop_importbids('/System/Volumes/Data/data/matlab/compare_eeg_pipelines/ds003061', 'eventtype','value','bidsevent','on','bidschanloc','on','outputdir','/System/Volumes/Data/data/matlab/compare_eeg_pipelines/ds003061/derivatives/eeglab');
ALLEEG = pop_select( ALLEEG,'nochannel',{'EXG1','EXG2','EXG3','EXG4','EXG5','EXG6','EXG7','EXG8','GSR1','GSR2','Erg1','Erg2','Resp','Plet','Temp'});
ALLEEG = pop_eegfiltnew( ALLEEG,'locutoff',0.5);
ALLEEG = pop_epoch( ALLEEG,{'noise','oddball_with_reponse','standard'},[-0.5 1] ,'epochinfo','yes');
STUDY = std_checkset(STUDY, ALLEEG);
STUDY = std_makedesign(STUDY, ALLEEG, 1, 'name','STUDY.design 1','delfiles','off','defaultdesign','off','variable1','type','values1',{'noise','oddball_with_reponse','standard'},'vartype1','categorical');
CURRENTSTUDY = 1; EEG = ALLEEG; CURRENTSET = [1:length(EEG)];

[CPernet]

it fails at EEGLIMO=pop_loadset([LIMO.data.data_dir filesep LIMO.data.data]); because in pop_loadset eegset = EEG.cellArray; but .cellArray doesn't exist

how can doing for ... psom_pipeline_debug work and doing parfor ... psompipeline not work this is the exact same code called, just not using fancy parallel or error catch stuff

[arnodelorme]

Yes, really weird. I cannot debug though. I would need to put a breakpoint in pop_loadset to see why the file is corrupted, but the parallel loop will prevent that.

The error above means that the EEG data is corrupted. It is trying to access EEG.cellArray because it is trying to decode as a very old EEG format. The EEG structure is probably missing the data field.

[CPernet]

or it is when calling psom_pipeline - a point break here https://github.com/LIMO-EEG-Toolbox/limo_tools/blob/master/limo_batch.m#L514 then execute subject = 1, limo_batch_import_data(pipeline(subject).import.files_in,pipeline(subject).import.opt.cat,pipeline(subject).import.opt.cont,pipeline(subject).import.opt.defaults) see if this still cause the problem

compare with psom_pipeline_debug(pipeline(subject));