Fix: export to BIDS

[dominikwelke]

hi there! here is a PR to enhance automagics export to BIDS.. closes #47

it is based on bids version 1.6.0 see https://bids-specification.readthedocs.io/en/stable

notes:

• it generates correct bids names and folder nesting, regardless of the sourcefile names

• i added the automagic quality rating to the free form desc- section of the bids filename (e.g. sub-01_task-01_desc-gi_eeg.set)

• required (and recommeded) BIDS files are generated or copied over from source folders, required fields are added

• i decided for the folder structure with raw data in the base:

  basefolder
  -sub-X
  -sub-Y
  -...
  -dataset_description.json
  -derivatives
  --automagic
  ---code
  ---sub-X
  ---sub-Y
  ---...
  ---dataset_description.json

  because this allows for exports into already existing BIDS datasets.

another BIDS compatible option would be adding raw data into a sepearte folder:

basefolder
-rawdata
--sub-X
--sub-Y
--...
--dataset_description.json
-derivatives
--automagic
---code
---sub-X
---sub-Y
---...
---dataset_description.json

limitations so far:

• jsonwrite only supports char and special characters like / are not supported. this destroys links, filepaths etc, and relevant entries like n/a
• some recommended BIDS files are so far copied verbatim (no check, whether there were changes related to automagic)
• i just realized i forgot to deal with particpants.tsv / participants.json files
• the automagic pipeline version is not yet fetched from Project or Block but hardcoded
• templates for the required dataset_description.json and sidecar json (*_eeg.json) are by now hardcoded in the function. might be more elegant to load them, because one could update them easier
• for BIDS EEG a taskname MUST be specified. if this cannot be derived from existing BIDS names, i set it to unspecified. setting it to 01 might be another option
• i am not entirely sure whether we can call the plots *_photo.jpg because this naming is used for photographies of electrode placements on the participant..

happy to hear your feedback! i tested it with several small projects and in my setup everything works fine so far.. btw, i was messing with your naming conventions. feel free to suggest other variable names!

[ksgfan]

Hi Dominik

I am back and have more time for Automagic again. Thanks for the updates, looks great! I am going to test it in the next few days.

Best, Dawid

[ksgfan]

I tested the new version with our datasets and had to change 2 things:

1.

[~, ~] = evalc("pop_writebva(EEG,newResFile1)"); throws an error, because BIDS_fnameRoot is a 1x3 string array, not a string

K>> BIDS_taskName = "unspecified"; % default taskName
BIDS_fnameRoot = [BIDS_subjectName '_task-' BIDS_taskName];

K>> BIDS_fnameRoot

BIDS_fnameRoot = 

  1×3 string array

    "sub-AA2"    "_task-"    "unspecified"

I changed "unspecified" to 'unspecified':

K>> BIDS_taskName = 'unspecified'; % default taskName
BIDS_fnameRoot = [BIDS_subjectName '_task-' BIDS_taskName];
K>> BIDS_fnameRoot

BIDS_fnameRoot =

    'sub-AA2_task-unspecified'

Also, I changed " " to ' ' everywhere, because strings don't work in Matlab prior to 2016.

2.

BIDS_sidecar_der.SoftwareFilters.Highpass.FilterType = 'highpass fir using pop_eegfiltnew()'; throws an error (Unable to perform assignment because dot indexing is not supported for variables of this type) because BIDS_sidecar_der.SoftwareFilters is char.

K>> BIDS_sidecar_der.SoftwareFilters = ’n/a’;
K>> class BIDS_sidecar_der.SoftwareFilters

ans =

    'char'

so I just changed 'n/a' to [] in both, BIDS_minimalSidecar.EEGReference and BIDS_minimalSidecar.SoftwareFilters

BIDS_minimalSidecar.EEGReference = []; % REQUIRED
BIDS_minimalSidecar.SoftwareFilters = []; % REQUIRED

What do you think about it? I am going to discuss the other limitations with my boss next week.

Best, Dawid

[dominikwelke]

hi @ksgfan ,

thanks for the feedback!

just fyi, you didnt need to merge (and thereby close) the PR to test the code.. i think it was not really ready, and keeping the development and debugging + related discussion in a seperate thread has some advantages :)

Also, I changed " " to ' ' everywhere, because strings don't work in Matlab prior to 2016

alright, good to know. i dont use matlab that much lately, and this whole char vs string thing was quite confusing to me :)

so I just changed 'n/a' to [] in both, BIDS_minimalSidecar.EEGReference and BIDS_minimalSidecar.SoftwareFilters

this is actually a problem we need to solve, because EEGReference and SoftwareFilters are required fields, and n/a is the entry to provide in BIDS jsons (as defined in the BIDS standards) this is related to the first limitation I listed above (json writer doesnt support backslash) and we need to find a solution here, because this behavior also destroys links and file locations (at least if generated on unix systems), or other entries in existing jsons that are copied over.

any idea? maybe including another function for writing json files might solve the problem (e.g. https://de.mathworks.com/matlabcentral/fileexchange/33381-jsonlab-a-toolbox-to-encode-decode-json-files - not tested though)

[ksgfan]

Hi Dominik

Actually BIDS_minimalSidecar.EEGReference = ['n/a']; is fine. SoftwareFilters will also be saved:

The function you recommended destroys links too. I will try to find a better solution.

Best, Dawid