Load Multiple Files

[kylemath]

Reference issue

Example: Fixes https://github.com/kylemath/mne-python/issues/4

What does this implement/fix?

Load multiple data files per person

Additional information

new dataset with more files

[kylemath]

@kuziekj
So I was just playing around with how to load in multi files

I added the new dataset link, and added a few more keys for the other 3 montages: py keys = ('mtg', 'elp', 'tol', 'a.001', 'b.001', 'c.001', 'd.001')

in this experiment, a and b are actually collected at the same time, with two different imagent machines. same with c and d, but different locations on the head.

so there are probably 16 sources and 16 detectors in a and c and 16 sources and 8 detectors in b and d

168 = 128 8 8 = 64 128+128+64+64 = 384

so the first simple thing I tried was changing the place where it looks for .001 to look for a.001 , then b.001, then c.001, by replacing the line,

 with open(files['b.001'],'r') as data:

It seems if you pick b or d, you get a plot of 769 locations (and it looks like a full head), instead of 385 something for a and c (and the plot looks like it has some holes)

I added the source and detector plotting code temporarily from the other branch as well:

---

So the .mtg and .elp files already have all the source and detectors, and they store their info in boxy_coords/all_coords and all_coords/all_labels

But then somehow you use the data file to tell how many sources and detectors you have in that file, but how do you know WHICH lines of the .mtg file correspond? Do the labels match? or do you use index?

I am a bit confused about this line:

            for i_coord in range(len(source_coords[0:total_chans])):

why only to total_chans, isn't the source_coords longer than that in the .mtg and .elp files

[kuziekj]

So for that line total_chans should correspond to the total number of channels in the boxy file you are loading. It assumes the file you are loading corresponds to the first n channels in the .mtg file. These lines find the source and detector coords by matching the labels in the .mtg file to the labels in the .elp file:

    ###get coordinates for sources###
    source_coords = []
    for i_chan in source_label:
        if i_chan in all_labels:
            chan_index = all_labels.index(i_chan)
            source_coords.append(all_coords[chan_index])

    ###get coordinates for detectors###
    detect_coords = []
    for i_chan in detect_label:
        if i_chan in all_labels:
            chan_index = all_labels.index(i_chan)
            detect_coords.append(all_coords[chan_index])

Right now, this is under the assumption that the order of channels in mtg are in the same order as the blocks/montages were run. So first n channels should be for block 1, the next n channels should be for block 2, etc.

Since we are now loading multiple files, we could just use source_coords and detector_coords and only grab the channels that correspond to the file we are loading.

I'll play around with those files you mentioned. If A/C has 16 sources and 16 detectors, and B/D has 16 sources and 8 detectors, shouldn't we have:

(A and C) 1616=256 (B and D) 168=128 (A, B, C, D) 256+128+256+128=768

If this is the case, when you plot A/C, are you getting a message saying it's plotting 768 fnirs channels/pairs when you run plot_alignment? If so, this is because the raw object contains data for ac, dc, and ph (which are all the same channels) so it's basically plotting each channel three times (256x3=768). If you want to separate them, you can try this:

import copy ac_data = copy.deepcopy(raw_intensity) ac_indices =[i_index for i_index,i_label in enumerate(raw_intensity.info['ch_names']) if 'AC' in i_label] ac_data.pick(ac_indices)

I'm also playing around with keys to load multiple files, and this seems to grab all the .001,.002, etc. files:

    files = dict()
    keys = ('mtg', 'elp', 'tol', '*.[000-999]*')
    for key in keys:
        if key == '*.[000-999]*':
            files[key] = [glob.glob('%s/*%s' % (fname, key))]
        else:
            files[key] = glob.glob('%s/*%s' % (fname, key))
        if len(files[key]) != 1:
            raise RuntimeError('Expect one %s file, got %d' %
                               (key, len(files[key]),))
        files[key] = files[key][0]

Still need to play around with it though.

[kuziekj]

So this might have to be a new branch as it crashed for me as well.

I think the issue is with how the emm0311 mtg/data files are organised.

The boxy files have 16 sources, and either 8 or 16 detectors, for each of the four montages. The mtg file says the montages have 92, 176, 83, or 176 channels (respectfully) but these number don't quite work with the number of sources/detectors indicated by the boxy files.

I assume this is because multiple optical systems were used/some channels are ignored for the emm recordings.

The code seems to run and plot fine when I use a different dataset (anc071). This dataset seems to be less complicated and organised in a way I'd expect (2 blocks, 2 montages, 8 detectors and 10 sources per file, mtg indicates the expected number of channels (80) for each montage).

So I think we should use this dataset (anc071) for further testing, for now, since I think it's closer to how our files will be organised once we start collecting data. Although perhaps I'm wrong, what do you think?

[kylemath]

Sounds good to me , as long as we have the full dataset with MRI

On Tue., May 12, 2020, 10:53 a.m. kuziekj, notifications@github.com wrote:

So this might have to be a new branch as it crashed for me as well.

I think the issue is with how the emm0311 mtg/data files are organised.

The boxy files have 16 sources, and either 8 or 16 detectors, for each of the four montages. The mtg file says the montages have 92, 176, 83, or 176 channels (respectfully) but these number don't quite work with the number of sources/detectors indicated by the boxy files.

I assume this is because multiple optical systems were used/some channels are ignored for the emm recordings.

The code seems to run and plot fine when I use a different dataset (anc071). This dataset seems to be less complicated and organised in a way I'd expect (2 blocks, 2 montages, 8 detectors and 10 sources per file, mtg indicates the expected number of channels (80) for each montage).

So I think we should use this dataset (anc071) for further testing, for now, since I think it's closer to how our files will be organised once we start collecting data. Although perhaps I'm wrong, what do you think?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/kylemath/mne-python/pull/5#issuecomment-627465619, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA36GFIP2GHNBHVPD5LMW7TRRF5HZANCNFSM4M3VQBRQ .

[kylemath]

@kuziekj great comments, thanks for playing, I replied to each

[kylemath]

ok this looks good to me and I think we are ready to merge into importBOXY, i'm gonna do it