Implementing calibration data to Deepmreye

[jonathantol]

Hello, first of all thank you for this library! I wanted to ask how would you recommend to implement the calibration data acquired through the Matlab code you published. Should I need to turn the edf Eyelink files to a specificformat (asc, csvor other)? I am also confused at which part of the main Python script (as shown on the exemplary data on the Jupyter Notebook) the calibration data is used?

[Remi-Gau]

I wanted to ask how would you recommend to implement the calibration data acquired through the Matlab code you published. Should I need to turn the edf Eyelink files to a specific format (asc, csv or other)?

There is a beginning of answer in this FAQ: https://deepmreye.slite.page/p/cHQpFuNTG/Input-and-preprocessing

I am also confused at which part of the main Python script (as shown on the exemplary data on the Jupyter Notebook) the calibration data is used?

If I am correct I think it is on this line in this section:

this_label = preprocess.load_label(gaze_data, label_type="calibration_run")

# Combine processed masks with labels
for participant in participants:

    if participant.startswith("s"):

        print("Running participant {}".format(participant))

        participant_folder = os.path.join(functional_data, participant)

        participant_data, participant_labels, participant_ids = [], [], []

        for run_idx, run in enumerate(os.listdir(participant_folder)):
            if not run.endswith(".p"):
                continue
            # Load mask and normalize it
            this_mask = os.path.join(participant_folder, run)
            this_mask = pickle.load(open(this_mask, "rb"))
            this_mask = preprocess.normalize_img(this_mask)

            # Load labels (in visual angle)
            this_label = preprocess.load_label(gaze_data, label_type="calibration_run")

            # omitted the rest for brevity

[CYHSM]

Yes, @Remi-Gau is correct. When you acquired your own calibration data you can replace or adjust the preprocess.load_label() function to return your label in the same format (num_samples, 2), so your preprocessing could look like this:

for participant in participants:        
        print("Running participant {}".format(participant))
        participant_folder = os.path.join(functional_data, participant)
        participant_data, participant_labels, participant_ids = [], [], []
        for run_idx, run in enumerate(os.listdir(participant_folder)):
            if not run.endswith(".p"):
                continue
            # Load mask and normalize it
            this_mask = os.path.join(participant_folder, run)
            this_mask = pickle.load(open(this_mask, "rb"))
            this_mask = preprocess.normalize_img(this_mask)

            # Load labels (in visual angle)
            this_label = get_labels_from_file(fn_participant_label)

def get_labels_from_file(file):
    # Load labels from EDF, CSV, etc...
    return np.array(labels)

For the exemplary dataset, we load the same data for every participant as they run through the same calibration script. If you have the same, you can just load it the same way as above (no need to load any data from EyeLink)

[jonathantol]

Thank you! Another question I had, if we use multi-echo scans how would you recommend us to implement the algorithm? Should we choose one echo or average it in some manner before the preprocessing and extracting the eye masks? In our lab we use a 7T scanner so we hope to update you on the results and how this library works for our protocols.

[Naubody]

I wanted to ask how would you recommend to implement the calibration data acquired through the Matlab code you published. Should I need to turn the edf Eyelink files to a specific format (asc, csv or other)?

@jonathantol, if you acquire camera-based eye tracking data, preprocess those e.g., by removing noise & outliers, detrending and slight smoothing of the time series. The exact steps are up to you but this worked for us. Then, split those cleaned data into the different functional volumes using the scanner trigger pulses stored in the log file, and then downsample to 10 horizontal and 10 vertical gaze position values per volume (10x2 values x n volumes). If you do not acquire camera-based eye tracking data, just use the fixation-cross positions in the log file and split those into the volumes.

I am also confused at which part of the main Python script (as shown on the exemplary data on the Jupyter Notebook) the calibration data is used?

I guess Markus and Remi have already answered this question? Let us know if anything is still unclear.

Another question I had, if we use multi-echo scans how would you recommend us to implement the algorithm? Should we choose one echo or average it in some manner before the preprocessing and extracting the eye masks?

You could always run it on the averaged data, which is simple and should work. However, with multi-echo, you might be able to get a better sub-TR temporal resolution, which might be fun to try. We are trying this right now too but do not have an answer yet. You could run the model on each individual echo separately, and then combine the decoded gaze labels afterwards. If you do this, and if you train your own model, I recommend training for each echo separately as the different echoes are expected to lead to different eyeball shapes. Let us know how this goes if you actually try - Extremely curious!

[Naubody]

Added the multi-echo question to the FAQ: https://deepmreye.slite.page/p/~e2FzIFY2/Data-acquisition

[Naubody]

@jonathantol Do you have any further questions or should we close this issue?

[jonathantol]

Hello again, We just now acquired some data at the fMRI, and while the algorithm seems to have successfully extracted the masks from the SE, we got some odd results for the ME, with the first echo working properly but the other 2 distorted. Do you think you know what could be the cause of this? Or does this mean that we should focus only on the 1st ME scan and ignore the other 2? I attached the example html reports generated by the algorithm.

Also, if our protocol mainly uses ME, would a model trained on SE calibration data be able to predict on the ME scans?

Thanks!!

[jonathantol]

I would also like to add another question: In the fixation calibration task, our data has 100 coordinates of fixation targets, each appearing for 1.5 seconds - meaning 150 seconds of calibration for this task (these are the default parameters set in your calibration code in MATLAB). Our TR is 2 seconds, which yields 75 volumes for the calibration. With these parameters, how are we meant to attach a label (xy coordinates) to a volume? This requires the 4th index of the mask shape (=n_volumes) to be equal to the first index of the label shape (=n_coordinates). In your exemplary data on the calibration dataset, it is 135 volumes and 27 labels - which are then multiplied by 5 (I suppose because each label appear for 4 seconds and your TR was 0.8 seconds, and 4/0.8=5) to make it equal to 135. For reference, I attached the specific lines of your code which does this process. We thought about somehow smoothing and interpolating the 100 coordinates to 75 coordinates, but unsure if this is valid. We can of course change the fixation time to match the TR and make both 2 seconds, but it could only help with future scans and not fix the problems we have with our current data.

From the main script: The load labels function:

Thanks again!