Check Functions for Figures 4, 5, 6

[KirstensGitHub]

Here, we are checking our sliding window plots. This involves checking the data to make sure the behavioral files have the correct information (Check 0, CHECK A), making sure our sliding windows average correctly over the data (Check B), and verifying that our timepoint-by-timepoint ttests correctly identify points when the timecourses are significantly different (CHECK C).

CHECKS:

• CHECK 0: behavioral data check (see issue #77)

• CHECK A: novel_label_check.pdf

• • A1: I show each run has an equal number of Novel Cued and Novel Uncued images.
• • A2: I show, in the sustained attention experiment, all Novel Cued images from a given run are from one category, whereas all Novel Uncued images from that run are from the other category.
• • A3: I randomly select data from two runs and do a spot check of Novel Image labeling by hand (exp1_16_0_novelTest_colored.xlsx, exp2_33_4_novelTest_colored.xlsx).
    
    
    

• CHECK B :

• • To generate our plots, we applied a sliding window over each run (for each attention level), averaged these values across runs within each subject, then plotted. In sliding_window_check_2.pdf, I first step through our sliding window code, piece by piece. I next step through the alternative method of averaging across runs first, THEN applying the sliding window for each subject. I plot the results of both approaches and show that they are very slightly different. To explore the cause of this difference, I use toy datasets. I show that a toy dataset/dataframe with numerical values in every cell will yield the same result from both methods, but a toy dataset with Nan values in some cells will yield a slightly different value from the two methods, due to Pandas automatic averaging over Nans.
• • Since our data contains nans, our choice of method makes a very slight difference. As such, we opt to change and take the mean across a subject's runs first, then apply the sliding window to the subject's data. We updated our sliding window function and verified that it yields the mean of the raw values within the window, for an individual subject (
    Sliding_Window_Check_1 (1)(1).pdf).
    
    

• CHECK C: timepoint_ttest_2-1.pdf

• • We also use a function that checks for significant difference between the mean familiarity ratings at each timepoint across the memory runs and draws lines to represent points where the values are meaningfully different. To review this function, I generated a timecourse plot with the corresponding significance lines, and also ran an individual ttest for the datasets at each time point, printing any trial numbers generating p values less than .05. I then matched the printed timepoints to the lines on the plot to show that each timepoint yielding a significant difference was represented in the plot.
    
    

• CHECK D: see comments

• • We also compared the timecourse plots with the violinplots. We did this visually and numerically. Visually, the timecourse plots appear to match the violin plots. Numerically, we find that the means are similar but not identical. We further explored with a toy dataset, and found that the mean from the sliding window averages over a dataset will not be identical to the mean of the raw data. Additionally, we show that if we use our code with a sliding window of size 1, we get identical means from the raw data and timeseries data, but as the window sizes get larger, the differences in means becomes greater (though it is always small, generally). This confirms that it is not our code's treatment of the data that causes the change, but that this is, rather, an effect of the sliding window.

FINDINGS / CHANGES:

• For sliding window plots, first take the mean for each position in the memory trial across all runs (by subject), then apply the sliding window, then plot (as opposed to first applying the window over each run, then averaging). Plots change very minimally.

• Optional update: change code so that parts of the plot at which only one timepoint has a significant difference show a dot instead of a very short (.02 width) line over the timepoint. ( @jeremymanning I find the former to be more visually consistent / easier to see, but perhaps a dot is more accurate..?)

[KirstensGitHub]

CHECK A1-A2: Novel Image Label, Code Check

novel_label_check.pdf

---

CHECK A3: Novel Image Label, Manual Check

exp1_16_0_novelTest_colored.xlsx exp2_33_4_novelTest_colored.xlsx

---

CHECK B Check our sliding window function, and check theoretical with toy dataset

sliding_window_check_2.pdf

---

CHECK C

Check significant difference plotting at each timepoint for sliding windows

timepoint_ttest_2-1.pdf

[KirstensGitHub]

previous sliding window function:

def apply_window(combo, window_length): ''' input: dataframe of behavioral data from an entire experiment output: dataframe of same shape where raw values have been replaced by rolling window mean '''

# select data from memory runs
data = combo[combo['Trial Type']=='Memory'][['Attention Level','Familiarity Rating','Trial','Subject','Run']]

# re-structure the data - each row is a trial, each column is an attn level
df = data.pivot_table(index=['Subject','Run','Trial'], columns='Attention Level', values='Familiarity Rating')

# apply rolling window, for each run in each subject
window_data = df.groupby(['Subject','Run']).apply(lambda x: x.rolling(window_length, min_periods=1, center=True).mean())

return(window_data)

updated window function:

`def apply_window(combo, window_length): ''' input: dataframe of behavioral data from an entire experiment output: dataframe of same shape where raw values have been replaced by rolling window mean '''

# select data from memory runs
data = combo[combo['Trial Type']=='Memory'][['Attention Level','Familiarity 
Rating','Trial','Subject','Run']]

# re-structure the data - each row is a trial, each column is an attn level
df = data.pivot_table(index=['Subject', 'Trial'], columns='Attention Level', values='Familiarity Rating')

# apply rolling window, for each run in each subject
window_data = df.groupby(['Subject']).apply(lambda x: x.rolling(window_length, min_periods=1, center=True).mean())

return(window_data)`

[KirstensGitHub]

CHECK D

plots showing timecourses and violins together, for visual comparison:

[KirstensGitHub]

CHECK D Numerical comparisons of means from sliding windows and raw data

---

Here I calculate the mean familiarity ratings from the raw data, as well as the mean familiarity from the timeseries data with sliding windows of sizes 1, 5, and 20.

We see that, using our code to apply a sliding window of size 1, then averaging, yields the same mean as the mean from the raw data, but larger window sizes yield means that are slightly, increasingly different. This shows that the code implementing the window isn't inherently changing the data in an undesired way (given that window size of 1 yields the same mean values), but that the actual act of averaging over the sliding window yields a mean that is not identical.

[KirstensGitHub]

CHECK D

toy data showing that average of sliding windows and average of raw data are not identical (start section 5: Show that the mean..."): Sliding_Window_Check_1 (4).pdf