codecov-io
commented
6 years ago Codecov Report
Merging #2 into master will increase coverage by
0.59%. The diff coverage isn/a.
@@ Coverage Diff @@
## master #2 +/- ##
==========================================
+ Coverage 88.92% 89.51% +0.59%
==========================================
Files 8 8
Lines 677 677
==========================================
+ Hits 602 606 +4
+ Misses 75 71 -4| Impacted Files | Coverage Δ | |
|---|---|---|
| remodnav/tests/test_detect.py | 98.57% <0%> (+5.71%) |
:arrow_up: |
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact),ø = not affected,? = missing dataPowered by Codecov. Last update 64e2534...944bfb1. Read the comment docs.
adswa
mih
ATM we cannot reproduce (reason is subject to research). Here is a summary of the differences: duration stats for fixations, saccades, PSOs and pursuits. The first value is our's, the one in parenthesis is the one reported in the paper. Note, our values are not own detection results, but stats computed from their released data. Subjectively substantial deviations are in BOLD, although there really should be no deviations at all, and the code to compute all stats is the same for all events (see PR).
Conclusion: We get what is in the paper for saccades and PSOs, but something substantially different for number of fixations.
Fixation durations
Saccade durations
PSO durations
Pursuit durations
Confusions
Assuming we make no mistakes extracting the Anderson labels, here is how our algorithm performs re confusions.
MN vs. RA
This gives us the baseline
algorithm vs. coder MN
algorithm vs. coder RA
Mis-classification summary stats
For all pairwise comparisons, this shows the overall misclassification rate (using timepoints as unit of measure, and limited to timepoints that have been labeled with FIX, SAC, PSO, or PUR by any method, hence ignoring NaN/blinks and undefined (which is rarely used)), same misclassification rate as before, but ignoring PUR events too. The remaining numbers are percentages of labels used in die misclassified samples. In contrast to the paper the method label that is misclassifying is given (not "over" and "under", as I found this confusing).
images
Analog to table 8 in the paper
dots
Analog to table 9 in the paper
videos
Analog to table 10 in the paper
Interim conclusion
Performance looks good. Without pursuit this looks better than the stats in the paper (although I am not 100% confident that we compute things the exact same way). Confusion patterns with and without pursuit look sensible.
Critical feedback appreciated!