Final report draft - Feedback

[joelostblom]

@MattTPin @sbabicki @mgaroub @YikiSu

The repo and final data product is looking to be in great shape! The report is good overall, but I think there is room to flesh out the motivation and pros & cons more on several places in the report. You have done this nicely when explaining your choice of streamlit as the dashboarding library but it is missing elsewhere e.g. for each of the individual visualizations (what value to each of these bring to the researcher? why do we need a brain map and a 3D skull map, why is 2D important, why are there 2 connectivity circles (talk about the new viz you created that is not available anywhere else)).  This shouldn't be an issue length-wise as you are currently around 1300/2000 words. Below are specific comments I made on different section of the report (not as nicely formatted as last time in the interest of time....):

“Executive Summary” (“Final_Report.pdf”, p. 1) Make this a summary of the entire report (including your results), rather than just your plan (it's a typo that is say "proposal" here in the instructions). Move the enumeration of the goals to the end of the introduction and elaborate on the importance/value of each one in 1-2 sentences (some of this is already in the end of the intro so it will tie in nicely).

“3.2 Introduction” (“Final_Report.pdf”, p. 1) Clear, well-written intro

“Figure 1: Examp” (“Final_Report.pdf”, p. 1) Too small, I can't read any labels in the figure.

“Data Science Methods” (“Final_Report.pdf”, p. 2) Also include brief description of the data and any preprocessing you did.

“Clear documentation” (“Final_Report.pdf”, p. 2) GREAT documentation! Thorough guide and great idea with the demo example animations for the user interface (if it is an easy fix I would make them a bit bigger, it is hard to read the numbers and see what you are clicking). It's also nice that you show how they can use any MNE functions with the epochs in case they need to extend it in the future. Also great that you have gif and animated versions for the functions. A few minor comments:

• load_ext and autoreload are ipython magics and will only work in IDEs running ipython.
• The docstrings don't show up under Creating EEG objects -> Module overview.
• Raw Voltage Visualizations -> Create the raw voltage plots: Why is single epoch two lines and multi epoch one line? And under "Saving the plot" do you need the format= when you have have the file extension.
• Do you need the suppression of the filter warnings and the magics on every page? Or should this note be in just one place?
• In the  places where long output is generated, e.g. 3D brain viz "generate forward", do you know why there is one code box per line instead of a single big one? It makes it a bit harder to read, but not a big deal. I also think the main readme gives a really nice overview and the helpful links section is great! Maybe include the link of the documentation again under "Package Usage" so that it is clear that detailed info is available.

“the code is tested” (“Final_Report.pdf”, p. 2) Comprehensive tests, well done!

“Figure 2: Visualization comparison” (“Final_Report.pdf”, p. 2) This figure is great and I think part of the motivation I am asking for can be addressed by writing out some of the key pros & cons indicated in this figure in the appropriate places of the report.

“In this context, “brain states” refers to a distinct temporal pattern of electrical activity within the 19 channel EEG data.” (“Final_Report.pdf”, p. 2) Start with this general description of the problem as well as a high level indication of what the data looks like that you want to cluster.

“3.4.1 Python Package” (“Final_Report.pdf”, p. 2) Include motivation/justification to why you have created these plots. What purpose/value to they add to the partner? The same way you added this to your slides after we discussed it.

“3.4.2 User Interface” (“Final_Report.pdf”, p. 5) And you can download figures, that's pretty important for them. Also describe why anyone would even use the package instead of the UI: e.g. more control/functionality working in notebooks, etc (maybe goes best as an intro to the pkg section).

“2 Stretch goal” (“Final_Report.pdf”, p. 5) This is currently not documented or delivered in a way that makes it easy for the partner to understand and extend your work.

• Add a summary here in the report where you describe the problem to show that you have understood it and motivate your approach(es) to solving it. Also include results on a high level potentially including a key figure, but leave the details for your appendix or linked notebooks.
• When the partner clicks the notebook link to read about the clustering details, they end up in a folder on GtiHub without any readme describing its content. Likely they will try clicking the first notebook, which only contains a single cell and a warning message. Since they are not familiar with notebooks or Python, they will likely think something failed here and might not even go further.
• If they click the other files, they will read the same Introduction and Objectives sections in three different clustering notebooks. These section are good, I would flesh them out a little and move them to a common landing page, either a README in that folder or preferably a page in your Jupyter Book documentation. I think the latter is the best option as it collects the information in fewer places, which is easier to the partner. You can make it a separate chapter and write a disclaimer on top that the functionality is not yet available in the package it self. Then the notebooks can be subsections of that chapter.
• The motivation for why you chose hidden markov model is good and you should include the corresponding sections for the other clustering approaches you took also.
• It also seems like most of the notebook content can be cut down a lot, the Resampler notebooks seems like it can be deleted altogether and the SOM, range, & single notebooks all seem to contain the same code for over half the notebook. Please go through and make sure you are not redundantly repeating yourself in the different notebooks.
• In addition to the introductory sections and the suitable explanations of the different methods throughout the notebooks, you should write sections interpreting the results you are getting from the clustering. The partner will not be able to follow what the results mean unless you write down your interpretations.
• Currently there are error messages and notes lie "DO NOT RUN" in these notebooks, which is not something you can deliver to the partner. I assume you are already planning to remove these, and when you do that, also try to remove any redundant content and consolidate into into fewer notebooks if possible.
• Make sure the notebooks are run through in order and that all the plots show up. Using Jupyter book will help this (and probably also the subset math font not showing up correctly for the HMM intro).

“particularly with the SOM method which we deem to be the most promising” (“Final_Report.pdf”, p. 6) Why?

[sbabicki]

Thanks Joel! We have addressed almost all of the feedback in the latest version, with the exception of:

• Raw Voltage Visualizations -> Create the raw voltage plots: Why is single epoch two lines and multi epoch one line? And under "Saving the plot" do you need the format= when you have have the file extension - I think technically you don't need it and we would be willing to remove it in the future, but we did not remove it before the deadline.
• In the places where long output is generated, e.g. 3D brain viz "generate forward", do you know why there is one code box per line instead of a single big one? - this seems to be something to do with jupyterbook, because it was minor we did not find a way to update it before the deadline.
• ...demo example animations for the user interface (if it is an easy fix I would make them a bit bigger, it is hard to read the numbers and see what you are clicking - we tried to make them bigger, but were not able to find a way before the deadline.

A checklist of tasks that were addressed based on your feedback can be found in the following issues: https://github.com/UBC-MDS/simpl_eeg_capstone/issues/207 https://github.com/UBC-MDS/simpl_eeg_capstone/issues/113