Agenda 11/5/2021

[F-said]

Updates

• Merged in split-half #365
• Merged in ERP #367
• 90 CMI BIDS data generated on HPC
• Scaled back split-half & ERP metrics to one subject after introduction of bugs in preprocess

Blocks/Challenges

• After careful review of preprocessed data on BIDS, there seems to be two bugs that either cause the pipeline to fail or prevent final preprocessed data from being written

  • 368 set_ref

  • 370 ica_raw

• Feat-io bug #371

Next Actions

[yanbin-niu]

maybe briefly discuss the file positions / names of reliability.py,postprocessing.pyand run_metrics.py

[georgebuzzell]

discuss idea of upcoming Pepper retreat/hackathon to get us to pre-release. And, a second one to get us to v.01 release. George is paying for pizza and Mountain Dew!

Review/discuss the updated scope of PEPPER: Not a "pipeline" but a "framework for community-driven, reproducible EEG analysis across age/context/system" -modular, reproducible, "living pipeline" at center -EEG4p to facilitate rapid benchmarking and identification of parameter sets for different ages/contexts/systems -Intentionally designed infrastructure for collaborative development and use. To include standards for how to ensure reproducibility (container-based tests), and checks on code development (required prs, automatic tests run at commit), as well as consideration given to governance, credit/authorship, licensing.

Discussion of timeline for pre/full release(s) and what to include.

1. We NEED to get a prerelease out SOON (November), and the minimal requirements are: -fully tested docker, singularity -multiple runs of data, with sensible results that at least look "OK" -proofread of existing documentation -white paper that describes the project. validation results are not needed for the white paper, and EEG4p does not need to be fully implemented/tested and/or part of the pre-release, but the design needs to be fully fleshed out. *Unless we include an alpha version of EEG4p and some decent initial validation results, I would suggest the pre-release is not advertised. It can be sent to a few collaborators, but ideally, it would be kept mainly underground for now.

-Timeline for release .01 and minimal requirements: -all of pre-release, but also need to demonstrate that validation metrics produce at least comparable results to Matlab-MADE. -ideally, an alpha version of EEG4p is also included, and this should be a minimal additional requirement, given that a primary requirement of the .01 release is demonstrating validation results via metrics. I.e., since we have to calculate these, might as well put in a slight bit more work to set up in a sustainable/scalable fashion. Focus should be on implementing structure well for a limited set of metrics, as opposed to quick and dirty computation of many metrics -White paper must be revised/updated, and crucially, must now include results of validation metrics and description of eeg4p as well -revise/update documentation on github, and, initial standalone website for EEG4p that is linked to from NDCLab wiki (BUT, pepper is its own thing and independent from the NDCLab).

Organization for bEEG4p -file positions/names of reliability.py, postprocessing.py and run_metrics.py -discuss whether it is viable to have standard input/output for "measures" and "metrics/stats" -where to place benchmarking datasets

Sonya comments on what is/is not intuitive about use of github, etc.

Discussion of results of 100-sub run of pipeline (errors, compute time, etc) Discussion of ERPs computed on 100-sub run Discussion of reliability metrics computed on 100-sub run; SME too??

Next steps and reliable timeline.

Recruitment needs: -coding -eeg knowledge -documentation overhaul -usability/design -writing/marketing -website design