For Mon. February 20th

[rmarren1]

Monday Commitments

Status	Task	Deliverable
:panda_face:	Expanded Interpolation benchmarking, reporting at what number of channels bad interpolation actually helps metrics.md	.ipybn @rmarren1
🐼	Run auto-benchmarking algorithm designed by Ryan on RPCA.	.ipynb @tsunwong123
🐼	Use event triggers to further format Nicolas's data into trials and align patient data	.ipynb @tsunwong123

Thursday Checkpoint

Status	Task	Deliverable
:panda_face:	Abstracted and Automated benchmarking utility, can be used on cortex. (pip installable, ask Greg)	.ipybn demo detailed enough for teammates to use, link to github @rmarren1
:x:	Some sort of algorithm to align two subjects on the same task	@tsunwong123
:x:	Align format of Kara's data with Nicholas's data	@nkumarcc

[rmarren1]

@gkiar @nkumarcc @tsunwong123 @jovo

[rmarren1]

Week Summary:

• General: Created auto_bench which is a library which provides solutions to the following problems we faced while benckmarking:

  • Problem: Run times are too long.
  • Current Solution: Use the pathos parallel computing library to speed up the bulk processing.
  • Problem: RAM usage is too high.
  • Current Solution: Instead of loading many subjects into a list and then making computations on that list, define a generator which will load each subject from a .pkl file only when it is needed. Also, remove all intermediate variables when possible, so that it is never the case that the same subject exists twice in memory unless absolutely necessary.
  • Problem: Loading files takes too long. E.g., when using the generator definition above, a trial will be read from memory each time it is needed. Loading in from the .csv files given in the original data takes several seconds per subject, which adds up when you must load each subject several times.
  • Current Solution: Created a script which will convert the .csv files to binary .pkl files, which can be read from hard disk into RAM in 1/1000 of the time of a .csv.
  • Problem: Programs which run for a while get killed accidentally, wasting time.
  • Current Solution: Create a wrapper for the benchmarking task which will save progress after each bulk calculation, and will look for cached results before re-calculating (e.g. if a program is killed halfway through, if you run the same command again it will start halfway through and keep going.)

• Denoising Benchmarking

  • Ran the above auto_bench utilities to determine whether PCA or RPCA based denoising was better (projecting data onto top k principal directions). RPCA is calculated by using the Principal Component Pursuit algorithm.
  • Conclusions: RPCA and PCA both max out with 1 discriminibility in the data setting used (4 subjects, 11 trials from the first 3, 9 trials from the last, each trial is the full brain scan of around 150,000 time steps). The function of the discriminibility as k increases was different for RPCA and PCA, with RPCA decreasing faster initially but having larger discriminibility when k is large. The default discriminibility was around .64, so this is a large increase. We believe that these results should be corroborated against other metrics to make sure this is not just a weird property of the metric we used (frob. norm of the correlation matrix).

• Interpolation Benchmarking.

  • More thorough benchmarking was performed on 4 subjects with 11, 11, 11, and 9 full (~150k time steps) each.
  • The conjecture of some bifurcation or change around 30 bad channels was explored more. After around 25 bad channels, the interpolation will actually improve the discriminibility of the data. Before that, it makes discriminibility worse. We would like to look into this further when our computational methods are more honed, and we have everything set up on cortex.

[gkiar]

• @rmarren1:
  • for benchmarking you should use much smaller data (i.e. downsampled) so that the ram usage and compute time and all that are less. Or at least have a flag for "small" or something that does that. It'll make your life easier for small changes and testing if you have bugs or things.
  • notebook for interpolation looks cool, but I think I understand conflicting results? Prior weeks you showed that interp makes disc. worse, and here you show it makes it better? I think it's just that over some range it's bad and another range it's good, but can you make that more clear for me to understand?
• @tsunwong123:
  • you linked to a notebook in your first deliverable that ryan is the only contributor to? What did you do?
  • I don't see proof that you did the alignment? Can you provide some?
• @nkumarcc:
  • I forgot that we agreed you wouldn't have a big deliverable this week since you were busy, so let's in future write that down and why. That way you always have a thing and any on-looker can understand.