grand challenge: streaming brain decoding

[jeremymanning]

Achieving this grand challenge requires:

1. Support for streaming data
2. Interactive feature/event labels
3. On-the-fly decoding
4. Reading in brain data on the fly, e.g. from an OpenBCI device (e.g. see this project)

Here's the vision: The user wears their brain recording device, streaming data into hypertools. Periodically, they focus hard on imagining a word (e.g. picture an apple as intensely as possible for a few seconds). As this happens, they press spacebar and tag that brain pattern/event with the label "apple". This repeats for dozens (hundreds?) of words, and several presentations of each word. The decoding model (labeled brain patterns) is saved out to disk.

Now we switch to "decode" mode. Load in the decoding model and start streaming data from the headset again. Now the user picks a word from the labeled set and focuses hard on bringing that word to mind. Imagine their shock and delight when their brain trajectory moves to the appropriate labeled point, and the word they were thinking of is highlighted on the display!

Here's another (related) vision: This tool could be used as a benchmark for brain decoding challenges. For example, suppose someone writes a feature_extractor function for translating raw brain data into arbitrary features (power spectra, some sort of deep neural network's re-representation of the data, classifier outputs, etc.). In other words, the feature_extractor allows us to focus in on the features/components of brain activity we think are important.

We also need a decoding function (decoder). This could be based on the Euclidean distance between the current brain patterns and labeled patterns, correlation between patterns, etc. The decoder tells us how to map between extracted features and labeled points, ideally in a robust way.

Now brain decoding is a matter of finding the right feature_extractor function and decoder function.

[alokkumary2j]

Interesting thought, looks very exciting! @jeremymanning - Was there any work done on this Issue or any plans of revisiting it?

[jeremymanning]

Hi @alokkumary2j we haven't made any real progress on this, but I'm very interested in revisiting it! One idea we were throwing around was to support streaming data through Streaming Dataframes.

If you'd like to do some work on this, I'd be happy to coordinate about where it might make sense to get started, and/or help think through some of the implementation details.