Generate deep visualizations from EEG features

[benyaminmk]

Deep visualizations

Deep visualizations is a technique to use neural networks as generative models for pictures/video. It would be cool to generate audio from a GAN but many tools already exist to generate visuals so that is a better first step and would aid in learning about GANs

Goal

• [ ] Use BIGGAN or some other pretrained neural network.
• [ ] Feed a spectral EEG feature to control how the GAN moves through the latent space.
• [ ] Extra Credit: get it all running in real time.

Reading

• workshop by Jason Yosinski with companion paper and code
• Repo for using GANs for deep visualization from music

[benyaminmk]

^ lol the extra credit is such a reach!

[benyaminmk]

First take: use deep music visualizer

TODO

• [ ] manually select natural scenes from imagenet classes
• [ ] look at input vectors during cool patterned portions of deep music viz for radioactive to see how to potentially replicate it
• [ ] create input vector from local EEG data recorded by muse and manually selected imagenet classes.
• [ ] render video on GPU

[benyaminmk]

EEG band power for BigGAN class weight and noise vector generation

The main cool part of the the deep music visualization is that they have a smart way of decomposing an mp3 into class and noise vectors that for BigGAN inputs. I need to convert that logic to EEG.

First pass

as a first pass I'll use slow, theta, alpha, and beta for classes and gamma for noise. Noise vectors are 128dim and class vectors are 1000 dim. Class vectors can be sparse and values must be between 0 and 1.