Brainharmonic - Generating music from brain signal data using deep learning
Leaders
Mariano Cabezas
Collaborators
Aria Nguyen
Brendan Harris
Brainhack Global 2022 Event
Brainhack Australasia
Project Description
This project aims to develop a tool to generate music from brain signals using deep learning models. There has been a lot of work on generating new music from a large collection of music using deep AI models, as well as some work on generating music from brain EEG/fMRI signal through algorithmic or rule based approaches. However there has not been work done on using deep learning models to generate music from EEG/fMRI signal.
In this project, we will use deep generative models to allow EEG/fMRI data to be used as an input to generate music. More details of the design can be found on our Github page.
Develop a tool to generate songs from EGG/fMRI brain signals using deep learning
Good first issues
Currently we have a pipeline to generate musical motif from a time series and a deep learning model to output a song from the generated motif. This was tested with a list of EEG signals from multiple subjects. We can build upon the existing work in three aspects
Issue one:
Develop a pipeline to process time series signals (fMRI, EEG, etc) and feed the output to the music generation part
Issue two
Refine current DL to resolve some existing issues and produce better music
Issue three
Process signals from different brain regions as different instruments to create multitrack songs.
Title
Brainharmonic - Generating music from brain signal data using deep learning
Leaders
Mariano Cabezas
Collaborators
Aria Nguyen Brendan Harris
Brainhack Global 2022 Event
Brainhack Australasia
Project Description
This project aims to develop a tool to generate music from brain signals using deep learning models. There has been a lot of work on generating new music from a large collection of music using deep AI models, as well as some work on generating music from brain EEG/fMRI signal through algorithmic or rule based approaches. However there has not been work done on using deep learning models to generate music from EEG/fMRI signal.
In this project, we will use deep generative models to allow EEG/fMRI data to be used as an input to generate music. More details of the design can be found on our Github page.
Link to project repository/sources
https://github.com/marianocabezas/brainharmonic
Goals for Brainhack Global
Develop a tool to generate songs from EGG/fMRI brain signals using deep learning
Good first issues
Currently we have a pipeline to generate musical motif from a time series and a deep learning model to output a song from the generated motif. This was tested with a list of EEG signals from multiple subjects. We can build upon the existing work in three aspects
Communication channels
https://mattermost.brainhack.org/brainhack/channels/brainharmonic
Skills
Python: intermediate Pytorch: beginner EEG/fMRI data processing: beginner
Onboarding documentation
https://github.com/marianocabezas/brainharmonic
What will participants learn?
EEG/fRMI data preprocessing Deep learning generative models Existing music-to-music deep AI frameworks
Data to use
No response
Number of collaborators
3
Credit to collaborators
Project contributors are credited on Readme file on project Github page
Image
Leave this text if you don't have an image yet.
Type
coding_methods, method_development, pipeline_development
Development status
1_basic structure
Topic
deep_learning
Tools
fMRIPrep, Freesurfer, Jupyter
Programming language
Julia, Python
Modalities
EEG, fMRI
Git skills
1_commit_push
Anything else?
No response
Things to do after the project is submitted and ready to review.
Hi @brainhackorg/project-monitors my project is ready!