Controllable deep melody generation via hierarchical music structure representation

[tatsuropfgt]

Controllable deep melody generation via hierarchical music structure representation [ISMIR21]

Abstract

• music framework generates rhythm and basic melody using two separate transformer-based networks
• then, generate the melody conditioned on the basic melody, rhythm, and chords

Background

Deep learning for music generation has three problems

1. modeling large music structures and multiple levels of repetition
2. controllability to match music to condition
3. scarcity of training data

Method

• hierarchical structure in the music framework

• Basic melody, the most common pitch in each 2-beat, segment is generated by the original melody

• in phrase-level music, they first generate a basic melody and basic rhythm form
• then, generate the rhythm using the basic rhythm form
• finally, generate a new melody given the basic melody, rhythm, and chords copied from the input song

• melody. basic melody and rhythmic pattern are generated by the Transformer-LSTM network

[tatsuropfgt]

demo