search

magenta / magenta-js

Magenta.js: Music and Art Generation with Machine Learning in the browser
https://magenta.tensorflow.org
Apache License 2.0
1.96k stars 311 forks source link

sequenceProtoToMidi used on a quantizedSequence generates a MIDI file without song (perhaps: unquantizeSequence is bugging) #462

Closed Jars-of-jam-Scheduler closed 4 years ago

Jars-of-jam-Scheduler commented 4 years ago

Hi everyone,

Summary

  1. The API I speak about

  2. The context

  3. The problem: Actual result + Expected result

  4. What I already have tried

  5. Clues to help you to help me

  6. Minimal and Testable Example: Prerequisites and Steps to follow + Sources

The API I speak about

https://github.com/magenta/magenta-js , https://magenta.github.io/magenta-js/music/

The context

I try to generate a MIDI file that I can play with VLC on Ubuntu 18.04.4. Here are the steps of my instructions flow:

  1. I load a notes sequence array with arbitrary notes data
  2. I quantize it
  3. I get the bytes array of the MIDI format of these quantized notes
  4. I save it into a MIDI file that I download

The problem

Actual result

Step 4: the output MIDI file seems to contain the notes. However, when I play it, I hear nothing.

Expected result

Step 4: the output MIDI file contains the notes. When I play it, I hear the notes.

What I already have tried

Step 2: I don't quantize it Step 3: I get the bytes array of the MIDI format of these not-quantized notes Step 4: I save it into a MIDI file that I download

This way, the output MIDI files contains the notes. When I play it, I hear the notes.

Clues to help you to help me

Calling mm.sequenceProtoToMidi( mm.sequences.unquantizeSequence(quantized_sequence) ) triggers the same problem (in other words: calling mm.sequences.unquantizeSequence on the quantized sequence and then calling mm.sequenceProtoToMidi on it) .

However as I explained before: if I use a directly not-quantized sequence, it works.

Add to this previous fact this new one: your function sequenceProtoToMidi uses unquantizeSequence when it's necessary (i.e.: when isUnquantizedSequence returns true).

So I wonder if your function unquantizeSequence() works correctly or if it's bugging.

Minimal and Testable Example

Prerequisites and Steps to follow

Open the following file in your browser. It will automatically download the MIDI file that does not work as expected.

To reproduce what I've tried, follow these steps:

  1. Comment the line const quantizedSequence = mm.sequences.quantizeNoteSequence(sequence, 1)
  2. Replace the line var midi_bytes_array = mm.sequenceProtoToMidi(quantizedSequence) with var midi_bytes_array = mm.sequenceProtoToMidi(sequence)
  3. [OPTIONAL] Replace the line saveByteArray("test_with_quantized_sequence.midi", midi_bytes_array); with saveByteArray("test_with_not_quantized_sequence.midi", midi_bytes_array);

Sources

File: index.HTML

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Music</title>
    <script src="https://cdn.jsdelivr.net/npm/@magenta/music"></script>
</head>
<body>

<script>
        const sequence = {
          totalTime: 8,
          notes: [
            {pitch: 60, startTime: 0.0, endTime: 0.5},
            {pitch: 60, startTime: 0.5, endTime: 1.0},
            {pitch: 67, startTime: 1.0, endTime: 1.5},
            {pitch: 67, startTime: 1.5, endTime: 2.0},
            {pitch: 69, startTime: 2.0, endTime: 2.5},
            {pitch: 69, startTime: 2.5, endTime: 3.0},
            {pitch: 67, startTime: 3.0, endTime: 4.0},
            {pitch: 65, startTime: 4.0, endTime: 4.5},
            {pitch: 65, startTime: 4.5, endTime: 5.0},
            {pitch: 64, startTime: 5.0, endTime: 5.5},
            {pitch: 64, startTime: 5.5, endTime: 6.0},
            {pitch: 62, startTime: 6.0, endTime: 6.5},
            {pitch: 62, startTime: 6.5, endTime: 7.0},
            {pitch: 60, startTime: 7.0, endTime: 8.0},
          ]
        }

        try {          
            const quantizedSequence = mm.sequences.quantizeNoteSequence(sequence, 1)
            var midi_bytes_array = mm.sequenceProtoToMidi(quantizedSequence)
            saveByteArray("test_with_quantized_sequence.midi", midi_bytes_array);

          } catch (error) {
            console.error(error)
        }

         function saveByteArray(reportName, byte) {
            var blob = new Blob([byte], {type: "audio/midi"});
            var link = document.createElement('a');
            link.href = window.URL.createObjectURL(blob);
            var fileName = reportName;
            link.download = fileName;
            link.click();
        };
    </script>

</body>
</html>

Thank you very much for your help. Kind regards,

notwaldorf commented 4 years ago

I think this might be happening because Your notes don’t have any velocities. What happens if you add “velocity: 100” to every note?

On Sun, Jun 7, 2020 at 1:15 PM Jars-of-jam-Scheduler < notifications@github.com> wrote:

Hi everyone, Summary

1.

The API I speak about 2.

The context 3.

The problem: Actual result + Expected result 4.

What I already have tried 5.

Clues to help you to help me 6.

Minimal and Testable Example: Prerequisites and Steps to follow + Sources

The API I speak about

https://github.com/magenta/magenta-js , https://magenta.github.io/magenta-js/music/ The context

I try to generate a MIDI file that I can play with VLC on Ubuntu 18.04.4. Here are the steps of my instructions flow:

  1. I load a notes sequence array with arbitrary notes data
  2. I quantize it
  3. I get the bytes array of the MIDI format of these quantized notes
  4. I save it into a MIDI file that I download

The problem Actual result

Step 4: the output MIDI file seems to contain the notes. However, when I play it, I hear nothing. Expected result

Step 4: the output MIDI file contains the notes. When I play it, I hear the notes. What I already have tried

Step 2: I don't quantize it Step 3: I get the bytes array of the MIDI format of these not-quantized notes Step 4: I save it into a MIDI file that I download

This way, the output MIDI files contains the notes. When I play it, I hear the notes. Clues to help you to help me

I don't have clues for you, sorry! Minimal and Testable Example Prerequisites and Steps to follow

Open the following file in your browser. It will automatically download the MIDI file that does not work as expected.

To reproduce what I've tried, follow these steps:

  1. Comment the line const quantizedSequence = mm.sequences.quantizeNoteSequence(sequence, 1)
  2. Replace the line var midi_bytes_array = mm.sequenceProtoToMidi(quantizedSequence) with var midi_bytes_array = mm.sequenceProtoToMidi(sequence)
  3. [OPTIONAL] Replace the line saveByteArray("test_with_quantized_sequence.midi", midi_bytes_array); with saveByteArray("test_with_not_quantized_sequence.midi", midi_bytes_array);

Sources File: index.HTML

<!DOCTYPE html>

Music

Thank you very much for your help. Kind regards,

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/magenta/magenta-js/issues/462, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAKOIUURXSMQAOFKCGPIFFDRVPYN5ANCNFSM4NXTKL3Q .

Jars-of-jam-Scheduler commented 4 years ago

You are right. I solved this issue by adding "velocity: 100". However, I have the same problem with the following code (I try to output in a MIDI file a music generated by using MusicRNN):

const quantizedSequence = mm.sequences.quantizeNoteSequence(sequence, 1)
const improvisedMelody = await music_rnn.continueSequence(quantizedSequence, 20, 0.1, ['Bm'])
const midi_bytes_array = mm.sequenceProtoToMidi(improvisedMelody)

So, in this case, how could I specify a velocity for the notes of the improvisedMelody constant? Indeed, your function continueSequence doesn't seem to specify a velocity.

notwaldorf commented 4 years ago

Right, I think MusicRNN also outputs this incorrectly with no velocities (see https://github.com/magenta/magenta-js/issues/448), so you'd have to do it after the fact. Something like:

improvisedMelody.notes.forEach(n => n.velocity = 100)
Jars-of-jam-Scheduler commented 4 years ago

@notwaldorf Thank you!