search

lathoub / Arduino-AppleMIDI-Library

Send and receive MIDI messages over Ethernet (rtpMIDI or AppleMIDI)
Other
306 stars 66 forks source link

MIDI Note Off messages dropped #163

Closed serifpersia closed 10 months ago

serifpersia commented 10 months ago

Describe your project and what you expect to happen:

My ESP32 S2 Project expects MIDI Note On and Note Off to control WS2812B leds. Each LED will be toggled on and off depending on the MIDI message.

Describe your problem (what does not work):

Note Off notes are being dropped off so my leds don't receive signal to turn off so they hang...

Steps to reproduce

Spam more than 4 note midi messages at the same time very fast(humanly possible ofc)

Latency reported on rtpMIDI usually sits at 9ms Attached image showing not all midi note off messages got to ESP32S2 Attached image of rtpMIDI setup

AppleMIDI handle funcs in setup or loop doesn't matter still having the dropping note off messages issue. Verified with direct MIDI USB device connected to ESP32S2 USB Host, no issues with however many notes and frequency(slows down if CPU can't handle that many notes at that speed(test of 10 note chords spamming each 50ms, midi standard supports 32nd notes at 200bpm thatch avg of 93ms) My network latency should be able to handle this amount of data without dropping, the only reason why its dropping could be confusing note on with note offs or something else.

image image

 #include <WiFi.h>
 #include <WiFiClient.h>
 #include <WiFiUdp.h>
 #include <AppleMIDI.h>

 char ssid[] = "redacted"; //  your network SSID (name)
 char pass[] = "redacted";    // your network password (use for WPA, or use as key for WEP)

 int8_t isConnected = 0;

 APPLEMIDI_CREATE_DEFAULTSESSION_INSTANCE();

 void setup() {
   Serial.begin(115200); // Initialize the serial monitor

   Serial.println("Booting");

   WiFi.begin(ssid, pass);

   while (WiFi.status() != WL_CONNECTED) {
     delay(500);
     Serial.println("Establishing connection to WiFi...");
   }

   Serial.println("Connected to network");

   Serial.print(WiFi.localIP());
   Serial.print(" Port ");
   Serial.print(AppleMIDI.getPort());
   Serial.print(" (Name ");
   Serial.print(AppleMIDI.getName());
   Serial.println(")");

   MIDI.begin();

   AppleMIDI.setHandleConnected([](const APPLEMIDI_NAMESPACE::ssrc_t & ssrc, const char* name) {
     isConnected++;
     Serial.print("Connected to session ");
     Serial.print(ssrc);
     Serial.print(" Name ");
     Serial.println(name);
   });
   AppleMIDI.setHandleDisconnected([](const APPLEMIDI_NAMESPACE::ssrc_t & ssrc) {
     isConnected--;
     Serial.print("Disconnected ");
     Serial.println(ssrc);
   });

   MIDI.setHandleNoteOn([](byte channel, byte note, byte velocity) {
   noteOn(note,velocity);
   });
   MIDI.setHandleNoteOff([](byte channel, byte note, byte velocity) {
   noteOff(note,velocity);
   });
 }

 void loop() {
   // Listen to incoming notes
   MIDI.read();
 }

 void noteOn(uint8_t note, uint8_t velocity) {
   int ledIndex = mapMidiNoteToLED(note, lowestNote, highestNote, NUM_LEDS); // Map MIDI note to LED index
   keysOn[ledIndex] = true;

     controlLeds(ledIndex, hue, saturation, brightness);  // Both use the same index

   Serial.println("Note On: " + String(note) + " mapped to LED: " + String(ledIndex));  // Debug print
 }

 void noteOff(uint8_t note, uint8_t velocity) {
   int ledIndex = mapMidiNoteToLED(note, lowestNote, highestNote, NUM_LEDS); // Map MIDI note to LED index
   keysOn[ledIndex] = false;
   Serial.println("Note Off: " + String(note) + " mapped to LED: " + String(ledIndex));  // Debug print
 }
lathoub commented 10 months ago

I don't know how to interpret the messages in the log window, as it still contains mapping information.

Are you sing the version via the Arduino library manager, or a direct clone from github? The Arduino library manager is not the latest as this lib is awaiting an update from the base Arduino MIDI lib.

Your basic setup looks OK, but i'm unsure about the Live routing setting.

Regarding latency, have a look at Latency

serifpersia commented 10 months ago

EDIT: Tried with latest repo clone and there is still the same issue. I verified led strip control code is not the cause since it works with non-human speed note on and note off messages, the issue is probably latency or something else, why would latency cause overwrite of note off messages that are still being processed by appleMIDI?

Playing to many notes too fast causes few note off messages to drop off? Can a buffer fix this but not cause any noticeable delay in live midi processing?

If so please let me know basic way to accomplish that to make sure every midi note gets processed no matter if this would create some delay I don't mind as long as its not too apparent.

The mapping func just maps midi notes to correct leds basically 1:2 ratio. There is code that checks received midi note on & note off and prints every 5 seconds how many note on note off have been seen, as you can see few missing note off messages, reason being I played 4+ note chords very fast and few note off didn't get received by the esp? I'm not using DAW so picked MIDI Device in that Live Routing section, and it seems it sends the midi data like that, I verified that the issue is not that it happens even if I use DAW(the virtual port rtpMIDI creates). I am using version from library manager in arduino ide, I will try to clone the repo and see if there is change. Can latency cause midi messages for some reason only note off to drop? Even with latency all midi data should arrive to esp correct? I already have direct midi support on my project but as second option and for testing I'm using midi over network since my midi device is not MIDI USB so I can't use it at the moment with USB Host capable ESP32S2/S3

lathoub commented 10 months ago

why would latency cause overwrite of note off messages that are still being processed by appleMIDI?

Latency is (probably) not the cause.

Playing to many notes too fast causes few note off messages to drop off?

Yes - but not only Note Off messages (any kind of message could be lost, so not just Note Off). I have described it here. Albeit that relates to the use the W5500 chip. The issue of dropped messages was addressed in the latest release.

Can you provide a buffer of the MIDI messages being send?

You can modify the test project to simulate a buffer that contains all the messages that are transmitted at once.

serifpersia commented 10 months ago

I'm not very familiar with modifying source code files .h .cpp and so on. Is there something I can add to my example code to show you whats going wrong? I only have dropped note off messages. I suspect latency causing timing issue and some note off messages just get lost.. Maybe because I'm hosting EspAsyncwebserver and using websockets I'm overloading the wifi interface? But when I play midi there is no interaction with the webserver or no events sent to websocket so I don't think thats the reason. I'm going to write a new code and share the output.

serifpersia commented 10 months ago

EDIT: Seems like using network midi is not fast enough since fastled.show is heavy function so ye after adding led react its hanging again... So yeah I don't think its the library. Just fastled can't get the midi data in time.. Or could be wifi takes over the cpu core so fastled timing will be off. Thanks for the help I'm going to close this as there is no issue with the library.

This is suprising, I created bare bones example and it seems no midi note off message got dropped. I don't have any server hosting or websockets.

Buffer empty not played notes last 5 seconds, | First cycle played 5 notes slowly once, output 5 note on and off's Second cycle spammed fast 5 notes roughly 5 times nothing got dropped it seems which I find crazy now lol

21:44:35.987 -> Note On Buffer:
21:44:35.987 -> Note Off Buffer:
21:44:35.987 -> Total Note Ons: 0
21:44:35.987 -> Total Note Offs: 0
21:44:40.968 -> Note On Buffer:
21:44:40.968 -> Note On: 28
21:44:40.968 -> Note On: 21
21:44:40.968 -> Note On: 23
21:44:40.968 -> Note On: 24
21:44:40.968 -> Note On: 26
21:44:40.968 -> Note Off Buffer:
21:44:41.003 -> Note Off: 23
21:44:41.003 -> Note Off: 21
21:44:41.003 -> Note Off: 26
21:44:41.003 -> Note Off: 28
21:44:41.003 -> Note Off: 24
21:44:41.003 -> Total Note Ons: 5
21:44:41.003 -> Total Note Offs: 5
21:44:45.970 -> Note On Buffer:
21:44:45.970 -> Note On: 28
21:44:45.970 -> Note On: 21
21:44:45.970 -> Note On: 26
21:44:45.970 -> Note On: 24
21:44:45.970 -> Note On: 23
21:44:45.970 -> Note On: 28
21:44:45.970 -> Note On: 24
21:44:46.003 -> Note On: 26
21:44:46.003 -> Note On: 23
21:44:46.003 -> Note On: 21
21:44:46.003 -> Note On: 28
21:44:46.003 -> Note On: 26
21:44:46.003 -> Note On: 21
21:44:46.003 -> Note On: 24
21:44:46.003 -> Note On: 23
21:44:46.003 -> Note On: 28
21:44:46.003 -> Note On: 26
21:44:46.003 -> Note On: 24
21:44:46.003 -> Note On: 23
21:44:46.003 -> Note On: 28
21:44:46.003 -> Note On: 21
21:44:46.003 -> Note On: 26
21:44:46.003 -> Note On: 24
21:44:46.003 -> Note On: 23
21:44:46.003 -> Note Off Buffer:
21:44:46.003 -> Note Off: 26
21:44:46.003 -> Note Off: 28
21:44:46.003 -> Note Off: 24
21:44:46.003 -> Note Off: 23
21:44:46.003 -> Note Off: 21
21:44:46.003 -> Note Off: 28
21:44:46.003 -> Note Off: 26
21:44:46.003 -> Note Off: 24
21:44:46.003 -> Note Off: 21
21:44:46.003 -> Note Off: 23
21:44:46.037 -> Note Off: 28
21:44:46.037 -> Note Off: 26
21:44:46.037 -> Note Off: 24
21:44:46.037 -> Note Off: 23
21:44:46.037 -> Note Off: 28
21:44:46.037 -> Note Off: 24
21:44:46.037 -> Note Off: 23
21:44:46.037 -> Note Off: 26
21:44:46.037 -> Note Off: 21
21:44:46.037 -> Note Off: 28
21:44:46.037 -> Note Off: 26
21:44:46.037 -> Note Off: 24
21:44:46.037 -> Note Off: 23
21:44:46.037 -> Note Off: 21
21:44:46.037 -> Total Note Ons: 24
21:44:46.037 -> Total Note Offs: 24

Here is new code

#include <WiFi.h>
#include <WiFiClient.h>
#include <WiFiUdp.h>
#include <AppleMIDI.h>

char ssid[] = "redacted"; // your network SSID (name)
char pass[] = "redacted";    // your network password (use for WPA, or use as key for WEP)

int8_t isConnected = 0;

APPLEMIDI_CREATE_DEFAULTSESSION_INSTANCE();

// Buffers for storing MIDI messages
const int bufferSize = 100;
byte noteOnBuffer[bufferSize];
byte noteOffBuffer[bufferSize];
int noteOnCount = 0;
int noteOffCount = 0;

void setup() {
Serial.begin(115200); // Initialize the serial monitor
Serial.println("Booting");

WiFi.begin(ssid, pass);

while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.println("Establishing connection to WiFi...");
}

Serial.println("Connected to network");

Serial.print(WiFi.localIP());
Serial.print(" Port ");
Serial.print(AppleMIDI.getPort());
Serial.print(" (Name ");
Serial.print(AppleMIDI.getName());
Serial.println(")");

MIDI.begin();

AppleMIDI.setHandleConnected([](const APPLEMIDI_NAMESPACE::ssrc_t & ssrc, const char* name) {
    isConnected++;
    Serial.print("Connected to session ");
    Serial.print(ssrc);
    Serial.print(" Name ");
    Serial.println(name);
});

AppleMIDI.setHandleDisconnected([](const APPLEMIDI_NAMESPACE::ssrc_t & ssrc) {
    isConnected--;
    Serial.print("Disconnected ");
    Serial.println(ssrc);
});

MIDI.setHandleNoteOn([](byte channel, byte note, byte velocity) {
    // Store Note On messages in the buffer
    noteOnBuffer[noteOnCount] = note;
    noteOnCount = (noteOnCount + 1) % bufferSize;
});

MIDI.setHandleNoteOff([](byte channel, byte note, byte velocity) {
    // Store Note Off messages in the buffer
    noteOffBuffer[noteOffCount] = note;
    noteOffCount = (noteOffCount + 1) % bufferSize;
});
}

void printAndClearBuffers() {
Serial.println("Note On Buffer:");
for (int i = 0; i < noteOnCount; i++) {
    Serial.print("Note On: ");
    Serial.println(noteOnBuffer[i]);
}

Serial.println("Note Off Buffer:");
for (int i = 0; i < noteOffCount; i++) {
    Serial.print("Note Off: ");
    Serial.println(noteOffBuffer[i]);
}

// Summary
Serial.print("Total Note Ons: ");
Serial.println(noteOnCount);
Serial.print("Total Note Offs: ");
Serial.println(noteOffCount);

// Clear the buffers and reset the counters
noteOnCount = 0;
noteOffCount = 0;
}

void loop() {
MIDI.read();

static unsigned long lastPrintTime = 0;
unsigned long currentTime = millis();

// Print and clear buffers every 5 seconds
if (currentTime - lastPrintTime >= 5000) {
    printAndClearBuffers();
    lastPrintTime = currentTime;
    }
}
serifpersia commented 10 months ago

Just wanted to report using RTOS I created a midiTask to handle midi and I have no more issues. 1ms tick and 2048 memory size seems to do the trick.

void midiTask(void *pvParameters) {
   while (1) {
MIDI.read(); // Handle MIDI messages
vTaskDelay(pdMS_TO_TICKS(1)); // Adjust the delay as needed
  }
}

void setup()
{
  // Create the MIDI task
xTaskCreatePinnedToCore(midiTask, "MIDITask", 2048, NULL, 1, &midiTaskHandle, 0);
MIDI.begin();
}

void loop()
{
}
lathoub commented 10 months ago

Thanks for the clarification @serifpersia (and well spotted)