logue-cli load issue on Linux

[rwolfson]

Hi, I am trying to upload third-party oscillators using logue-cli to a Minilogue XD on Linux. Unfortunately, I seem to get different symptoms consistently for different respective oscillators. One symptom is a timeout, with the message ALSA: seq_midi: MIDI output buffer overrun showing up in dmesg at the time of the issue. The other symptom is an actual midi error from the application, without the same message in dmesg. I am speculating that these could be similar. I am wondering what steps I could try, to help get to the bottom of the issue. I was able to upload a self-compiled version of the waves sample oscillator, which makes me wonder if it's something specific with size or something else about the oscillators that causes the MIDI error. I tried both compiling the third-party oscillators myself and using the precompiled version, without luck. Please let me know if more verbose output would help; I can see a wall of hex when I run the tool in debug mode, before it times out.

r@r-desktop:~/scratch/korg_testing/eurorack-prologue/eurorack_minilogue-xd$ logue-cli load -i 2 -o 2 -u mo2_va.mnlgxdunit 
> Parsing minilogue xd unit archive
> Parsing manifest
> Parsing unit binary payload
> Handshaking...
> Target platform: "minilogue xd"
> Target module: "Oscillator"
> No slot specified, using first available.
size: 2bdc crc32: ba535d39
Error: Unit load request timed out.
Error: Failed to load unit.
r@r-desktop:~/scratch/korg_testing/eurorack-prologue/eurorack_minilogue-xd$ logue-cli load -i 2 -o 2 -u mo2_add.mnlgxdunit 
> Parsing minilogue xd unit archive
> Parsing manifest
> Parsing unit binary payload
> Handshaking...
> Target platform: "minilogue xd"
> Target module: "Oscillator"
> No slot specified, using first available.
size: 4174 crc32: e8c31dc6

MidiOutAlsa::sendMessage: error sending MIDI message to port.

Error: Unit load request timed out.
Error: Failed to load unit.

OS version: Linux Mint 19.1 (based on Ubuntu 18.04) Minilogue XD firmware version: 2.10 Kernel version: 4.15.0-88 alsa-base package version: 1.0.25

[tokuhira]

Did you specify a slot with the "-s" option as shown in the example in README.md? Like this:

$ ./logue-cli load -u my_oscillator.mnlgxdunit -s 2

The 2 as slot number in above is just a example. to determine the slot number, you should first check the list using the "-m" option. It is better to be careful whether the number starts from 0 or 1.

[rwolfson]

Hi, I had tried that, as well, when trying to overwrite the second "waves" oscillator, but unfortunately it had the same result. Re-tried output below; I was able to clear the second oscillator from the slot, so this is against an empty slot.

r@r-desktop:~/scratch/korg_testing/eurorack-prologue/eurorack_minilogue-xd$ logue-cli load -i 2 -o 2 -s 1 -u mo2_va.mnlgxdunit 
> Parsing minilogue xd unit archive
> Parsing manifest
> Parsing unit binary payload
> Handshaking...
> Target platform: "minilogue xd"
> Target module: "Oscillator"
size: 2bdc crc32: ba535d39
Error: Unit load request timed out.
Error: Failed to load unit.

r@r-desktop:~/scratch/korg_testing/eurorack-prologue/eurorack_minilogue-xd$ logue-cli probe -i 2 -o 2 -m osc
> Device: minilogue xd
> System version: 2.10
> Logue API version: 1.01-0
> Oscillator status:
[0]: "waves" v1.00-0 api:1.00-0 did:00000000 uid:00000000
[1]: free.
[2]: free.
[3]: free.
[4]: free.
[5]: free.
[6]: free.
[7]: free.
[8]: free.
[9]: free.
[10]: free.
[11]: free.
[12]: free.
[13]: free.
[14]: free.
[15]: free.

[tokuhira]

My NTS-1 has the following size limits, what are the numbers on your Minilogue XD? Is its size enough compared to mnlgxdunit files? I would like to try it out, so if it's an open oscillator I'd be glad that you share it.

$ logue-cli probe -i 1 -o 1
> Device: nutekt digital
> System version: 1.10
> Logue API version: 1.01-0
> Available modules:

Modulation FX: [ slot_count: 16, max_payload_size: 8180, max_load_size: 6144 ]
Delay FX: [ slot_count: 8, max_payload_size: 16368, max_load_size: 12288 ]
Reverb FX: [ slot_count: 8, max_payload_size: 16368, max_load_size: 12288 ]
Oscillator: [ slot_count: 16, max_payload_size: 36848, max_load_size: 32768 ]

[rwolfson]

Hi, I have a similar 32K limit on oscillators. All of the oscillator files are under 32K on disk, and a logue-cli "check" of the ones I have tried to upload has come back okay.

The oscillators I am trying are from https://github.com/peterall/eurorack-prologue ; pre-compiled versions are also available on the GitHub "releases" page for that repo.

r@r-desktop:~/scratch/korg_testing/eurorack-prologue/eurorack_minilogue-xd$ logue-cli probe -i
 2 -o 2                                                                                                                
> Device: minilogue xd                                                                                                 
> System version: 2.10                                                                                                 
> Logue API version: 1.01-0                                                                                           
> Available modules:                                                                          

Modulation FX: [ slot_count: 16, max_payload_size: 8180, max_load_size: 6144 ]       
Delay FX: [ slot_count: 8, max_payload_size: 16368, max_load_size: 12288 ]          
Reverb FX: [ slot_count: 8, max_payload_size: 16368, max_load_size: 12288 ]                                                                   
Oscillator: [ slot_count: 16, max_payload_size: 36848, max_load_size: 32768 ]

r@r-desktop:~/scratch/korg_testing/eurorack-prologue/eurorack_minilogue-xd$ ls -l mo2_*
-rw-r--r-- 1 r r 11371 Mar 31 19:19 mo2_add.mnlgxdunit
-rw-r--r-- 1 r r  8670 Mar 31 19:18 mo2_fm.mnlgxdunit
-rw-r--r-- 1 r r 11619 Mar 31 19:19 mo2_grn.mnlgxdunit
-rw-r--r-- 1 r r  9485 Mar 31 19:18 mo2_string.mnlgxdunit
-rw-r--r-- 1 r r  8200 Mar 31 19:19 mo2_va.mnlgxdunit
-rw-r--r-- 1 r r 14507 Mar 31 19:18 mo2_wsh.mnlgxdunit
-rw-r--r-- 1 r r 15838 Mar 31 19:18 mo2_wta.mnlgxdunit
-rw-r--r-- 1 r r 19940 Mar 31 19:19 mo2_wtb.mnlgxdunit
-rw-r--r-- 1 r r 21502 Mar 31 19:19 mo2_wtc.mnlgxdunit
-rw-r--r-- 1 r r 19844 Mar 31 19:18 mo2_wtd.mnlgxdunit
-rw-r--r-- 1 r r 21108 Mar 31 19:18 mo2_wte.mnlgxdunit
-rw-r--r-- 1 r r 19893 Mar 31 19:19 mo2_wtf.mnlgxdunit

[tokuhira]

Hi, thank you for sharing the nice package. The issue has been reproduced on my Ubuntu 18.04 too. The "dmesg" says following. Do you have a way to configure the buffer size? Unfortunately, I am not good at Linux sound architecture and have no knowledge.

[  254.284190] usb 1-4: new full-speed USB device number 4 using xhci_hcd
[  254.433367] usb 1-4: New USB device found, idVendor=0944, idProduct=0145, bcdDevice= 1.00
[  254.433372] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[  254.433375] usb 1-4: Product: NTS-1 digital kit
[  254.433378] usb 1-4: Manufacturer: KORG INC.
[  254.542783] usbcore: registered new interface driver snd-usb-audio
[  333.898165] dump_midi: 37 callbacks suppressed
[  333.898167] ALSA: seq_midi: MIDI output buffer overrun

[rwolfson]

It's good to see that the behavior is at least consistent and reproducible. I was thinking something along the buffer size lines, as well. I did a bit of searching before I opened the ticket and came to the conclusion that this looked like something that needs to change in the application. It was difficult to find resources online, but I believe I saw in a couple of places that the default buffer size was a system page (4096 bytes on x86). The demo "waves" oscillator seems to be under this threshold, at least in its compiled form on disk, which might explain how it gets by without overrunning the buffer.

The closest thing I've found is https://alsa-devel.alsa-project.narkive.com/6w35hffF/sysex-overflow-when-using-the-midi-sequencer-event-interface , which sounds like a similar symptom, and indicates that the "correct" fix is that this should be fixed in the software. There's also https://marc.info/?l=alsa-devel&m=154580126006810&w=2 describing a bit more about an easy way to fill the buffer. That said, I tried to take a crack at a workaround by changing this via /proc or .asoundrc but was unsuccessful, most relevant asound config online seems to be for audio and not MIDI; proc itself doesn't seem to have any "settings" files to write to for the MIDI connection, unlike audio interfaces.

r@r-desktop:/proc/asound/xd$ amidi -l
Dir Device    Name
IO  hw:4,0,0  minilogue xd MIDI 1
IO  hw:4,0,1  minilogue xd MIDI 2

r@r-desktop:/proc/asound/xd$ ls -l
total 0
-r--r--r-- 1 root root 0 Apr  4 11:55 id
-r--r--r-- 1 root root 0 Apr  4 11:55 midi0
-r--r--r-- 1 root root 0 Apr  4 11:55 usbbus
-r--r--r-- 1 root root 0 Apr  4 11:55 usbid
r@r-desktop:/proc/asound/xd$ cat midi0
minilogue xd

Output 0
  Tx bytes     : 0
Output 1
  Tx bytes     : 10887
Input 0
  Rx bytes     : 0
Input 1
  Rx bytes     : 3834

[scotus-49]

I have a similar problem to the above with my NTS-1 on Ubuntu:

> Parsing nutekt digital unit archive
> Parsing manifest
> Parsing unit binary payload
> Handshaking...
> Target platform: "nutekt digital"
> Target module: "Oscillator"
size: 1204 crc32: 9f12382c
Error: Unit load request timed out.
Error: Failed to load unit.

[aminixduser]

I have figured out a way to load a user oscillator on Ubuntu, though this should work any linux system that logue-cli will run on.

Use the "-d" switch with your load command. Ex. ./logue-cli-linux64-0.07-2b/logue-cli load -d -u ../../../eurorack_minilogue-xd/mo2_wta.mnlgxdunit -i 2 -o 2 -s 5

the last part on the screen (which is on one line BTW) should be the complete MIDI Sysex command for loading the oscillator It has some junk around the front and back, and commas, so I wrote a Perl script to remove them

So my flow works like this:

./logue-cli-linux64-0.07-2b/logue-cli load -d -u ../../../eurorack_minilogue-xd/mo2_wta.mnlgxdunit -i 2 -o 2 -s 5 > mo2_wta.log ./make_osc_load_sysex.pl -i mo2_wta.log

Then you find the ALSA rawmidi hardware port like this: @rwolfson was on to something there, so this helped me figure this out, thank you

amidi -l

The second minilogue xd port listed is the one that worked for me My Perl script puts the MIDI bytes with spaces in a file named "osc_sysex"

I send the oscillator to the minilogue-xd like this:

amidi -p hw:2,0,1 -S `cat ./osc_sysex`

I've loaded several of the free eurorack oscillators this way.

The real issue is that the program uses an ALSA sequencer port, and not an ALSA rawmidi port to send the sysex messages, the sequencer port limits the length of any message, while the ALSA rawmidi "port" has no such limitations.

I've attached my Perl script released under the MIT open source license, so it's just an attribution, and has a disclaimer in case it doesn't work for you... You may need to install perl on your linux system to use it.

make_osc_load_sysex.pl.gz

[schollz]

@aminixduser You are brilliant! This works well for me. I was having trouble loading an effect but this worked for me.

I had trouble with your perl script, but I used your approach with just sed.

Create the load log:

$ logue-cli load -v -i 1 -o 1 -s 0 -u tempo_delay.ntkdigunit -d > load.log

(my NTS-1 is on i/o ports 1 and 1)

Extract just the sysex

$ tail -n1 load.log| sed 's/,//g' | sed 's/}//g' | sed 's/{//g' | sed 's/>//g' | sed 's/^ *//g' > load.sysex

Send over amidi

$ amidi -l
Dir Device    Name
IO  hw:2,0,0  NTS-1 digital kit MIDI 1

$ amidi -p hw:2,0,0 -S `cat load.sysex`

Works beautifully. What a nice and simple solution, such a cool machine!

[aminixduser]

@schollz That's a great way, I should have thought of the tail -n1 to get the last line as many times as I've used it. I rarely use sed, but this is a great little batch of regex one liners.

[dromer]

The work-around with the perl script also works for me! (the shell one-liner should also be fine I guess)

Hope Korg will update the logue-cli to deal with this though.

[simeka]

@schollz and @aminixduser thx for the cool workaround! I tried both your methods (on ubuntu 16.04) but failed to send the extracted sysex file to my minilogue with amidi -p hw:2,0,1 -S `cat osc_sysex` which will take less than a second to finish, but i cant find any new osc on the minilogue...did I miss something there? or do you have any hints whats going wrong? thanks in advance!

Update: ok, just after positing here I was going over the settings in the minilogue again and changed my midi port from 3 back to 1. This somehow did the trick, in case someone else is having issues.

[apiel]

I am not very good with bash script but something like this should be a starting point:

#!/bin/sh

echo "Load with params: $@";

temp_file=$(mktemp)

echo "Tmp file $temp_file"

in_port=$(./logue-cli probe -l | grep NTS-1 | grep in | awk '{print $2}' | sed 's/.$//')
out_port=$(./logue-cli probe -l | grep NTS-1 | grep out | awk '{print $2}' | sed 's/.$//')
amidi_port=$(amidi -l | grep NTS-1 | awk '{print $2}')

./logue-cli load -d -i $in_port -o $out_port -u $@  > $temp_file
tail -n1 $temp_file| sed 's/,//g' | sed 's/}//g' | sed 's/{//g' | sed 's/>//g' | sed 's/^ *//g' > "$temp_file.sysex"
amidi -p $amidi_port -S `cat $temp_file.sysex`

[andy-twosticks]

Sad to say this workaround doesn't work for me, either with the Perl script or the tail ... sed thing.

amidi takes a fraction of a second to execute with no errors -- but the module has not been loaded.

[andy-twosticks]

Got it! On my NTS-1 I reset the midi channel to 0, and then it worked!

[ratherlargerobot]

I wrote a Python 3 script that takes the excellent work that @aminixduser shared and expands upon it a bit. Here it is:

logue-load.gz

Basically, this logue-load Python script automates some of the other steps described in the process of using make_osc_load_sysex.pl. It's using amidi, and pulling the SysEx data out of the logue-cli load debug output, in pretty much exactly the same manner.

To use the program, gunzip it, chmod it to be executable, and either put it somewhere in your $PATH or reference it with ./, just like any other script on Linux.

It accepts -i/--inport, -o/--outport, -u/--unit, and optionally -s/--slot, just like logue-cli. It does not accept any of the other arguments. This is a special-purpose tool to work around the limitation of not being able to upload larger unit files to the Korg NTS-1 and other devices on Linux.

Below is an example session in action. Most of the commands are just illustrating how to find the device and such using logue-cli, which should already be familiar. The new addition here is the use of this new logue-load Python script to do the upload.

logue-load will first try to perform an upload using the logue-cli command (assuming it is in your $PATH). If that succeeds (because the unit is small enough to be uploaded via logue-cli), then the program exits successfully, and you're all done. So far this is just like using logue-cli directly. However, if this fails, but the SysEx data can be extracted from logue-cli in debug mode, then the logue-load Python script will try one more thing. If it gets the SysEx data from the failed logue-cli attempt, it will then run amidi -l, try to find the device, and then upload the SysEx data to the device using amidi. For safety, if more than one Korg device that could accept this data is found (e.g. more than one NTS-1/minilogue xd/prologue in total), then it will refuse to perform the upload. Finally, if the script found exactly one device that is eligible for an amidi upload, the script will try to upload the SysEx data.

Ideally, the logue-cli program itself would use the rawmidi ALSA method used by amidi. But until that happens, this should make things a bit more convenient for Linux users.

#
# logue-load usage information
#
$ logue-load 
usage: logue-load [-h] -u UNIT -i IN_PORT -o OUT_PORT [-s SLOT]
logue-load: error: the following arguments are required: -u/--unit, -i/--inport, -o/--outport

#
# find the device using logue-cli
#
$ logue-cli probe -l
  Available MIDI inputs:
    in  0: Midi Through:Midi Through Port-0 14:0
    in  1: NTS-1 digital kit:NTS-1 digital kit MIDI 1 24:0

  Available MIDI ouputs:
    out 0: Midi Through:Midi Through Port-0 14:0
    out 1: NTS-1 digital kit:NTS-1 digital kit MIDI 1 24:0

#
# we're starting with no custom oscillators installed
#
$ logue-cli probe -i 1 -o 1 -m osc 
> Device: nutekt digital
> System version: 1.20
> Logue API version: 1.01-0
> Oscillator status:
[0]: free.
[1]: free.
[2]: free.
[3]: free.
[4]: free.
[5]: free.
[6]: free.
[7]: free.
[8]: free.
[9]: free.
[10]: free.
[11]: free.
[12]: free.
[13]: free.
[14]: free.
[15]: free.

#
# try to load a unit file that is too big for logue-cli to upload
#
$ logue-cli load -i 1 -o 1 -u cainCZ_1.3.0.ntkdigunit 
> Parsing nutekt digital unit archive
> Parsing manifest
> Parsing unit binary payload
> Handshaking...
> Target platform: "nutekt digital"
> Target module: "Oscillator"
> No slot specified, using first available.
size: 1430 crc32: 2546e09
Error: Unit load request timed out.
Error: Failed to load unit.

#
# load the unit file with logue-load
# this will first try with logue-cli, which will fail, but also
# give logue-load the SysEx data. then, logue-load will run
# amidi -l to find the device, and then upload the SysEx data
# to the device using amidi
#
$ logue-load -i 1 -o 1 -u cainCZ_1.3.0.ntkdigunit

#
# it worked! the oscillator was uploaded via amidi
#
$ logue-cli probe -i 1 -o 1 -m osc 
> Device: nutekt digital
> System version: 1.20
> Logue API version: 1.01-0
> Oscillator status:
[0]: "CainCZ" v1.03-0 api:1.01-0 did:00000000 uid:00000000
[1]: free.
[2]: free.
[3]: free.
[4]: free.
[5]: free.
[6]: free.
[7]: free.
[8]: free.
[9]: free.
[10]: free.
[11]: free.
[12]: free.
[13]: free.
[14]: free.
[15]: free.

[Jackojc]

I made the script a bit more succinct. removed extraneous use of grep & sed and removed the use of temporary files.

#!/usr/bin/env sh

[ -z "$@" ] && echo "no params" && exit 1

# Read ports.
read -r midi_in midi_out amidi <<EOF
$(logue-cli probe -l | awk '/NTS-1/ {ORS=" "; gsub(/[[:punct:]]/, ""); print $2}')$(amidi -l | awk '/NTS-1/ {print $2}')
EOF

echo "in: $midi_in out: $midi_out hw: $amidi"

# Try loading unit. If it times out, parse sysex messages and send them using amidi instead.
out="$(logue-cli load -d -i $midi_in -o $midi_out -u $@ 2> /dev/null)"

[ $? -ne 0 ] && (
    echo "logue-cli failed! trying amidi."; echo "$out" | awk 'END {gsub(/[[:punct:]]/, ""); print}' | xargs amidi -p $amidi -S
) && echo "done!"

[Noir-]

Great workarounds guys! But please @etienne-korg could you have a look into this? If you don't have the time to fix it, just open the sources and we would be pleased to fix it for you :)

[dagargo]

It's possible to resize the MIDI output buffer with this command, which solves this issue for me.

root@host:/# echo "65536" > /sys/module/snd_seq_midi/parameters/output_buffer_size

I've discovered this recently but looks like it's an old feature of the snd_seq_midi module that can be set and got by using the /sys filesystem.

As others have said, using the sequencer interface in Linux for SysEx or the like is totally inadvisable but cross-platform libraries such as PortMidi or RtMidi do this.

It could be fixed permanently if we configure the parameter so the OS uses it when loading the module but this is probably undesirable as the smaller the buffer the smaller the latency. After all, that's why we have 2 different MIDI API's.

[jakobfridesjo]

Here's a solution based on the workaround by @dagargo but it also restores the original buffer size Just save it as a script and use it the same way as you would use logue-cli

#!/usr/bin/env sh

# Save buffer size
buffer_size=$(cat /sys/module/snd_seq_midi/parameters/output_buffer_size)

# Set temporary buffer size
echo "65536" > /sys/module/snd_seq_midi/parameters/output_buffer_size

# Send arguments to logue-cli
logue-cli "$@"

# Restore buffer size
echo $buffer_size > /sys/module/snd_seq_midi/parameters/output_buffer_size

[lentferj]

root@host:/# echo "65536" > /sys/module/snd_seq_midi/parameters/output_buffer_size

solved it for me, too. Still, it is also important to set the device to MIDI #1, at least on my xd.