Add network support by omersiar

[probonopd]

Continuation from https://github.com/probonopd/MiniDexed/pull/744

This adds wired and wireless network for supported models:

• rtpMIDI a.k.a. "AppleMIDI" over WLAN
• UDP MIDI
• FTP over WLAN (e.g., to update without takin the microSD card out of the Pi)
• DNS-SD service discovery (a.k.a. "Bonjour", "Zeroconf")

Closes https://github.com/probonopd/MiniDexed/issues/43 Thanks @omersiar

This allows us to send MIDI e.g., from DAWs, Dexed on the PC, or even sysex for voices from sites like patches.fm

Documentation: https://github.com/probonopd/MiniDexed/wiki/Networking#ethernet-and-wlan-support

[github-actions[bot]]

Build for testing: MiniDexed_2024-11-09-6d76e38 Use at your own risk.

[probonopd]

https://github.com/dwhinham/mt32-pi/wiki/Networking%3A-RTP-MIDI-%28AppleMIDI%29 says

mDNS/"Bonjour" for zero-configuration is not implemented, so mt32-pi will not be auto-discovered by your RTP-MIDI host software. You must add mt32-pi as a peer manually.

In the meantime, the awesome @rsta2 has implemented this. Do you think you could incorporate it?

• https://github.com/rsta2/circle/discussions/427

An updated class CmDNSResponderis on the develop branch now with a test in test/mdns-publisher. The class now supports multiple services in one instance of the class and multiple TXT entries.

This means that we could advertise both RTP-MIDI and FTP (if enabled) on the network, making it much simpler to connect (without having to remember the IP address).

• https://github.com/rsta2/circle/tree/master/test/mdns-publisher

[github-actions[bot]]

Build for testing: MiniDexed_2024-11-09-21db866 Use at your own risk.

[omersiar]

One can already connect to minidexed by its hostname (defaults to "minidexed") if the router supports home network domain. I am aware of the mdns publisher but I don't know if minidexed checkouts to point where it is implemented

[probonopd]

Yes, but that won't cause it to show up in music or FTP applications on the PC/Mac automagically.

So far we are not using the mdns publisher in MiniDexed yet at all, so it would be a matter of porting it over from https://github.com/rsta2/circle/tree/master/test/mdns-publisher.

Probably something for a later/separate pull request?

[omersiar]

Oh sorry I meant the circle as a submodule. Currently minidexed do not checkout to required point for <circle/net/mdnspublisher.h>

[probonopd]

Probably have to adjust

https://github.com/probonopd/MiniDexed/blob/main/submod.sh

so that the needed branch/commit is checked out?

We are using this script since editing git submodules on the GitHub web interface is less than user friendly.

[omersiar]

Yeah this ⬆️, but changing this can break things. Let me check if latest circle release does break anything.

[probonopd]

Testing on a Windows 11 PC:

After the usual

MiniDexed
Loading...

it says

Network      IP
192.168.0.122     >

(Minor nitpick: The > suggests there is a submenu while there isn't; I think we should leave the > away here)

Trying to connect via FTP from the FTP client built into Windows, I cannot get a connection using ftp://minidexed (which could well be my router's fault). When using ftp://192.168.0.122 it asks for username and password, I used admin for both.

I do see a SD directory which contains the whole SD card contents. Cool! 👍

Now I'll move on to testing the MIDI aspects...

[omersiar]

It compiles with Step48 with these warnings:

net/ftpworker.cpp: In member function 'bool CFTPWorker::_ZN10CFTPWorker4PortEPKc.part.0(const char*)':
net/ftpworker.cpp:464:9: warning: 'char* strncpy(char*, const char*, size_t)' specified bound 512 equals destination size [-Wstringop-truncation]
  464 |  strncpy(Buffer, pArgs, sizeof(Buffer));
      |  ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
net/ftpworker.cpp: In member function 'const TDirectoryListEntry* CFTPWorker::BuildDirectoryList(size_t&) const':
net/ftpworker.cpp:363:11: warning: 'char* strncpy(char*, const char*, size_t)' specified bound 6 equals destination size [-Wstringop-truncation]
  363 |    strncpy(VolumeName, VolumeNames[i], sizeof(VolumeName));
      |    ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
minidexed.cpp: In member function 'void CMiniDexed::SetVoiceName(std::string, unsigned int)':
minidexed.cpp:1656:9: warning: 'char* strncpy(char*, const char*, size_t)' specified bound 10 equals destination size [-Wstringop-truncation]
 1656 |  strncpy(Name, VoiceName.c_str(),10);
      |  ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~
../Synth_Dexed/src/dexed.cpp: In member function 'void Dexed::setName(char*)':
../Synth_Dexed/src/dexed.cpp:1699:10: warning: 'char* strncpy(char*, const char*, size_t)' output may be truncated copying 10 bytes from a string of length 155 [-Wstringop-truncation]
 1699 |   strncpy(name, (char*)&data[DEXED_VOICE_OFFSET + DEXED_NAME], 10);
      |   ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[omersiar]

(Minor nitpick: The > suggests there is a submenu while there isn't; I think we should leave the > away here)

I wanted to give impression that this is not stuck and move the encoder one step to see actual menu. Dirty hack. I am thinking to add a menu item for it.

[probonopd]

I actually like that the IP address shows up automatically and doesn't go away until you press a button. Let's keep that!

Maybe we could do the following: In the main menu

            MiniDexed
TG 1                >

we could show

IP      192.168.0.122
TG 1                >

instead, until the first button press. What do you think?

[probonopd]

...RTP-MIDI on Windows 11 using loopMIDI and rtpMIDI works very nicely as described here. Playing MIDI from REAPER to MiniDexed over WLAN 👍

When I use Dexed, I can set rtpMIDI up like this

and Dexed like this

and now I can send voices from Dexed to MiniDexed over WLAN, and can use Dexed as a GUI "programmer" for MiniDexed, meaning that any changes in Dexed are reflected in MiniDexed.

[probonopd]

It compiles with Step48 with these warnings

Most likely this points to minor bugs that were there before but didn't trigger a warning so far?

[omersiar]

Directory listing is now working

https://github.com/probonopd/MiniDexed/commit/19908839d8b80aed4864a4477cd874aa8191321c

[probonopd]

Raspberry Pi 5 build seems to fail now. Apparently we need to use circle master.

[github-actions[bot]]

Build for testing: MiniDexed_2024-11-09-1183cf8 Use at your own risk.

[probonopd]

Trying to update MiniDexed by FTPing over these files seems to crash the FTP client built into Windows 11. It doesn't even ask whether to overwrite the files, it just seems to hang.

Need to check with other FTP clients.

Update: With WinSCP it works. But it has frequent reconnects and upload speed is around 200KB/s. During upload, MIDI playing via WLAN has audible jitter. Well, I guess it shows that we are not running a true multitasking OS.

Update 2: From the ftp client built into Mac OS X 10.6.8 I can connect to ftp://192.168.0.122, but the window remains empty (no "SD" folder shown) and hence it is unusable. Strange.

[probonopd]

Directory listing is now working

I can output MIDI to MiniDexed with Logic Pro 9.1.8 on Mac OS X 10.6.8; autodiscovery is working 👍

On the Mac:

• Open Audio MIDI Setup
• In the MIDI Studio window, double-click Network
• In the MIDI Network Setup window, MiniDexed will appear after a while in "Directory". Set it up like below.

[github-actions[bot]]

Build for testing: MiniDexed_2024-11-09-cccb19e Use at your own risk.

[omersiar]

Trying to update MiniDexed by FTPing over these files seems to crash the FTP client built into Windows 11. It doesn't even ask whether to overwrite the files, it just seems to hang.

For me it asks for individual file, did not try to upload more than one file. advised to use only 1 connection

Update 2: From the ftp client built into Mac OS X 10.6.8 I can connect to ftp://192.168.0.122, but the window remains empty (no "SD" folder shown) and hence it is unusable. Strange.

My mac tries to open multiple connections (more than 5 I see) and ftpdaemon complains about maximum number of connections

[probonopd]

In rtpMidi, it shows up as "MiniDexed" now (like Apple, let's avoid too many technical words).

[omersiar]

Also mdns works briefly after booting, after awhile if it is not cached by the network peers, can not be able to discover minidexed.

https://github.com/rsta2/circle/discussions/427#discussioncomment-11196574

[probonopd]

I had mine play some MIDI sent over WLAN from REAPER and after a while (within one hour or so) MiniDexed just froze, making a permanent unchanging sound. Might be a power supply issue, but worth watching out for.

Update: At least on this particular hardware setup, I seem to be able to reproduce this issue. This time it happened around ~30 minutes.

Update 2: After ~30 minutes it froze again, while playing in a loop the attached MIDI from REAPER over rtpMIDI on Windows 11. Since it is always happening after ~30 minutes: Could it be that something is flowing over?

Update 3: After ~39 minutes it froze again

Update 4: ~36 minutes

Update 5: ~36 minutes

midi_export.zip

[Banana71]

FTP connection with Total Commander works. Every now and then my miniDexed shows the IP address again. The Raspberry Pi4 is built into a metal case, could that be why the WLAN connection is not good enough?

However, the data transfer is not really bad at around 2 MB/s.

I am very happy that this function is finally being introduced into the miniDexed, so you have more flexibility when building the case and no longer necessarily need easy access to the SD card. Thank you @omersiar & @probonopd

[probonopd]

@Banana71 can you send MIDI to it and it plays for longer than half an hour without issues?

[Banana71]

@Banana71 can you send MIDI to it and it plays for longer than half an hour without issues?

After installation I was able to control the miniDexed via WLAN, but then the miniDexed crashed and since then I have no longer been able to establish a connection via WLAN. 🤷‍♂️

[omersiar]

Do you have heatsinks for the Pi4?

[probonopd]

When mine freezes, power cycling makes it work again.

[Banana71]

It takes up to several minutes for the connection to be established via rtpMIDI. Once the connection is established, it is more or less stable (repeated crash). After changing a performance, there is a delay of about 0.5 seconds until the sound can be heard. The RPi 4 does not get hot.

[github-actions[bot]]

Build for testing: MiniDexed_2024-11-09-042f747 Use at your own risk.

[omersiar]

Without console logs it is hard to guess what is getting wrong. Maybe we can put it to SD card so that can be read via FTP. https://github.com/rsta2/circle/issues/93#issuecomment-957345526 from this discussion I am thinking WLAN code can be improved further.

Also my RPi 3 was getting lightning icon during WLAN initialization that wasn't there before, so after switching to a new good USB micro cable and power source it got away. BYW I noticed, during lightning icons (voltage brownouts), the red led indicator turns off for some reason

[github-actions[bot]]

Build for testing: MiniDexed_2024-11-09-56619b7 Use at your own risk.

[probonopd]

As it is currently implemented, using this will expose the WLAN password openly to anyone on the network (by being stored in clear text on a FTP server with a default password). While I don't think that a hobby synthesizer needs to be built like an enterprise server regarding security, I do think we should find some solution around that (e.g., hiding the wpa_supplicant.conf file in FTP). Possibly this should be taken care of in Circle so that everyone uses it could benefit? (And no, setting an FTP password would not improve the situation much as FTP is unencrypted.)

[omersiar]

I wonder if ftpdaemon respects FAT file attributes, if then the files not needed to touched by users can be hidden

[probonopd]

Every few minutes I get "network disconnected" lines but it seems to reconnect immediately, because the mp3 (posted above) keeps playing over WLAN.

The red line appears when MiniDexed stalls after ~30-40 minutes of playing the mp3 over WLAN.

[probonopd]

All that network logging should be optional, @omersiar do you think you could introduce a VerbosityEnabled option and only print out any network related messages if it is set to 1?

https://github.com/probonopd/MiniDexed/wiki/Files#debug

We made all the other messages opt-in because it is known that printing messages to HDMI can cause audio issues on some RPi models.

[probonopd]

Right now it seems to have a tendency for stuck notes (notes playing forever) when using an attached MIDI controller (keyboard).

To reproduce, just play a glissando up and down (put finger on keyboard and slide left and right all over the range of the keyboard).

Update: Same when networking is disabled.

Update 2: With the latest build from main, MiniDexed_2024-09-29-8c18e60.zip, I cannot reproduce the "stuck notes when playing glissando directly after booting". I am not saying that the networking code is necessarily at fault; maybe updating the dependencies introduced this?

Update 3: It is not only directly after booting.

Update 4: Updating circle from fff3764 to c243194 may have introduced this issue, as https://github.com/probonopd/MiniDexed/pull/750 is also affected.

[omersiar]

I see, stuck notes did not happen to me, tasks exceed can happen when to much new mdns task created. okay I may have idea for that. Since your wifi connection is not reliable it creates a new task every time network reconnects. or maybe it is because of TTL setting.

[probonopd]

MiniDexed hanging already was the same before I shortened the TTL. Stuck notes seem to be unrelated to the networking (see above).

[omersiar]

https://github.com/omersiar/MiniDexed/commit/3e77f72a7ab943edaf399f17e9569a9ac20e5ba3 https://github.com/omersiar/MiniDexed/commit/17d905a1ae150d56a57c85486d5d87ea7a27d47f

These may help with the unreliable wlan connections, if my understanding from this discussion is correct > https://github.com/rsta2/circle/issues/93#issuecomment-957427536

[omersiar]

Update 4: Updating circle from fff3764 to c243194 may have introduced this issue, as #750 is also affected.

Yeah, this was predicted. Unfortunately my low level c skills are not up to this extend.

[omersiar]

All that network logging should be optional, @omersiar do you think you could introduce a VerbosityEnabled option and only print out any network related messages if it is set to 1?

https://github.com/probonopd/MiniDexed/wiki/Files#debug

We made all the other messages opt-in because it is known that printing messages to HDMI can cause audio issues on some RPi models.

Other than network disconnect and init messages, rest is coming from libs.

[probonopd]

https://github.com/omersiar/MiniDexed/commit/3e77f72a7ab943edaf399f17e9569a9ac20e5ba3 https://github.com/omersiar/MiniDexed/commit/17d905a1ae150d56a57c85486d5d87ea7a27d47f

I am having a hard time applying these cleanly, could you send a pull request based on the probonopd/MiniDexed branch networking please?

Yeah, this was predicted.

Can you elaborate on this or point me to where this was discussed/predicted? Thanks!

Other than network disconnect and init messages, rest is coming from libs.

I see, so probably no easy way to disable for now.

[omersiar]

I am having a hard time applying these cleanly, could you send a pull request based on the probonopd/MiniDexed branch networking please?

Sure, I'll update you shortly

Can you elaborate on this or point me to where this was discussed/predicted? Thanks!

https://github.com/probonopd/MiniDexed/pull/747#issuecomment-2466147464

I wonder if ftpdaemon respects FAT file attributes, if then the files not needed to touched by users can be hidden

I checked this and it seems it is possible. Let me try.

[probonopd]

Can you elaborate on this or point me to where this was discussed/predicted? Thanks! https://github.com/probonopd/MiniDexed/pull/747#issuecomment-2466147464

Indeed, the build from https://github.com/probonopd/MiniDexed/pull/747#issuecomment-2466136275 (before adding mDNS and updating Circle to c243194 for it) does not suffer from the stuck notes. So either adding the mDNS feature or updating Circle to c243194 likely causes the stuck notes. Since https://github.com/probonopd/MiniDexed/pull/750 is also affected, I tend to think that updating Circle to c243194 is likely causing the stuck notes.

Maybe @rsta2 has an idea?

[omersiar]

rebased downstream. now you should be able to merge.

[github-actions[bot]]

Build for testing: MiniDexed_2024-11-09-eb96c5e Use at your own risk.

[Banana71]

Build for testing: MiniDexed_2024-11-09-eb96c5e Use at your own risk.

Network not working, the version number of miniDexed is displayed, after a few seconds the display jumps to TG1 and crashes. If I set NetworkEnabled=0 in the minidexed.ini it works again.

Raspberry Pi 4, unfortunately I have no way to connect a monitor via HDMI

[omersiar]

After reverting the changes applied to main branch, resolving ftp compiler warnings and handling wireless disconnections properly, I can say that my setup is now stable.

If I build rebased (main with sustain pr) branch, there are more than one issue which is hard to isolate.