Pioneer DJ controllers and Rekordbox

[SanderVocke]

Creating this issue because I have had two people come forward with unsuccessful attempts to connect DJ controllers to Rekordbox on Wine in Linux using this wrapper. Let's discuss here.

[SanderVocke]

@elvetemedve reported using the following config for a DDJ-FLX10 after using the wrapper to determine the correct values on Windows:

{
    "log": "midi_rename.log",
    "popup": true,
    "rules": [
        {
          "match_name": "DDJ-FLX10 - DDJ-FLX10 MIDI.+",
          "match_direction": "out",
          "replace_name": "DDJ-FLX10",
          "replace_man_id": 65535,
          "replace_prod_id": 65535,
          "replace_channel_mask": 65535,
          "replace_driver_version": 274,
          "replace_technology": 1
        },
        {
          "match_name": "DDJ-FLX10 - DDJ-FLX10 MIDI.+",
          "match_direction": "in",
          "replace_name": "DDJ-FLX10",
          "replace_man_id": 65535,
          "replace_prod_id": 65535,
          "replace_driver_version": 274
        }
    ]
}

With the following result:

Request for output device capabilities:   name: DDJ-FLX10 - DDJ-FLX10 MIDI 1
  man id: 255
  prod id: 1
  driver version: 1
  technology: 1
  voices: 0
  notes: 0
  channel mask: 65535
  support: 0

--> Matched a replace rule. Returning:   name: DDJ-FLX10
  man id: 65535
  prod id: 65535
  driver version: 274
  technology: 1
  voices: 0
  notes: 0
  channel mask: 65535
  support: 0

Request for input device capabilities:   name: DDJ-FLX10 - DDJ-FLX10 MIDI 1
  man id: 255
  prod id: 1
  driver version: 1

--> Matched a replace rule. Returning:   name: DDJ-FLX10
  man id: 65535
  prod id: 65535
  driver version: 274

... But not recognized in Rekordbox.

I got a very similar experience in a private message, in that case it was a DDJ-RB.

[SanderVocke]

To me, there are a couple of options of what could be going wrong:

1. Maybe there is still something wrong in the values configured in the configuration. It seems suspicious that on Windows, we always get man id and prod id of 65535 - it could be a bug. A next step could be to go on Windows and find a tool which can also inspect these properties independently. If such a tool also would report 65535 on Windows, we know we are doing the correct thing here and we can rule out this potential issue.
2. Maybe there is still some other property missing in the wrapper. It seems unlikely to me - the properties supported, as far as I know, completely cover all device properties that the WinMM midi API has. You could verify this independently: https://learn.microsoft.com/en-us/windows/win32/api/mmeapi/
3. Maybe the Rekordbox software is talking to the controller using more than just MIDI. After all, it is a USB device. If it turns out that Rekordbox verifies the device with some non-MIDI communication, this wrapper tool is not going to help. It will be difficult to get around in that case, because AFAIK, WINE does not support USB drivers in the Windows side.

One thing I would suggest is to install MidiView (https://hautetechnique.com/midi/midiview/) both in Windows, and in WINE on Linux. Use it to monitor the MIDI communication when the device is connected. Check if the kinds of messages being sent and received are the same in both environments.

In my own use-case (a Joue Play MIDI controller and app), the wrapper worked fine, but of course the application could be doing less checks.

I myself am happy to support by e.g. fixing bugs if it turns out that, for example, the man id / prod id thing is somehow broken. But I am not willing to actively help analyze communications or something of that nature - I would rely on input from you.

Good luck, I hope it ends up working!

[SanderVocke]

There is one level deeper you could go in analyzing the differences in Windows and WINE: use something like API monitor (http://www.rohitab.com/apimonitor). You could track exactly what Windows Multimedia API calls the Rekordbox application is making, including the exact values returned by Windows or WINE to the application.

Maybe this would reveal some difference in behavior between WINE and Windows that this wrapper could solve.

[SanderVocke]

One last suggestion: I read that the DDJ-FLX10 also supports Serato DJ Pro. Maybe install a free trial of that in WINE and see if it picks up the device? Just as a sanity-check, you never know.

[SanderVocke]

... And another addition. I found a blog post of someone in 2019 achieving the connection with DDJ-SX2. I don't know which WINE version and which Rekordbox version was used. https://www.erhan.es/blog/using-pioneer-controller-with-rekordbox-in-linux/

He essentially does the same thing as this wrapper library does, but instead of using a wrapper DLL he modified WINE itself to do the same name re-mapping. He was able to get it working by only remapping the name, not the IDs.

Please have a detailed look at what he did, and maybe you can replicate. Who knows, maybe you should only remap the name and not the IDs?

[SanderVocke]

In any case, I would appreciate if you let me know here if/when your attempts are successful. Maybe someone else will benefit.

[elvetemedve]

@SanderVocke Thank you for sharing your view and the suggestions.

I explored some of these options. The API Monitor was the one that provided the most details:

• It confirmed that the values for man_id, prod_id are indeed set to 65535 on Windows.
• It also confirmed that this wrapper renamed the device as expected (DDJ-FLX10).
• It pointed out that enumerating the midi devices work well, but as soon as a message is sent to the device, it fails. This is indicated by the midiInMessage() call returning MMSYSERR_INVALIDHANDLE, while midiOutMessage() returning MMSYSERR_NOTSUPPORTED value. There is only one midi device listed on Windows, therefore one call is made to midi*Message() where the first argument (device handle) is NULL (I guess NULL means default device) and the return value is MMSYSERR_NOERROR. I also checked that the function return values are the same even if I remove the wrapper (winmm.dll). I also noticed that Rekordbox sends the same midi message to all devices and these virtual devices like "PipeWire-RT-Event input" also return an error. To me it looks like the problem is not hardware specific, but a general issue with Alsamidi (maybe only local setup related).
• I installed the open source DJ software Mixxx to see if a native application can communicate with the DJ controller. I was able to map few buttons, faders which is an indication that the MIDI communication is possible under Linux.

The DJ controller is the device ID 3 and the name is "DDJ-FLX10". All midiInMessage() call failed.

This shows that both midiInMessage() and midiOutMessage() failed, but with different return value.

Device ID 1 is the virtual device "PipeWire-RT-Event input".

[SanderVocke]

That's interesting info, and also taught me something I didn't know yet about the MIDI API on Windows.

Looking in the docs of midi*Message() (see e.g. https://learn.microsoft.com/en-us/windows/win32/api/mmeapi/nf-mmeapi-midiinmessage), apparently these functions are used not to send MIDI messages, but to send messages (queries) to the driver.

the message is the second argument (2061 in your screenshots). Digging a bit further, the following is the list of queries mentioned in the MSDN docs above:

#define DRV_QUERYDEVNODE (DRV_RESERVED + 2)
#define DRV_QUERYMAPPABLE (DRV_RESERVED + 5)
#define DRV_QUERYMODULE (DRV_RESERVED + 9)
#define DRV_PNPINSTALL (DRV_RESERVED + 11)
#define DRV_QUERYDEVICEINTERFACE (DRV_RESERVED + 12)
#define DRV_QUERYDEVICEINTERFACESIZE (DRV_RESERVED + 13)
#define DRV_QUERYSTRINGID (DRV_RESERVED + 14)
#define DRV_QUERYSTRINGIDSIZE (DRV_RESERVED + 15)
#define DRV_QUERYIDFROMSTRINGID (DRV_RESERVED + 16)
#define DRV_QUERYFUNCTIONINSTANCEID (DRV_RESERVED + 17)
#define DRV_QUERYFUNCTIONINSTANCEIDSIZE (DRV_RESERVED + 18)

where DRV_RESERVED is 0x800 (so 2048).

That would suggest that Rekordbox is querying for DRV_QUERYDEVICEINTERFACESIZE, which is usually the first step before querying DRV_QUERYDEVICEINTERFACE to get a driver-level device name.

In other words, looks like Rekordbox is using this advanced method to query the device name, instead of just relying on the device properties we have spoofed. If we are lucky, this is just a final check before really starting to talk to the device. And these calls could be spoofed by this wrapper just like we are spoofing the other calls.

I would suggest:

• Can you have a close look on Windows with the API monitor, and see what calls to midi*Message are made exactly? What I am referring to is all the different second arguments used to these functions (2061 in your screenshots). Then we know exactly what queries Rekordbox makes.
• Unfortunately I don't think you can see the values returned from these calls in the API monitor (they are passed by pointers). When I find some time this week, I will add some more logging to the wrapper DLL so it also logs these queries and what they return.
• Combining these two steps, I could then add extra features to be able to spoof these queries.

No guarantees that this will work, but at least it is not a dead end.

[SanderVocke]

One more thing, does the API monitor tool support exporting/importing a logged session? If so, could you record the behavior on Windows and share it?

[elvetemedve]

Looking in the docs of midi*Message() (see e.g. https://learn.microsoft.com/en-us/windows/win32/api/mmeapi/nf-mmeapi-midiinmessage), apparently these functions are used not to send MIDI messages, but to send messages (queries) to the driver.

You are right. I didn't pay attention to this detail. Thanks for noticing. It looks like before talking to the device it has to call midi*Open() first.

---

Can you have a close look on Windows with the API monitor, and see what calls to midi*Message are made exactly?

As far as I see midi*Message() functions are called in pairs, first with 2061 as message, secondly 2060. Based on your observation, these must be the DRV_QUERYDEVICEINTERFACESIZE and DRV_QUERYDEVICEINTERFACE messages. That must be the way Rekordbox obtains the device name.

---

No guarantees that this will work, but at least it is not a dead end.

Yeah, there is hope fortunately. Though I noticed that these function pairs are called multiple times until the midi*Open() call takes place. So there could be a loop, or callback that is waiting until a condition is met (the open question is what are these conditions and will these be fulfilled under Wine).

---

Combining these two steps, I could then add extra features to be able to spoof these queries.

If you can put the effort into that, it would be very nice.

---

One more thing, does the API monitor tool support exporting/importing a logged session? If so, could you record the behavior on Windows and share it?

Sure, I've saved the capture of function calls. You should be able to open and inspect it after extracting the zip file: midi-api-capture.zip

[SanderVocke]

Nice. I will get back to you some time later this week with an update to log these calls.

[SanderVocke]

In the meantime, you can already use this tool I found (midienum.exe) to check and compare the device interface name on WINE vs. Windows:

https://matthewvaneerde.wordpress.com/2012/09/21/enumerating-midi-devices/

[SanderVocke]

You can have a try with this modified wrapper:

release-dll.zip

It should now also log the queries to DRV_QUERYDEVICEINTERFACE(...) in the log file.

You can add an entry to your replace rules: "replace_interface_name", to match the Windows name in WINE.

I didn't test this myself, sorry, so it may not work correctly. Otherwise, it would be good to also have a look with the midi enum tool above.

[elvetemedve]

Thank you for the updated version.

I've checked out the interface name by both methods on Windows 10 and got the same text:

\\?\usb#vid_2b73&pid_0041&mi_03#9&2f46a9a1&0&0003#{6994ad04-93ef-11d0-a3cc-00a0c9223196}\global

Based on that I've updated the config file: midi_rename_config.json

Unfortunately it does not load the rules properly ("# of rules loaded 0").

Could you have a look what might be wrong there, please?

[SanderVocke]

Sure, I'll have a look. I really should have tested it myself first. I can't get around to it until Monday though.

I am curious, what is the interface name when running with wine?

[elvetemedve]

Possibly I made a mistake somewhere, because now I got different info in the popup for the same config JSON:

---

Sure, I'll have a look. I really should have tested it myself first. I can't get around to it until Monday though.

I am curious, what is the interface name when running with wine?

The interface name is just a number under Wine. For example --> Transparently queried the device interface with result: Queried device interface size for input. Native result: 1448556300 midi_rename.log

[SanderVocke]

It could be that the interface isn't always exactly the same. But it wouldn't be hard to imagine that Rekordbox does some basic check on it so that it looks "Windows-like" and not just a number like WINE.

Hang in there - I don't have a lot of time this week but I will get around to fixing the config bug at some point.

[elvetemedve]

It could be that the interface isn't always exactly the same. But it wouldn't be hard to imagine that Rekordbox does some basic check on it so that it looks "Windows-like" and not just a number like WINE.

Actually Rekordbox needs to recognize the controller for two purposes:

1. To load the device specific midi mapping. Maybe the device name as text is good enough for this.
2. Certain paid features are unlocked when a Pioneer controller is connected. I can imagine they do some trickery because of that to prevent getting those features for free by simply faking the name.

Hang in there - I don't have a lot of time this week but I will get around to fixing the config bug at some point.

Sure, thanks for the update.

[SanderVocke]

Actually Rekordbox needs to recognize the controller for two purposes:

I agree, I just meant that maybe it doesn't use only this interface name for device recognition. It could be doing some checks on the interface name and then combining that with the Midi device name.

It is a bit unclear to me, but I think that the interface name contains information both about the device and about the Midi hardware and/or driver used to connect to it.

[SanderVocke]

release-dll(2).zip Thanks for waiting, I got around to it. Bug with loading rules should be fixed (along with strange characters that were printed in the message box), although I am still not able right now to test this with a real MIDI device.

Can you try again with this DLL and the same config file?

Also one remark about --> Transparently queried the device interface with result: Queried device interface size for input. Native result: 1448556300: notice that it says "interface size", not "interface name". It first asked how long the name is. So it is supposed to be a number - although 1448556300 is a strange answer.

Anyway, please have a try and next time if something is wrong I will be able to respond faster.

[elvetemedve]

No worries. Thanks for your effort to keep investigating this.

Can you try again with this DLL and the same config file?

I tested it and the rules are loaded successfully this time. However faking the Windows API call still does not look right to me. In the log I see empty string as device name (--> Transparently queried the device interface with result:), while API Monitor still captured midiInMessage(*, 2061, *) calls returning MMYSYERR_INVALIDHANDLE, and of course Rekordbox does not recognize the MIDI controller.

notice that it says "interface size", not "interface name".

My bad, sorry about that. I add the full log here.

midi_rename.log

Do you have any other idea what went wrong?

[SanderVocke]

Sorry for my delayed response. I guess responding quickly didn't really happen.

Do you also have logs from the API monitor (both on Windows and on WINE)?

[elvetemedve]

No worries. 🙂

Sure, I will do, but can't until the middle of next week.

[elvetemedve]

Here are the logs captured under both Windows and Linux: logs.zip

[SanderVocke]

Thanks! This is very informative. The API monitor gives a lot of info, although not all - unfortunately, arguments passed as pointers to the system layer have invisible values to the API monitor.

But it seems you are right about the MMYSYERR_INVALIDHANDLE. For some reason, making these driver-level calls to the Wine implementation returns errors and empty device names. I am not sure if that is some kind of bug - it could simply be that these features are not supported in Wine. I had a quick look at the Wine source code but couldn't track it down.

We don't necessarily need to know the exact problem though. We are trying to spoof the device name anyway. I think we can just return a success code to Rekordbox and the spoofed name, even if the actual system gives error codes. To Rekordbox it would look like a successful call.

I'll go ahead and implement that and you can try it out.

[SanderVocke]

OK, I added a new release on the front page (1.4.1). This one will return MMSYSERR_NOERROR to Rekordbox, even if Wine gives an MSMSYSERR_INVALHANDLE.

I would expect that this will at least trigger Rekordbox to then query the device name next. Let's see whether it accepts the result and moves on to actual MIDI communication.

Can you give it a try and report back with the new API monitor log?

[elvetemedve]

Thanks.

It made me thinking that the Midi implementation in Wine must be incomplete. The related wiki page is from 2016, which tells me that it's not actively developed.

I've tried out the new release. Unfortunately I don't see the name in the log or MMSYSERR_NOERROR in API Monitor. It looks like the rule does not match in handle_QUERYDEVICEINTERFACESIZE(), because the text starting with "_Matched a replace rule. Returning MMSYSERRNOERROR with size" is not captured in the log.

See logs.zip