Add support for X1800 (and other?) mixers

[MarByteBeep]

As described here: https://github.com/MarByteBeep/StageLinq/issues/4 When trying to connect to a mixer, the code crashes.

So the code needs to bypass the mixer, either by skipping result.source === "DN-X1800Prime" or by actually implementing support for it. But for that, I need to know what exactly is crashing. @honusz can you help me out with more info on this?

[honusz]

This is a case where the Go-stagelinq code has the same issue:

StageLinq-ts

[15:39:13] [INFO] Found 'DN-X1800Prime' Controller at '192.168.2.248:50010' with following software: { name: 'JM08', version: '1.00' }
[15:39:21] [ERROR] Error: Failed to connect to '192.168.2.248:50010'
    at c:\Users\honusz\code\StageLinq\dist\utils\tcp.js:12:15
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at async Object.connect (c:\Users\honusz\code\StageLinq\dist\utils\tcp.js:11:5)
    at async Controller.connect (c:\Users\honusz\code\StageLinq\dist\Controller.js:73:27)
    at async main (c:\Users\honusz\code\StageLinq\dist\main.js:29:5)
    at async c:\Users\honusz\code\StageLinq\dist\main.js:76:9

Go-StageLinq

2022/01/09 16:33:56 192.168.2.248 "DN-X1800Prime" "JM08" "1.00"
2022/01/09 16:33:56     attempting to connect to this device…
2022/01/09 16:33:58 WARNING: dial tcp 192.168.2.248:50010: connectex: No connection could be made because the target machine actively refused it.

I've attached a capture of SoundSwitch connecting with the X1800. I see a few interesting things:

• The go and ts code both try to connect to port 50010 on the x1800, and are refused.
• It looks (and maybe I'm wrong about this) like the x1800 actually initiates the connection after reading the announcement from Soundswitch (packet 15)?
• Soundswitch uses the port the x1800 reached out from (49156) to reply.
• At packet 21, they agree to move to port 49517 (this doesn't seem like an error or anything, it happened when I tried again at roughly the same point in the sequence). soundswitchcapture-filtered.zip

[MarByteBeep]

Thanks for this! I'll just write down here the stuff that I notice in that capture:

X1800 = .248
SS    = .16

P1-P12 X1800 announces itself @ port 0xc35a = 50010

P13
SoundSwitch announces itself @ port 0xe0f3 = 57587

P15-P16 I assume port 49156 is randomly chosen by X1800 and it tries to initiate connection to port 57587 of the SoundSwitch, as was broadcast in P13.

P17 X1800 sends ServiceRequest message to SS, followed by 00 00 00 00 00 00 00 00 80 00 00 05 95 01 ab 6f I have no idea yet what those numbers are and if they are in any way important.

P18 SoundSwitch sends code 0x0000 0001 to X1800, which I believe is a 64 bit timestamp, not interesting

P19 X1800 -> SoundSwitch, now this one is interesting. Here a timestamp is also sent back to the SoundSwitch, but the 00 00 00 00 00 00 00 00 80 00 00 05 95 01 ab 6f pattern of P17 is also included in that message. No clue as to why.

P20 SS -> X1800. This one is a ServiceRequest message, now from the SoundSwitch back to the X1800. But this is different from what i've seen so far. The Go code and my code just send 0x0000 0002 + a 16 byte token, but this packet shows a whole lot more. I need to try to see what's in there.

P21 Suddenly the communication has moved one port up => 49157. Why? 49157 = 0xC004, but I haven't seen that number in P20. Perhaps it's just always 1 up. Could you verify?

That's it for tonight. I'll look further tomorrow!

[honusz]

X1800 sends ServiceRequest message to SS, followed by 00 00 00 00 00 00 00 00 80 00 00 05 95 01 ab 6f I have no idea yet what those numbers are and if they are in any way important.

Yeah, I've seen some weird ones with leading FF FF FF FF FF FF as well. It looks like, a mask? or flags? Maybe the token isn't really a token. Is that just an assumption IceDream made? Definitely something to look into more.

Could you verify?

Yup, it happens every session, even with other StageLinq software that connects to x1800.

Suddenly the communication has moved one port up => 49157. Why? 49157 = 0xC004, but I haven't seen that number in P20. Perhaps it's just always 1 up.

It looks like P21 is opening a new connection (SYN flag), acknowledged by the client in P22. My guess is, regardless of the port specified in the Discovery Announcement, if the device sends the first message (as in this case), the client knows to respond on that port, and if that port changes, it knows to follow it. We could do the same, we'd need to pass socket.remotePort down to the messageHandler I guess.

[honusz]

Interesting discovery: The X1800 actually passes the volume levels from the mixer to the SC6000s. So if you adjust volume levels on the mixer (either by the channel faders, or the crossfader), it's readable directly from the SC6000's statemap (much like with Prime 2/4/Go I imagine.)

I'm not sure what other information from the mixer is useful for us. Some StageLinq software, if it bothers to connect to the mixer at all, is just sending timestamps back and forth.

[MarByteBeep]

I'm a bit confused: that screenshot looks like NodeJS output, so did the X1800 succeed in connecting to the PC client?

[honusz]

I'm a bit confused: that screenshot looks like NodeJS output, so did the X1800 succeed in connecting to the PC client?

Oh no, sorry, to clarify; That is from Node, but connected the SC6000, however, that value updates when I move the fader on the x1800. So Sc6000 is using its own connection to retrieve that value and hold it in the state map.

[MarByteBeep]

@honusz ok this is interesting. It would mean that we really don't need to connect to the X1800, right? If the fader information is already propagated to the SC6000's as well. In fact, adding support for the mixer, could only complicate things, since I'd have to filter out double fader info.

Or is there information sent from the X1800 that is needed and not sent through the SC6000's? Can you see that in WireShark?

[MarByteBeep]

@honusz and @kkirjala can you test the listen branch again, which is now full of WIP code ;) This is what I changed:

"Initial stab at connecting multiple devices. Cannot test without having multiple devices. One known issue: I cannot seem to reconnect a once lost device properly. I get a requestServices timeout. Need WireShark info for that."

I'd love to know the results on your multi device setups. And if the mixer gives you trouble, simply add the source name to EXCLUDE_DEVICES in common.ts and retry. As I mentioned in my previous post: perhaps we don't even need to connect to that mixer.

@honusz would you be able to make a WireShark profile for me of those devices trying to connect to the app?

[honusz]

I'd love to know the results on your multi device setups. And if the mixer gives you trouble, simply add the source name to EXCLUDE_DEVICES in common.ts and retry.

Everything working great, though I did get an UnhandledPromiseRejectionWarning when I tried disconnecting and reconnecting the mixer. Interestingly though; it wasn't a fatal crash. I stopped Wireshark capture, but forgot to disconnect the debugger. According to the debugger output, it was able to reconnect to the device after 10 minutes. This is actually something I've seen before, while working on my FileTransfer fix; It seems like the SC6000 (and maybe all Denon devices? v2.1.1 at least?) possibly detects misbehaving software, and puts the offender on a timeout.

I'll see if I can reproduce this (and leave Wireshark capturing this time), but here's my logs for now. 0bbed38-capture-filtered.pcapng.zip 0bbed38-debug_log.txt

Just as a note, the pcap is filtered with (ip.addr == 192.168.2.143 && tcp) || ((ip.addr == 192.168.2.144 && tcp)) || (udp.port == 51337 && ip.addr != 192.168.2.248), which is just capturing tcp packets to/from player1(..2.143) && player2 (..2.144), udp messages on the discovery port excluding those from the x1800 (..2.248). I also suppressed console.log messages from statemap for log brevity. If you'd prefer I use less filter, and leave code untouched, just lmk =)

[honusz]

I modified controller.connect() to handle the promise rejection, and now it's able to reconnect to the controller and statemap no problem. It is still assigning new id during the reconnection, but at least it's not being timed out by the controller.

try {
    await this.requestAvailableServices();
} catch (err) {
    console.error(err)
} finally {
    return this.servicePorts
}

0bbed38-02-filtered.pcapng.zip 0bbed38-02-debug_log.txt

[honusz]

Should we consider enabling Discussions for this repo? Just wondering if these issues will get a bit off-topic of numerous. I've done a bunch of digging into the timestamp messages and have some interesting findings, but I'm not sure if I should create a new issue to discuss it.

[MarByteBeep]

@honusz awesome idea, I just added the discussions section. Feel free to start a new discussion there. It helps keeping these issues clean

[kkirjala]

@honusz and @kkirjala can you test the listen branch again, which is now full of WIP code ;) This is what I changed:

"Initial stab at connecting multiple devices. Cannot test without having multiple devices. One known issue: I cannot seem to reconnect a once lost device properly. I get a requestServices timeout. Need WireShark info for that."

I'd love to know the results on your multi device setups. And if the mixer gives you trouble, simply add the source name to EXCLUDE_DEVICES in common.ts and retry. As I mentioned in my previous post: perhaps we don't even need to connect to that mixer.

I can confirm that the current listen branch (commit 0bbed380253bd3cfbf7278f1fd6c50267116ac9c) is able to detect both of my SC5000 players, connect to them and receive load track/BPM change/external mixer volume change events from all layers (4 in total).

[honusz]

Okay, I've made a bit of progress on this.

It seems like SoundSwitch (and other StageLinq software) actually wait for the SC6000, x1800, and probably other, devices to initiate the connection, rather than the way we do it. Perhaps this is because some devices can communicate directly with each other (like two SC*000, or a Prime 4 and an LC6000). The x1800 behaves differently. It won't let you initiate the connection, it needs to be the one to initiate it. That's why the port it includes in it's discovery message doesn't work: It's not listening on that port.

So how does it connect? We need to do a few things:

• We need to create a tcp server listening on a random port.
• That port needs to be added to the discovery message we write.
• We need to have a valid token (more on that later).

I have a proof of concept in the DiscoveryListen branch of my fork

@kkirjala this should work for you if you have your mixer on, but your players off. @MarByteBeep it should work for you with the Prime GO, though you'll get the OfflineAnalyzers trying to connect as well.

The devices will only try to connect if the token is valid. What makes a valid token? I'm not sure. The last 4 bytes don't seem to matter. I added this to common.ts:

//this token works every time. 
//export const CLIENT_TOKEN = new Uint8Array([255, 255, 255, 255, 255, 255, 74, 28, 155, 186, 136, 180, 190, 25, 163, 209]);

//////////////////////////////////////////////////////////////////////////////////////////
// A certain portion of the end of the token seems to be able to be randomized and the Controller will still try to initiate the connection. 
// If too much of the token is random (i.e. not conforming somehow), then the Controller won't try to connect to us.
// Removing elements from the end of the CLIENT_TOKEN_part1 array will fill the remaining portion with random bytes.
const CLIENT_TOKEN_part1 = new Uint8Array([255, 255, 255, 255, 255, 255, 74, 28, 155, 186, 136, 180])
const CLIENT_TOKEN_part2 = Uint8Array.from({ length: (16 - CLIENT_TOKEN_part1.length) }, () => Math.random() * 255);
let mergedToken = new Uint8Array(16);
mergedToken.set(CLIENT_TOKEN_part1);
mergedToken.set(CLIENT_TOKEN_part2, CLIENT_TOKEN_part1.length);
export const CLIENT_TOKEN = mergedToken;

Removing elements from the CLIENT_TOKEN_part1 array will force more elements at the end to randomize. As you can see in the code, the last 4 bytes seems to be fine if random. You can try to randomize more, however, if the controller gets a bad token, it may need a restart before you try again.

Again, I'm not sure how useful this is at the moment. It doesn't seem like theres anything we really need from the x1800. But if we have issues in the future, we may need to implement this type of connection flow.

@MarByteBeep apologies for the brutal, hacky rewrite of your code for this POC, haha. I'm at the limits of my amateur programming abilities here 😂

edit: I'll make a post in Discussions going into tokens a bit more. But it may be a day or two, as my girlfriend is going to kill me if I spend any more time on this now, haha

[kkirjala]

This is the console output I got when running the current https://github.com/honusz/StageLinq/tree/DiscoveryListen code:

CLIENT TOKEN: 255,255,255,255,255,255,74,28,155,186,136,180,165,160,123,43

[23:13:25] [LOG] 0.0.0.0 53932

[23:13:25] [LOG] opened server on 0.0.0.0:53932

[23:13:28] [LOG] message from: 192.168.1.101:49153

[23:13:28] [LOG] message received ServicesRequest

[23:13:28] [LOG] message received ServicesAnnouncement

[23:13:28] [LOG] message received undefined

node:internal/process/promises:246

      triggerUncaughtException(err, true /* fromPromise */);

      ^

AssertionError [ERR_ASSERTION]: Unhandled message id '365824'

at Controller.messageHandler

(/Users/kkirjala/Documents/StageLinq_honusz/dist/Controller.js:141:37)

at Socket.<anonymous>

(/Users/kkirjala/Documents/StageLinq_honusz/dist/Controller.js:102:26)

at Socket.emit (node:events:390:28)

at addChunk (node:internal/streams/readable:315:12)

at readableAddChunk (node:internal/streams/readable:289:9)

at Socket.Readable.push (node:internal/streams/readable:228:10)

at TCP.onStreamRead (node:internal/stream_base_commons:199:23) {

generatedMessage: false,

code: 'ERR_ASSERTION',

actual: undefined,

expected: undefined,

operator: 'fail'

}

Mixer is Denon DJ Prime X1800. NodeJS version 16.13.1 running on MacOS Monterey 12.1

On Sun, Jan 16, 2022 at 10:43 PM honusz @.***> wrote:

Okay, I've made a bit of progress on this.

It seems like SoundSwitch (and other StageLinq software) actually wait for the SC6000, x1800, and probably other, devices to initiate the connection, rather than the way we do it. Perhaps this is because some devices can communicate directly with each other (like two SC000, or a Prime 4 and an LC6000). The x1800 behaves differently. It won't let you initiate the connection, it needs to be the one to initiate it. That's why the port it includes in it's discovery message doesn't work: It's not listening on that port.*

So how does it connect? We need to do a few things:

• We need to create a tcp server listening on a random port.
• That port needs to be added to the discovery message we write.
• We need to have a valid token (more on that later).

I have a proof of concept in the DiscoveryListen branch of my fork https://github.com/honusz/StageLinq/tree/DiscoveryListen

@kkirjala https://github.com/kkirjala this should work for you if you have your mixer on, but your players off. @MarByteBeep https://github.com/MarByteBeep it should work for you with the Prime GO, though you'll get the OfflineAnalyzers trying to connect as well.

The devices will only try to connect if the token is valid. What makes a valid token? I'm not sure. The last 4 bytes don't seem to matter. I added this to common.ts:

//this token works every time.

//export const CLIENT_TOKEN = new Uint8Array([255, 255, 255, 255, 255, 255, 74, 28, 155, 186, 136, 180, 190, 25, 163, 209]);

//////////////////////////////////////////////////////////////////////////////////////////

// A certain portion of the end of the token seems to be able to be randomized and the Controller will still try to initiate the connection.

// If too much of the token is random (i.e. not conforming somehow), then the Controller won't try to connect to us.

// Removing elements from the end of the CLIENT_TOKEN_part1 array will fill the remaining portion with random bytes.

const CLIENT_TOKEN_part1 = new Uint8Array([255, 255, 255, 255, 255, 255, 74, 28, 155, 186, 136, 180])

const CLIENT_TOKEN_part2 = Uint8Array.from({ length: (16 - CLIENT_TOKEN_part1.length) }, () => Math.random() * 255);

let mergedToken = new Uint8Array(16);

mergedToken.set(CLIENT_TOKEN_part1);

mergedToken.set(CLIENT_TOKEN_part2, CLIENT_TOKEN_part1.length);

export const CLIENT_TOKEN = mergedToken;

Removing elements from the CLIENT_TOKEN_part1 array will force more elements at the end to randomize. As you can see in the code, the last 4 bytes seems to be fine if random. You can try to randomize more, however, if the controller gets a bad token, it may need a restart before you try again.

Again, I'm not sure how useful this is at the moment. It doesn't seem like theres anything we really need from the x1800. But if we have issues in the future, we may need to implement this type of connection flow.

@MarByteBeep https://github.com/MarByteBeep apologies for the brutal, hacky rewrite of your code for this POC, haha. I'm at the limits of my amateur programming abilities here 😂

— Reply to this email directly, view it on GitHub https://github.com/MarByteBeep/StageLinq/issues/6#issuecomment-1013948914, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEAFC3TDPUDHTPFY3IKVABDUWMUVLANCNFSM5LSFMCOA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- Kalle Kirjalainen @.***> Tel: +358 (0)40 8232662

[honusz]

Looks like messageHandler in controller.ts received an unhandled messageID (the default case in that switch statement).

Which isn't surprising, the discover listener I added just passes every .on('data') event to @MarByteBeep 's existing messageHandler (with a few tweaks), which is kind of as far as I got.

But the mixer is at least reaching out to initiate the connection based on the discovery announcement the app sent out, which is a success, at least, as far as this P'of'C goes, haha 😂

[MarByteBeep]

@honusz thanks for your suggestion! I'll take a deeper look at it, once I know what it solves. I have a couple of question:

• Do we need to generate a token? Up till now I've simply used the same token over and over again. I took it from my PrimeGo and it seems to work fine in my setup. If we don't need to generate it ourselves, I'd rather stick to using the one that works
• Since @kkirjala told us that the current branch connected to all his devices AND relayed beat/fader info back, do we really need to be able to connect the X1800 for instance? We seem to be getting the fader information directly from the SC5000s as well. Wouldn't it be much, much easier to simply skip the mixers and only focus on the information we're getting from the players?

What are your thoughts on this @kkirjala and @honusz ?

And good luck with your girlfriend, hope she won't kill you, at least not until we solve this ;)

[honusz]

Since @kkirjala told us that the current branch connected to all his devices AND relayed beat/fader info back, do we really need to be able to connect the X1800 for instance? We seem to be getting the fader information directly from the SC5000s as well. Wouldn't it be much, much easier to simply skip the mixers and only focus on the information we're getting from the players?

100% I agree. Sorry if I wasn't clear, I don't mean to propose this as change, it was mostly just as an academic exercise for the purposes of understanding how the protocol works. And to answer the question "how do you connect to a mixer?" This is how. Do we need to? At this point, no 😁

Do we need to generate a token? Up till now I've simply used the same token over and over again. I took it from my PrimeGo and it seems to work fine in my setup. If we don't need to generate it ourselves, I'd rather stick to using the one that works

Again, I totally agree. There's no need to change anything at this point, and we should only do it if it becomes necessary to access desired functionality in the future. Like I did with the timecode messages that I posted a discussion about, I'm just trying to gain a better understanding of how the protocol works. Mostly to save you having to do the detective work at some future time 😂

[MarByteBeep]

Your work is invaluable @honusz , thanks a lot for all this effort you put in this to create those WireShark captures! It's good to know that @kkirjala and yourself seem to be experiencing a much more stable connection now. So I'll try to see if I can find some time soon to cleanup the code a bit and integrate this branch into the trunk, so others can test it as well