Mirror Function not working correctly for different channel types

[glnces]

Doing some more testing on v3.1.1, and the mirror function is a bit sporadic. From my testing, it seems like from the channel types (Input, Software, and Output), it only functions correctly with one of them at a time (i.e it only works for all Software channels or all Input Channels). For example if I refresh the Stream Deck and Mute/Unmute an Input channel, the Software channels will no longer update when changes are made in Totalmix. That and initiating a mute from the Stream Deck will send the Mute command, but won't be updated back with the channel status (i.e. muted as the current status) - so the button reverts back to an 'unmuted' state, meaning I can no longer send a mute command:

For Output channels, I haven't been able to get the mirror to function at all. Let me know if I can help in any way!

[shells-dw]

Thanks for the testing! That's great feedback! Oh well, hello there old friend - I thought I'd fixed that one, darn. I've had that issue myself for a while and thought I fixed it. Doesn't happen on my system anymore despite how long the plugin (or TotalMix) is running and doesn't happen on a friends' machine, but it might on another one I asked to test it too. Nothing better than sporadic issues on remote machines. sigh

You got a UFX+, so you've probably set the faders in bank to 48 in the OSC setup? I wonder if that changes things, I didn't let it run for extended periods of time with 48 faders/bank... 🤔 I'll test that. I take it you opened the second OSC listener in TotalMix, right? Also - did you replace/readd older existing buttons after the update or reuse the ones that were set with a previous version? StreamDeck stores available settings "with the button" so to say and a new version where this button has new settings/values or other functionality will just break the button more or less obviously, depending on what changed (there doesn't seem to be a way to replace them during plugin update programmatically, at least that I know of). I doubt that's the reason, but then again, just to be sure. I think it's a different problem entirely, but sometimes the easiest solution is the best. I'll produce a version with more logging and see if the root of that one can be found... Oh and regards to the question in the old issue thread: based on my experience anything between a day and two weeks, but I haven't asked them to update yet, it's been downloaded quite a bit and I want to make as sure as I can that it's working before I have them push it out - as you can see :)

[glnces]

I had it set to 16 per bank because it seemed to take a good bit longer to read the state of the channel when 48 was selected, but I'll do some testing on 48 for a bit to see if that helps.

The second OSC listener is "In Use" and enabled in "Submix linked to OSC Controller".

I created new buttons when installing the new version. I can try wiping all of them, closing the app, then re-creating too to see if that helps.

[glnces]

Okay, setting the faders per bank to 48 for both 1 and 2 seems to make everything work, but I think due to the amount of channels the plugin needs to check it's very slow. Does that sound about right? Takes around 10 seconds to get the updated state back to the the plugin. Toggling from the Stream Deck takes a while since (from what I can tell) it needs to read the state of Totalmix before it sends the mute/unmute command instead of acting like an independent toggle if mirror is enabled.

This may be difficult to see because the Stream Deck UI doesn't show when you press a button, so I mirrored my physically button presses with a highlighted mouse click. Hopefully that helps visualize what's happening:

[shells-dw]

Thanks for your testing and feedback again! Much appreciated! Hmm. It should not depend on the amount of channels that are set. I assume I made a mistake somewhere and instead of always running synchronously, the device querying runs asynchronously somewhere - so instead of querying all banks in a row and waiting for it to be received and processed, then doing the same for the next bank I assume what happens somewhere is that this thread starts running asynchronously and multiple run at the same time in parallel overwriting the same variables with different data, basically overwriting each other at the same time - hence it appears as if always only one bank is available (so for example mirror only works on output channels but nowhere else). I made that mistake before, hence I think I have a good idea what's causing it, just gotta find where 😉

Yeah, the mirroring can become slow(er) on 48 channels. On 48 channels the device sends a dump of 576 settings per bank, so somewhere around 1728 values total that have to be gathered and sent by the device over a networking protocol (which has overhead as well, even though it's not actually using a network wire the protocol overhead still remains), received and processed, which takes a bit of time per step - that adds up. I didn't really put huge amounts of testing in having so many channels yet and it would've been impossible to do with the old lib (which took way more time to process than the new lib/method), so having so many channels and how the plugin reacts to them is a rather new thing for me too. Interesting find with the toggle not working until the mirror update. It makes sense why it acts that way (it's coded to do it like that 😅), but that's probably not optimal, but wasn't really a problem with 16 channels... will think about how to improve that. I'm with you, you should be able to toggle even without an "updated" state. I have a couple of ideas of how to improve all that as well and make it quicker, but didn't have time to look into it yet. Especially not since there are more pressing issues with the basic logic not working reliably. This is a spare-time hobby thing competing with other things and I need to earn money too don't forget 😉 I'm on it though. Also again I appreciate your feedback, it's really helpful to have the plugin run and tested on different machines! Things rarely just work as on the dev machine... I'll update you with a new version today hopefully :-)

[glnces]

Hey yeah no rush, take your time! I really appreciate the work you put into this plugin even though you don't even use an RME interface anymore haha! Just trying to help where I can.

[shells-dw]

Check out v3.1.2 I think I figured out what my head was wrapped around when I wrote that. I'm writing the plugin parallel for the Loupedeck thing, so my brain sometimes mixes things up regarding what I've done there and here and how things work differently on both platforms. Doesn't explain why it would coincidentally work for me during testing for hours, because sure enough when I booted it up today, it showed exactly the same issue you described... 🤷‍♂️

It should now update as it's supposed to... Well, we'll see 😅 I think the main reason why it took so long to update on 48 channels for you was that it ran through dozens of iterations before it found the correct value at the right spot and was able to show it. At least that's what happened for me today. Now with everything (hopefully) working as it should, on my machine (highly OCed CPU though) it takes roughly 1-2s to update on 48 channels, so I would assume with the background task working as it should, that should improve for you as well to similar ranges. Anyhow for performance I'd suggest to set that to how many channels you actually need. If you only use 16 regularly or run all in Stereo anyway it's probably faster to run it with only 24 as everything above isn't useable anyway in that case and just causes overhead. I've improved it a bit as well by just storing values I'm actually interested in. Shouldn't make that huge of a difference, as most of them are needed, but heyho.

Now... when it comes to the buttons being able to switch actions "during" updates with mirroring enabled. I've looked into it again and that's not something that's easily fixed. There is just one variable for the button that holds its state (1 or 0), so when the button is pressed, it obviously gets what the current state is and sends the opposite value (if state is 0 it means it's off, so sending 1 to the device will enable it). So far so easy. When the button is updated by StreamDeck with mirroring enabled (which constantly happens in the background), the value is read from the variable that holds the latest response from the interface and replaces what's currently held for the button state (to be able to reflect changes made in TotalMix directly). This has to be done so the button knows on next press if it should enable or disable the function when you change a value on the device (edit: and also obviously to show the correct on/off image). So I can't think of a way of knowing that what the button has to do is different from what the last device response says, if that makes sense. I can't tell if the current device response is correct or not. At least nothing that is not utterly complicated (could add a varible with expected values for each button based on timeticks and settings or something thereabouts) - but that's probably not really required by anyone using the plugin and seems somewhat working around other issues 😆 With the update speeds being better, it should be less of a problem. Also you probably wouldn't mute/unmute (for example) a channel every half a second for the lulz in real work, you'd probably mute it and if the StreamDeck needs 2 seconds to reflect that change and display the "on"-image that should be fair enough.

[glnces]

Works perfectly! Tried with both 16 and 48 (I don't really need 48 haha) and it's updating very quickly now! 2 seconds is completely reasonable and perfect for an everyday workflow outside of just testing. Great work!!

Don't hate me but I found a little issue for OSC Global Functions when a Snapshot is selected. Looks like it doesn't pull a custom snapshot name when "Display Channel Name" is checked, and if unchecked it still displays the snapshot number instead of clearing the field below the image. For the "Trigger Global Function" when tapping the button for a split second the text will also show "Global Mute" regardless of which function is selected. Let me know if you want me to open another issue or if this isn't worth working on since it doesn't impact functionality.

[shells-dw]

Works perfectly! Tried with both 16 and 48 (I don't really need 48 haha) and it's updating very quickly now! 2 seconds is completely reasonable and perfect for an everyday workflow outside of just testing. Great work!!

Nice! Try running that version for a while and see if it works reliable over a couple of days too. 😅 Don't hate me but I found a little issue for OSC Global Functions when a Snapshot is selected. Looks like it doesn't pull a custom snapshot name when "Display Channel Name" is checked, and if unchecked it still displays the snapshot number instead of clearing the field below the image.

Haha, I appreciate you buddy. Thanks for taking the time to test stuff out and report back. It's probably not really a selfless action 😛 😉 , but still it's taking your time, so thanks for that. Anyhow. That was absolutely an oversight, I've noticed that too the other day when I was looking at something else and made a mental note to come back to that, which I -of course- immediately forgot again 🤣 For the "Trigger Global Function" when tapping the button for a split second the text will also show "Global Mute" regardless of which function is selected. Let me know if you want me to open another issue or if this isn't worth working on since it doesn't impact functionality.

Interesting... Didn't yet notice that one. Nah, that's fine mentioning it here for now. It's probably just a bad chain of actions leading to it passing over the part that sets the "Global Mute" text and image every time before hitting what it's supposed to overwriting it again. I'll look into it :)

[glnces]

Okay. After having it run for a few hours, it does end up losing connection to Totalmix. A restart of the Stream Deck software is needed to resume normal functionality.

[shells-dw]

Mh, there's something going on occassionally, I've seen it too today. Not sure what's happening, seems like the socket is GCed before it's closed, which shouldn't happen - got to figure that out, so far it's quite elusive and takes a while to happen 😩

Oh, quick feedback regarding the snapshot names... the names of snapshots are not transmitted via OSC by TotalMix, I can't get them. Nonetheless the option to disable the automatic titles on Global Functions will work in the next version (didn't want to produce a new release just for that though)

[shells-dw]

Alright... v3.1.4 I've fixed the Global function Display Names checkbox being ignored and the Global Mute flicker thing. Also I've quickly added a function to read out snapshot names from the local TotalMix config file as they are not transmitted via OSC. Haven't tested that a lot (cough at all), so ... 🤞 it'll work 😉

Regarding it losing its connectivity (send or mirror capability): I don't know. I've had it running constantly over the weekend when my PC was running and it never lost the connection or didn't execute button presses once. Not saying it doesn't for you, but I couldn't reproduce it on my box with my UC, maybe the UFX II acts slightly different or it's some other thing going on ... somewhere on your PC that just doesn't happen on mine for whatever reason. I've added some amount of logging for now, so if it happens again, drop me the logfile %appdata%\Elgato\StreamDeck\Plugins\de.shells.totalmix.sdPlugin\pluginlog.log - maybe that enlightens me what might be going on there. While looking closely I found that there is very occassionally an exception thrown by the listener thread on socket level and that one I have a very hard time figuring out. It seems to happen on System Socket level which I can't really debug. It might actually be .NET causing it in that case (I found some threads about this actual problem in a somewhat similar case and there is went with upgrading to a newer .NET version, which isn't really going to happen due to dependencies here), or maybe it's a bug in the lib I use. I don't know. It can happen after 2 minutes, an hour or 8 runtime, and I don't have time to figure that out or try a different library right now. Work picking up until Christmas, then Christmas and all... so I think it might stay that way for the time being. However: It shouldn't be a problem, the plugin exits (well, crashes technically), StreamDeck detects that it exited and immediately restarts it which doesn't seem to cause anything odd. Check it out, see if it works better here

[glnces]

Awesome! The snapshot names already work great. I'll run this for a while and attach the logs if I see anything weird. Have a great Christmas and New Year!

[glnces]

Here's the log after running for a few hours and losing connection. Seems like it attempts to relaunch but runs into this error Unhandled Exception: System.ObjectDisposedException: Safe handle has been closed:

pluginlog.log

[shells-dw]

Huh. That safe handle thing (when an already disposed UDP Socket is tried to be accessed for unknown reasons) seems to happen every other minute on your system. Interesting. I've had that happen 5 times over a day, you had that happen 6 times during 15 minutes, roughly estimated. Also for me it happened but StreamDeck always restarted the plugin without... just not restarting it. Hmm. I'll think about it and look into it. I'm curious now... 😅

[shells-dw]

Hey Matt, just to give a little update on how things are going 😃

I've not really eliminated the bug, but got it to a point where it happens every 1-5 hours on my system. That's surely not perfect, especially for debugging it (I've had more than once a YES-I-FIXED-IT cheer, when 2 hours later that thing crashed again with the same darn exception) and right now I'm thinking about just not using the Library and code that stuff myself as I can't debug where it happens, when or how. It just happens for no apparent reason (to me, right now, anyway 😅). Year's end is busy with actual work and social stuff, you know how it is before the holidays, eh? Sigh. I work on it, whenever I have a new idea or it bugs me, that I can't figure it out (and have time...). So nothing Christmas gift-y fully working plugin yet, but we'll see if there are miracles happening this week haha 😉 I might just undo all my trial-and-error logging stuff for now and push the version that at least only crashes after a couple of hours until I finally figure that out, if it does the same for you it might be less of a problem than it's probably now.

Also - thank you very much for the Sponsor/coffee! I didn't yet say thanks, the notification got buried in my inbox and I've just noticed it today. So thanks buddy, I really appreciate it ❤️

[shells-dw]

Just pushed a release - v.3.2.0 This should hopefully fix the plugin crashing and becoming unresponsive (and fixed some other things too). Had a client cancel for today so spare time the whole day to look into it again. I've more or less just caught the exception that is still happening, but that shouldn't cause issues. In the end it was just a callback waiting for a socket that was already closed. Which it doesn't like (rightly so) but isn't a major deal for the process flow in this case. Why that is the case - don't know yet. But for the time being that should work. I let it run for 5 hours without anymore issues on my machine, so I think it's good to go out to you too.

Check it out and let me know if anything's odd. Cheers Have a great Christmas :)

[glnces]

Hey! Been testing on and off over the past week and it's been solid!! The only thing I've found is that if the computer goes to sleep and wakes up, in order to re-establish connection to Total Mix I need to toggle "Enable OSC Control" in TotalMix off and back on again. Upon waking from sleep it seems like there's a broken link and it uses a good amount of CPU because of it:

[shells-dw]

Hey 😃 Hmm. Permanently 8% CPU usage sounds like something's in an infinite loop causing 1 core to be to utilized 100%. Not sure what that would be, as there is waiting time in between the task that does the mirroring and other loops inside that have checks for lost connectivity, so it should never be able to run permanently... Not saying it doesn't, I have probably overseen a case where this could happen, not sure where at this time though 🤔 Anything enlighting in the log file?

Does a restart of StreamDeck help as well? If not and only toggleing OSC in TotalMix fixes it, it's a TotalMix problem. I've had very occassionally times where TotalMix kind of hung itself and kept just sending 2 values infinitely (even with no plugin running and I was just testing out OSC values with other software to figure out what it does for some commands), no matter what I asked it to do. I had that maybe 3 times over the time I've been developing the plugin, so I figure it's rare - but maybe something like that can be caused by putting the PC to sleep. That at least would explain the infinite loop when the reading task is connected and permanently receiving, but only receiving jibberish - or at least not what it expects - and the device never stops sending that stuff until you basically reset TotalMix by toggleing the OSC server... that would be an explanation, if a shot in the dark - a far fetched at that as well.

I don't really use sleep on my machine because it doesn't like it. I have a custom watercooling system with a highly (EFI-) OCed CPU and sleeping makes it drop its fan/pump settings and cause issues with RAM timing for whatever reason - among other things. Also my internal Realtek audio chip (which I use mainly for "normal" stuff to safe power for the Fireface when I don't need it) doesn't work after sleeping. I only use hibernate which makes the PC kind of shut down (but it's booting way quicker and restoring everything that ran) - and that's not a problem, at least not on my machine. Other than that I only shut down the display. And hibernating with everything running works fine, I do that basically every day and it restores itself every time.

I've reenabled sleeping just to test it out, but it came back for me just fine. Put it to sleep 5 times and every time everything continued working (well, except for my water cooling system).

So.. try if a StreamDeck restart helps as well, or if it's only TotalMix that can fix it - and if a StreamDeck restart helps, check out the plugin log, if there's anything useful in there - and if so, please attach. Cheers

[shells-dw]

Quick update... I think I can reproduce the issue. When TotalMix crashes or gets shut down while the plugin is running, it doesn't check anymore, if the socket is open and ends up in a loop without pausing (hence one CPU is fully utilized). However, as soon as TotalMix is started or responds again, the plugin catches that and returns to nominal state (as intended) and the high CPU load stops. So while it's probably not optimal for the plugin to end up in that state if TotalMix crashes (and I'll think about how to avoid that for future updates), it's not too much of an issue generally, I think. You'll notice when TotalMix goes down, then nothing works anymore anyways. If you just shut down your auidio interface, TotalMix will still serve the plugin. This issue will thus only occur if TotalMix itself crashes, is shut down manually or stops sending OSC packets. So... I think something's up with TotalMix after sleeping (either generally or on your machine). If my theory is right, restarting the StreamDeck should result in the plugin complaining about TotalMix not being available.

[glnces]

Yeah only toggling OSC from Totalmix reestablishes the connection. Weird that it's just me! I could try removing all the buttons and re-adding them. I'll keep testing!

pluginlog.log

[shells-dw]

Plugin version: 3.1.4

Try 3.2.3

[glnces]

Still have to toggle OSC with the new version after sleep yeah. I'll keep testing though!

[shells-dw]

Strange indeed. Has been running fine for me so far (but I'm not really having my PC sleep at all). I still kind of suspect TotalMix being the culprit though.

I'll close this issue-thread here now. I don't think, there's much I can do at this point. In case you find something that might help, feel free to re-open this issue or put in a new one.

Thanks for all your beta-testing and support anyway ❤️

Cheers

[glnces]

Do you by chance have a build of the 2.0 version where you can disable mirroring? I'm consistently forgetting to toggle OSC in Total Mix after sleep which is causing my cpu to pull too much power.

[shells-dw]

You can disable mirroring on this version as well. Set miorroringRequested in the config to false.

And yes, I made a "Classic" version of the v2 branch. Wouldn't recommend it, but heyho.

To tackle your issue finally... does shutting down and restarting TotalMix fix it? I could then implement a config flag that would allow killing and restarting TotalMix if it should cease to respond. That would probably not be the most beautful solution, but I looked in their config file that could in theory be used as well, but for once it's read only when TotalMix starts (which would need this killing and restarting action anyway) and also their XML is as true to XML specs as their OSC is to OSC specs, so this would be a beeping mess to try and programmatically fiddle with that...

Edit: Oh and I forgot to mention, I released v.3.3.2 which tackles a little issue with the wrong image being shown when mirroring is deliberately disabled.

[shells-dw]

I'm consistently forgetting to toggle OSC in Total Mix after sleep which is causing my cpu to pull too much power.

Thanks for the hint with the high CPU load. Wasn't aware this would happen.

I just released a version (v3.3.3) that will now recognize that the mirroring isn't available anymore and stop the listening task (this might take 10s or so, I'm quitting gracefully and don't just kill it) so it doesn't run in this infinite loop situation - once the OSC server is back up, it'll resume the operation.

Cheers

[glnces]

Sweet! Thank you for that.

Yes, exiting Totalmix and re-opening does reestablish the OSC connection with the plugin.

[shells-dw]

v3.3.4 has a config flag to enable killing and restarting TotalMix which should put it back in a working state after sleep for you - still can't reproduce it, so that's a should here. Set killAndRestartOnStuck to true to enable it. Please read my release notes on that though. TotalMix doesn't seem to store settings (or anything for that matter) unless it's gracefully closed down (even settings dialog changes that close by OK buttons and stuff 🤷‍♂️), so... use with that in mind.