Sending DMX data to USB only if changed?

[paradajz]

Disclaimer: this is not an issue or bug, but a question.

I'm developing a dual-mode device which registers on PC as both MIDI and CDC interface (virtual COM port). I'm using CDC interface for OLA via usbserial plugin. I have QLC+ connected to OLA, OLA sends data to CDC (aka my device) and then my device sends DMX signals to actual fixtures. On my device, I have internal DMX buffer and I am taking care of all the timings, meaning I basically need only raw data. My device implements Enttec DMX USB Pro Widget API so that the communication with OLA is possible. All fine and dandy - at least on devices with STM32 MCUs.

This entire setup also works with some AVR (8-bit, 16MHz) MCUs that I'm using, however, there is a considerable lag between changing values in lightning software and the actual change in lightning on fixtures. The reason for this is quite obvious: OLA sends values for all DMX channels constantly to device. Now, this is not fault of OLA, it's just the way Widget API works, I guess. The result of this that my AVR-based device needs to poll USB constantly for data that isn't changing all that often, so most of that processing time is spent for nothing. Normally, due to the high amount of time spent in polling and processing data from USB, the output will lag as well.

With all this, my question is: does OLA support "diff mode" protocol for DMX via USB, where only the changed data would get sent to device? Does any of the supported devices work like this? I've checked but haven't found exactly what I'm looking for, but then again, I might've missed something. The idea is basically this:

• OLA receives DMX data from some source (in my case, QLC+)
• OLA keeps internal buffer for all DMX channels as the source-of-truth
• Upon receiving new DMX data, OLA compares data for DMX channels with the values in source-of-truth buffer
• If the data differs, internal buffer is updated, and OLA sends a packet something like this:
  • <start> <channel> <new_value> <end>

Would something like this be feasible?

EDIT: Bit more info. My device, as well as many other devices, has internal DMX buffer which gets sent out constantly out of UART, which is why timing from PC side isn't important - just the raw data, since the timing is handled on MCU side. Having the PC sending DMX data for all channels constantly is basically duplicated work since the MCU is already doing that. MCU just needs information which channels have changed the value to update its internal buffer which, like I mentioned, is already sent out constantly. I've noticed that OLA via the web interface already does something similar to what I want (when QLC+ isn't using OLA): data is sent out every 1 second or so, and then on any change in values on sliders (channels) in the web interface.

[peternewman]

Disclaimer: this is not an issue or bug, but a question.

Thanks for the up front note.

This entire setup also works with some AVR (8-bit, 16MHz) MCUs that I'm using, The result of this that my AVR-based device needs to poll USB constantly for data that isn't changing all that often, so most of that processing time is spent for nothing. Normally, due to the high amount of time spent in polling and processing data from USB, the output will lag as well.

I'm a bit surprised by that, I didn't think our Enttec implementation on an Arduino had those sort of issues but I may be wrong. https://github.com/OpenLightingProject/rgbmixer

With all this, my question is: does OLA support "diff mode" protocol for DMX via USB, where only the changed data would get sent to device? Does any of the supported devices work like this?

Funny you should ask, see #1737 @clxjaguar knows the protocol far better than me. The Milford Instruments stuff also did some odd modes like that, but I don't think we implemented them.

Would something like this be feasible?

If you don't need RDM, then yes it should be. Sending channel=value isn't necessarily very efficient if you've got a lot of changes at once (e.g. a blackout)

EDIT: Bit more info. My device, as well as many other devices, has internal DMX buffer which gets sent out constantly out of UART, which is why timing from PC side isn't important - just the raw data, since the timing is handled on MCU side.

Sending full frames when they change (and at intervals) might be the safe approach, otherwise you need to tell it when you've finished a frame so it doesn't transmit a partial frame (and it probably needs another DMX buffer internally).

See also some of the proposals in #280.

[paradajz]

Thanks for the response!

This entire setup also works with some AVR (8-bit, 16MHz) MCUs that I'm using, The result of this that my AVR-based device needs to poll USB constantly for data that isn't changing all that often, so most of that processing time is spent for nothing. Normally, due to the high amount of time spent in polling and processing data from USB, the output will lag as well.

I'm a bit surprised by that, I didn't think our Enttec implementation on an Arduino had those sort of issues but I may be wrong. https://github.com/OpenLightingProject/rgbmixer

It's probably different when the only thing the device does is DMX handling. On my device, I need to handle incoming/outgoing MIDI traffic combined with lots of incoming CDC/DMX traffic. Several UART interrupts are running, ADC interrupt and two timer interrupts. On top of that, I/O processing takes place to determine whether a new MIDI message needs to be sent or not, and lots of EEPROM access as well to read out the parameters for the components I need. With all this combined, having 0.5kB of USB traffic constantly appear on the USB interface of the device really bogs it down - especially when most of the time, the received data is of no real use since it hasn't changed.

With all this, my question is: does OLA support "diff mode" protocol for DMX via USB, where only the changed data would get sent to device? Does any of the supported devices work like this?

Funny you should ask, see #1737 @clxjaguar knows the protocol far better than me. The Milford Instruments stuff also did some odd modes like that, but I don't think we implemented them.

Wow, haven't noticed that. usbdmx protocol looks like exactly the thing I need. Looking at that issue, it seems like some coding help is needed. I'd be more than happy to help! The only difference in my case is that my device isn't FTDI but CDC. I guess this could be handled with just a plugin variant (kind of like various Enttec-compatible devices are currently handled).

Would something like this be feasible?

If you don't need RDM, then yes it should be.

I don't. To be honest, I don't know much about RDM yet, so I also don't know how does the "diff protocol" affect it.

Sending channel=value isn't necessarily very efficient if you've got a lot of changes at once (e.g. a blackout)

Looking at the usbdmx protocol, there is a special one-byte command for blackout.

Sending full frames when they change (and at intervals) might be the safe approach, otherwise you need to tell it when you've finished a frame so it doesn't transmit a partial frame (and it probably needs another DMX buffer internally).

Are you referring to the fact here that usbdmx doesn't have a command to update more than one channel at the time? If so I see how that would be problematic timing-wise. Maybe protocol extension could be added, kind of like:

<bulk_update_cmd> <number_of_channels_to_update / high byte> <number_of_channels_to_update / low byte> <ch1_new_val> <ch2_new_va> <etc>

[peternewman]

It's probably different when the only thing the device does is DMX handling.

Ah yeah, that makes sense.

Wow, haven't noticed that. usbdmx protocol looks like exactly the thing I need. Looking at that issue, it seems like some coding help is needed. I'd be more than happy to help!

Yeah @clxjaguar was going to have a go at the device specific bit (I've been bitten too many times in the past writing code for someone else's dongle and not being able to debug/fix it remotely.

The only difference in my case is that my device isn't FTDI but CDC. I guess this could be handled with just a plugin variant (kind of like various Enttec-compatible devices are currently handled).

It might not even need that, we're using libusb to do the comms, so it would probably just be some discovery or a different endpoint at most.

I don't. To be honest, I don't know much about RDM yet, so I also don't know how does the "diff protocol" affect it.

The diff protocol doesn't really affect it as such, but you'd need to write a whole OLA RDM stack side to talk to it if you wanted RDM, whereas if you're on the Enttec protocol, we already support RDM on the OLA side. Alternatively you could also add a diff option to the https://github.com/OpenLightingProject/ja-rule /OLE code, or arguably a config option in OLA and keep using the Enttec protocol but allow it to only send DMX packets on change.

Sending channel=value isn't necessarily very efficient if you've got a lot of changes at once (e.g. a blackout)

Looking at the usbdmx protocol, there is a special one-byte command for blackout.

Yes, that was just one example, consider a sine wave running along all the channels (to do a nice chase across lights of lights), or something doing a chasing red, green, blue on pixels, both will cause the entire universe to change each frame.

Are you referring to the fact here that usbdmx doesn't have a command to update more than one channel at the time? If so I see how that would be problematic timing-wise.

More that I don't think it had a commit (in the database style), so you might get round to changing half the channels, then it sends a DMX frame, then you finish changing the others, and it sends another frame, so the first one is a mix of half the old frame and half the new one.

Maybe protocol extension could be added, kind of like:

<bulk_update_cmd> <number_of_channels_to_update / high byte> <number_of_channels_to_update / low byte> <ch1_new_val> <ch2_new_va> <etc>

Yeah possibly (although you then need some means of configuring when you can use the extension like a version or model), a few of the other usbdmx protocols implement something similar, although I think you have to split the frame into two to fit into USB packet size limitations.

[paradajz]

Thanks for all the input. As an proof-of-concept, the easiest thing for me was to just modify existing behavior for Enttec USB Pro widget. See the commit here: https://github.com/paradajz/ola/commit/f291147b42570b4a64fe3cc1a7420dd44e244cd1 With this change my devices work as I'd like them to work.

I would appreciate any guidance to turn this POC into actual MR. I don't quite understand the difference between for example "UsbProDevice" and "EnttecUsbProWidget" - which parts should go under "device" and which under "widget"?

Ideally, I'd like for my devices to be handled in exactly the same way as Enttec variants with Widget API using FW 1.0 (DMX send only) with the only difference being sending data on change - mainly because this is a simple change compared to implementing new protocol (usbdmx.com). How should the discovery be handled for my devices?

[peternewman]

Thanks for all the input. As an proof-of-concept, the easiest thing for me was to just modify existing behavior for Enttec USB Pro widget. See the commit here: paradajz@f291147 With this change my devices work as I'd like them to work.

So you're sending three bytes for every actual byte! That's going to be expensive, especially at the start.

I'd have a look at @kripton 's work on https://github.com/kripton/rp2040-dongle/branches where they've tried various RLE and similar encoding to compress the packets, which is probably worth considering.

I would appreciate any guidance to turn this POC into actual MR. I don't quite understand the difference between for example "UsbProDevice" and "EnttecUsbProWidget" - which parts should go under "device" and which under "widget"?

This documentation (from the usbdmx plugin), should explain it: https://github.com/OpenLightingProject/ola/blob/master/plugins/usbdmx/README.developer.md

Ideally, I'd like for my devices to be handled in exactly the same way as Enttec variants with Widget API using FW 1.0 (DMX send only) with the only difference being sending data on change - mainly because this is a simple change compared to implementing new protocol (usbdmx.com). How should the discovery be handled for my devices?

It's probably worth implementing the protocol extensions, then we can detect your device as being slightly different and behave in your special way automagically. You'd just need to register an ESTA ID which is easy enough https://wiki.openlighting.org/index.php/USB_Protocol_Extensions

As an aside, if you see these two, you could implement the Receive DMX on Change (label = 8) and Received DMX Change Of State Packet (Label=9) as a transmit option if you wanted a more compressed packet (in a different format): https://github.com/OpenLightingProject/ola/issues/1290 https://dol2kh495zr52.cloudfront.net/pdf/misc/dmx_usb_pro_api_spec.pdf

[paradajz]

Thanks for all the input. As an proof-of-concept, the easiest thing for me was to just modify existing behavior for Enttec USB Pro widget. See the commit here: paradajz@f291147 With this change my devices work as I'd like them to work.

So you're sending three bytes for every actual byte! That's going to be expensive, especially at the start.

It's expensive only if lots (eg. > 128) channels change at the time. Now in this case, it might be simpler to just send full 512 channels at once, so the device could support both normal, Enttec packets (but even then, just on change!) and diff packets, where only the changed channels get sent. The packet type could be determined by the device simply by looking at the length value: if hard limit is 160 (example) channels in diff mode, that's 480 bytes total, so anything less (and including) that length is diff-mode, while 512 bytes is normal, full-sized Enttec DMX packet.

I would appreciate any guidance to turn this POC into actual MR. I don't quite understand the difference between for example "UsbProDevice" and "EnttecUsbProWidget" - which parts should go under "device" and which under "widget"?

This documentation (from the usbdmx plugin), should explain it: https://github.com/OpenLightingProject/ola/blob/master/plugins/usbdmx/README.developer.md

Thanks, missed that page.

Ideally, I'd like for my devices to be handled in exactly the same way as Enttec variants with Widget API using FW 1.0 (DMX send only) with the only difference being sending data on change - mainly because this is a simple change compared to implementing new protocol (usbdmx.com). How should the discovery be handled for my devices?

It's probably worth implementing the protocol extensions, then we can detect your device as being slightly different and behave in your special way automagically. You'd just need to register an ESTA ID which is easy enough https://wiki.openlighting.org/index.php/USB_Protocol_Extensions

Do I need to register my own or can I use Open Lighting one (asking since I don't have a company which seems to be requirement)?

As an aside, if you see these two, you could implement the Receive DMX on Change (label = 8) and Received DMX Change Of State Packet (Label=9) as a transmit option if you wanted a more compressed packet (in a different format):

1290

https://dol2kh495zr52.cloudfront.net/pdf/misc/dmx_usb_pro_api_spec.pdf

I'll take a look into this.

[peternewman]

So you're sending three bytes for every actual byte! That's going to be expensive, especially at the start.

It's expensive only if lots (eg. > 128) channels change at the time. Now in this case, it might be simpler to just send full 512 channels at once, so the device could support both normal, Enttec packets (but even then, just on change!) and diff packets, where only the changed channels get sent. The packet type could be determined by the device simply by looking at the length value: if hard limit is 160 (example) channels in diff mode, that's 480 bytes total, so anything less (and including) that length is diff-mode, while 512 bytes is normal, full-sized Enttec DMX packet.

Yes, I more meant that e.g. compared to some other methods, it's still a lot of excess bytes (although admittedly is a very simple protocol).

So you're more interested in a custom packet rather than simply only sending complete frames (using the existing protocol) whenever any value changes (which may be more data, it really depends on what is happening in the data).

Thanks, missed that page.

No worries.

Do I need to register my own or can I use Open Lighting one (asking since I don't have a company which seems to be requirement)?

It's far less complicated if you register your own as there are various places where ranges would have to not clash (as well as allocation of UIDs). There are a number of individuals have successfully registered IDs in the past.

As an aside, if you see these two, you could implement the Receive DMX on Change (label = 8) and Received DMX Change Of State Packet (Label=9) as a transmit option if you wanted a more compressed packet (in a different format):

1290

https://dol2kh495zr52.cloudfront.net/pdf/misc/dmx_usb_pro_api_spec.pdf

I'll take a look into this.

I should clarify none of those are standard to be used in this reverse direction, but you could either use that format (as its designed and specified already, and is potentially more compact than your existing one) on a new label, or you could use the pair in the opposite direction.

It probably depends on how much compatibility you want with other existing software, or is the response rate so slow as to be unusable, and therefore wider compatibility is less relevant?

[kripton]

Hi, I'm jumping into this discussion since I was mentioned by Peter due to the compression tests I did (Thanks for hinting this are me!). First, let me state the obvious:

As always, it depends on the kind of data you expect. Do you have only a few channels "active" (!= 0) to start with? => Only send the first X channels OR use compression if you can. Sending only the first X channels won't help you much if channels 0 to 500 are 0 and 501 to 512 are used. Compression won't help you much if many channels are used but they have pretty "random" values. OR if your hardware doesn't have the resources to do decompression fast enough or at all.

Do you have a lot of channels "active" but only few change at a time? => have some kind of "diff" frame. As Peter noted, diff frames don't help you much since if many channels change, you drastically increase the amount of data to be exchanged.

However, the good thing is (at least in your case) that the machine OLA is running on usually has quite some CPU power. So OLA could decide which "transfer method" is the most efficient one (if supported by the dongle, of course): Either sending a full frame, sending a partial frame, sending a diff frame or sending a full or partial frame compressed. Since you stated that it's about AVR chips, I assume that de-compressing data might either increase your code/binary size too much and/or will take too long, runtime wise. Of course, you are free (and invited to ;)) play around with different options (using protocol extensions for any of the available protocols) and enhance OLA to support "more intelligent ways" of transferring data via USB.

In case it might help you, I did some compression algorithm in the Raspberry Pi Pico board (using one Cortex-M0+ core) to see which algorithm fits best for fixed-size, 512 byte long (coincidence, he? :D) data. The results are here: https://github.com/kripton/rp2040-compressiontest. However, be aware that this is on a 250MHz, 32 bit CPU. Even though "heatshrink" is made for embedded systems, has very low code size and memory requirements, it felt too slow with taking over 8ms in the worst case. zlib performed surprisingly well for most use cases and was pretty fast, too. However, I finally decided to go for Google's snappy since it was still faster than zlib and only a bit worse looking at the compression ratio.

Another side note since you mentioned QLC+: QLC+'s "hid" output plugin sends the DMX data in chunks (I assume it's 16 channels at a time, might be more, can't remember). And as far as I know, it only emits the chunks that actually changed. So if you don't touch any channels, no data will be sent to the dongle. This has a negative side as well: In that implementation, the dongle will never know, when a complete frame has been received. If all channels change values, it's easy: if the last chunk has been received, the frame is complete. But if only the first XXX chunks have been received, how do you know the values should be used NOW? If you update your dongle's internal buffer everytime you get a "chunk changed" message, you WILL have tearing in the DMX data (means: if the first X chunks have been updated, but the last Y chunks not, your dongle will send DMX frames that contain partly new and partly old data until all chunks are "current" again).

So ideally, your custom "diff" protocol should have some kind of "sync" frame, telling the dongle that now all diffs are sent and the data is valid from now on. If I remember correctly, ArtNet and sACN also have this kind of sync mechanism, especially if multiple universes are being sent.

I hope it wasn't too much to read, I just wanted to share my current state of knowledge regarding this topic :) Interesting discussion nevertheless!

[paradajz]

Thank you for the detailed response.

As always, it depends on the kind of data you expect. Do you have only a few channels "active" (!= 0) to start with? => Only send the first X channels OR use compression if you can. Sending only the first X channels won't help you much if channels 0 to 500 are 0 and 501 to 512 are used. Compression won't help you much if many channels are used but they have pretty "random" values. OR if your hardware doesn't have the resources to do decompression fast enough or at all.

This is why I proposed two methods: if the amount of changed channels is greater than some specific value, send the full frame. Otherwise, send just the changes.

OR if your hardware doesn't have the resources to do decompression fast enough or at all.

Yeah it definitely doesn't.

As Peter noted, diff frames don't help you much since if many channels change, you drastically increase the amount of data to be exchanged.

Well, that's fine as far as I'm concerned. What I'm really interested in is that my device isn't bombarded with bytes that don't change. And if someone wants to use my firmware for stuff that changes a lot, I can just mention that STM32-based boards are way to go since they don't have performance issues as described, even when receiving full frames constantly. For light usage, AVR is fine, but only with some kind of diff protocol which we're discussing here. I've tested those boards with the simple commit I've linked and it worked really well. With the default implementation, however (with the full frames being sent all the time), my AVR-devices are basically unusable for anything else (and the main purpose of them is the MIDI part actually).

Another side note since you mentioned QLC+: QLC+'s "hid" output plugin sends the DMX data in chunks (I assume it's 16 channels at a time, might be more, can't remember). And as far as I know, it only emits the chunks that actually changed. So if you don't touch any channels, no data will be sent to the dongle. This has a negative side as well: In that implementation, the dongle will never know, when a complete frame has been received. If all channels change values, it's easy: if the last chunk has been received, the frame is complete. But if only the first XXX chunks have been received, how do you know the values should be used NOW? If you update your dongle's internal buffer everytime you get a "chunk changed" message, you WILL have tearing in the DMX data (means: if the first X chunks have been updated, but the last Y chunks not, your dongle will send DMX frames that contain partly new and partly old data until all chunks are "current" again).

So ideally, your custom "diff" protocol should have some kind of "sync" frame, telling the dongle that now all diffs are sent and the data is valid from now on. If I remember correctly, ArtNet and sACN also have this kind of sync mechanism, especially if multiple universes are being sent.

Maybe I'm missing something really obvious here but... Doesn't all software interfacing with OLA send the full frames, always? If that's the case (like QLC+), then I always know when the packet is done - each message a single packet after which state on fixtures should be updated. OLA receives full frame, but depending on the amount of changes I send either full frame or just the changes in a single message. I don't see why would I need "sync" frame here?