Implement checking of stop bits and channel counter

[HakanL]

In my application I need to handle DMX input using sources that may send less than 512 channels. The issue is that I don't know how many channels are sent, a user can connect any source, so I need to dynamically support any number of channels (24-512) and accurately detect how many channels were sent. I have been able to implement checking of the stop bits in PIO, which could be used to detect the end of a frame. But I'm struggling with how to communicate the number of received channels back to the C code. I've toyed with raising a IRQ at the end of the frame (which essentially is when the stop bits are missing = invalid). But I can't figure out how to capture that in the C code and use that as a channel counter, and somehow sync that with the DMA transfer. I've also thought about using a 16-bit buffer for the DMX transfer, and set the high byte to indicate when the stop bits are wrong, so I could then count the number of channels in the C code. But not only would it be a waste of memory, it would also be a fairly breaking change to this library that I would like to avoid. Would it be possible to have a counter in PIO and then push that at an invalid stop bit somehow? Is there another way? I'm fairly new to PIO so I may be missing something obvious.

[jostlowe]

Hmmmm interesting ideas!

I have not worked on this library in a little while so I have to admit that I'm a bit rusty on the whole PIO shctick. I can at least throw out some ideas!

What might be possible is to modify the PIO assembly so that it reads the number of channels received into a scratch register as the data is received by the PIO. Hopefully it is possible to read the scratch registers from software (?)

Another way might be, as you suggest, to trigger a IRQ when the PIO detects the end of a frame. After a IRQ, a call to dma_channel_abort could stop the DMA transfer of the data before the completion of the full 512 channels have been recieved. Perhaps it is possible to read the number of bytes transferred by the DMA instance somehow?

What are your thoughts?

[MikeEllis-personal]

Working on my forked version of this code (in pure Python), one major challenge I still have to resolve is how to handle "short" DMX frames containing less than 512 channels. The DMX standard (ANSI E1.11-2008 R2018) only allows you to identify the start of the next frame, not the end of the current one. The idle time (Mark Before Break) can be anything from zero up to one full second, and the receiver must be capable of handling "a signal with a refresh rate of 0.8Hz to approximately 836Hz". My current solution is to raise an IRQ as soon as the BREAK (or a malformed value) is detected, i.e. after a constant low for more than 9 bit periods, or 36us) - but this is remarkably close to the start of the next frame which could occur only 56us later, thus presenting a potential race condition.

The other issue I'm still struggling with is handling non-zero START codes, which is essential if working with RDM-enabled devices.

[HakanL]

@jostlowe Yeah the issue I'm having is how to communicate the channel count over (and INCing a register isn't that straightforward in PIO, but it can be done). I could PUSH it into the fifo, but that means the DMA transfer will be short, so I don't think that will work. I could push 512 bytes, plus the counter, but that's also annoying. I've read that it's possible to read the number of transfers in DMA, so it may be possible to do that, but I'm a little rusty on how to catch that IRQ and I can't find an example that does something similar.

@MikeEllis-personal I was going to use the lack of start+stop bits as an indicator that it's the end of the frame. I realize there's an maximum gap between channel bytes, but I've yet to see a source that has more than negligible gap, so it's a trade-off I'm ok with doing. I have several sources that send less than 512 channels, so that's a more important issue to resolve. It sounds like you're doing the same thing. Do you still use PIO, but with the "program" in Python, or are you saying you "bitbang" in Python as well? FWIW I don't see how a non-zero START would cause issues, it's just the first byte, right? But I haven't attempted to work with RDM devices.

[MikeEllis-personal]

@HakanL

One trick I've only just spotted re-reading the standard as a result of your query is that for START=0 frames, once one frame has been received, ALL subsequent (compliant) START=0 frames MUST be the same length - thus knowing how long the frame is makes it MUCH easier to generate the "end of frame" IRQ more easily. It also makes proper handling of non-zero START frames even more critical, notwithstanding that the bytes which follow the non-zero START are NOT dimmer values - they can be ASCII text messages, used to configure the DMX receivers in other devices, to poll the DMX chain for capabilities and a whole host of other functions.

I'm using a modified version of @jostlowe 's PIO code for the core the DMX receive and transmit, with DMA transfers to/from the main Python code and an IRQ at the end of frame - and it is Python's slowness at responding to that IRQ that causes the race condition currently with short frames. It is hosted on GitHub (https://github.com/MikeEllis-personal/DMXfire) - feel free to look. I expect that following this conversation, I might pick up the project again and implement the START=0 checks and improve the short-frame handling.

[HakanL]

Hmm, interesting, but how would one know when the source changes? For example I have an ArtNet-DMX device that can send X channels per frame. I can send ArtNet packets of various length to it, and it will output that many channels per frame, but I believe it just changes on the fly, to my knowledge there's no "down period" to reset the frame length (but I haven't verified). I think it's "safer" to assume that any frame can be of any length, but assume that the gap between bytes is minimal and constant. At least that's what I'm planning to go with :) And use the lack of a valid byte to indicate the end of a frame.

[MikeEllis-personal]

It would be worth checking the behaviour of the ArtNet-DMX device - it could perhaps use a non-compliant "long idle" (>1 second) to signal that it has changed the frame length, or maybe driving both lines low at the same time? I am nervous about assuming that the inter-byte gap is "tiny" (<40us) - I've seen some pretty bursty DMX frames, some consoles (e.g. ETC Colorsource) allow the user to select "Max/Fast/Medium/Slow" DMX, and some devices (e.g. the EntTec DMX USB Pro) even allow the user to configure the timings manually.

Perhaps the best option, however, is to combine both approaches. Initially "learn" the frame length, and assume that subsequent frames will be the same length (as the DMX standard requires), but also check to see if longer/shorter frames are received and use this to trigger a re-learn.

Do watch out for the non-zero START frames though - these almost certainly will be a different length, and there is no restriction that they all have to be the same length!

[HakanL]

I hooked it up, it seems that it's just changing the frame length on the fly. Granted I didn't hook up a scope, but looking at DMXcat I can see the number of slots changing almost immediately when I vary the number of channels sent over ArtNet. The "speed" settings on some devices/consoles are typically indicating the update/refresh rate (up to 44 Hz). I have an Enttec clone and these are the settings: Source: https://dmxking.com/downloads/ultraDMX%20Micro%20User%20Manual%20(EN).pdf It does note that in "maximum compatibility mode" that it adds a 44uS interslot delay. I have yet to hear of anybody actually using this (I've sold these devices since 2015 in the US/CA). It was primarily added for non-compliant cheapo light fixtures, but I believe those are probably long gone by now. Definitely a valid concern though, but I still think that even if there is an interslot delay, it would be constant in the frame. Your point about non-zero START frames is definitely valid, and I think if for no other reason than that, that it would be best to support variable frame length in each frame, using some learned frame length in PIO seems a little too complicated (and we're still limited to 32 instructions, I'm at 17 now with stop bit detection). FWIW I'm not trying to argue, just trying to come up with a solid solution (I'm selling the hardware where my code will run) :)

[kripton]

Please see also https://github.com/jostlowe/Pico-DMX/issues/21#issuecomment-1462486196: I've used rp2040's "alarm timer" to detect whenever the line is low for at least 88µs (= BREAK) to detect the end of a frame. It's quite reliable but due to the high number of IRQs, the CPU load is a bit high. However, the "end of frame, prepare for next frame" doesn't work as I would expect it. Yet. We could also use a second PIO SM to detect "the condition" (= line has been low for 88µs) but that would require two SMs (out of 8) for each universe. Best option would be to have a "LOW for 88µs"-detection in the same state machine as the DMX input code. But I didn't manage that since it would increase the number of instructions and we might need a higher clock rate so we can run "more code" and still don't miss a single bit. I will have a look at your PIO code to see if we can combine that. Maybe we don't need the DMA's "byte/frame" counter at all and can just rely on the BREAK detection?

[HakanL]

@kripton The issue I see with BREAK detection is that in the scenario of <512 channels then we'll miss the last frame sent (unless you have yet another alarm timer to catch when you don't get the next BREAK. Or did I misunderstand how your timers work? I have a discussion on the RPi Forum as well, and at the moment I'm leaning towards trying to abort the DMA transfer when the start/stop bits don't show up (i.e. about one byte after the last byte). I'm not sure this will work, and I haven't figured out how to signal an interrupt in PIO and catch it in the c-code to abort the DMA (and then figure out the transfer count).

[MikeEllis-personal]

@HakanL I get it absolutely, and not seeing you as "arguing" at all - these discussions are where the great ideas come from.

You might like to look at my Python code to see how to do the IRQ from PIO - it's actually very simple. My revised code also includes "IRQ-on-BREAK" detection - but (as I discovered) waiting for the BREAK means the IRQ can be raised too close to the start of frame for the Pico to process it and configure the DMA correctly before the data starts flowing again, resulting in at best the loss of a frame and at worst reception of a corrupted frame. Writing the code in C rather than Python might be enough to resolve this though.

Most senders I've seen use the BREAK as "start of frame" (and the standard is written with that implication) rather than "end of frame", hence the longest and most variable idle time is the MBB (Mark Before Break) which is allowed to be anything from 0us (i.e. no gap at all between the last STOP bit and the start of the BREAK) all that way up to one full second - around 200us seems to be common. The MAB (Mark After Break) is generally pretty stable and close to the minimum allowed (12us Tx, 8us Rx), as is the BREAK itself (92us min Tx, 88us min Rx).

I'm not sure that "missing the last frame" is a real concern - frames must occur at least once per second, and the transmitter is not allowed to assume that the receiver receives every single one (e.g. due to interference), so the "last frame" really is an edge case for a system which is already broken to all practical intents and purposes. A potentially bigger concern is that if the transmitter is sending the DMX at the slowest rate permitted (1/sec), waiting for the BREAK will introduce almost 1 second of potentially avoidable latency.

[kripton]

@MikeEllis-personal : I'll have a look at your PIO code later, I'm interested about the condition you trigger the IRQ on. Missing a frame is not ideal but tolerable in "only NULL Start code" frames. Losing one frame is really bad when doing RDM though

[kripton]

What might be possible is to modify the PIO assembly so that it reads the number of channels received into a scratch register as the data is received by the PIO. Hopefully it is possible to read the scratch registers from software (?)

As far as I can see from the PIO register map in the RP2040's data sheet, reading (or writing) the scratch registers from the CPU is not possible. Obscure ways one can transfer additional data:

• Push more "bits per DMX channel" to the FIFO, using the upper/unused bits for "status" (as proposed by @HakanL)
• When the PIO SM is "paused", one can make it execute a simple program that loads the sratch register and push it to the FIFO. Before re-setting the SM, that is
• There is a register DBG_PADOUT for each PIO. It's 32 bit wide (of course) and probably only some GPIOs are driven at a time from one state machine. One could probably set some unused bits from the PIO SM and read that info from the CPU. Crude, Hacky and possibly hard to do with the limited number of instructions

[jostlowe]

@HakanL Just out of curiosity: What in your application is it that needs to know the exact length of the DMX frame? ☺️

[HakanL]

@jostlowe I'm building a DMX "re-mapper", that can change DMX channels on the fly so you can for example drive LED RGB fixtures (that may need some channels fixed at 255 to turn on, etc) with simple fader consoles. One use case is for installations where it's hard to change the programming (it was created during installation and then left as-is) but the fixtures have to be moved/replaced. Think theme parks where the programming was created externally when an attraction was built. So I need to output the same stream that I'm receiving (and making changes on the fly), if I receive 24 channels in then I need to output 24 channels. It could be possible to always output 512 channels, but the issue is that some devices with <512 channels run at higher refresh rate, I need the output to mirror the input as much as possible.

[HakanL]

Thanks for all your input. I was able to get this to work now. I ran into one major issue where I needed to do a WAIT, but with a timeout, which obviously doesn't exist in PIO. I had to do a loop while checking for a LOW input, and to get the timing precision I opted to increase the processing speed to 2 MHz (and just adjusting the delays to fit the DMX-512 spec). Since these are breaking changes I'm not sure if it's appropriate to incorporate them here as a PR, I've forked a version and made it available here: https://github.com/DMXCore/Pico-DMX It now handles varying size packets and I've tried with my fastest sources, my Enttec-clone set to slowest speed, and it handles 1 through 512 channels without issues. The simple console that outputs data at 200+ Hz (24 channels) works great too. In summary I ended up using the lack of the start/end bits with a 100 uS timeout as an indicator for the end of the packet. Note that I've only tested this on PIO0 with SM0, it's possible there are some necessary changes to make it work with PIO1 and SM1-3, I didn't look into that.

[HakanL]

And for clarity, what I ended up using to communicate the number of channels is by reading the DMA transfer count. Because of how DMX works, there is a very defined quiet period, so the DMA is just stalled at that point, which makes it reliable to read the transfer count at that time. Initially I had code that used a counter in the PIO and then pushed that to the FIFO after raising an IRQ. I then compared that to the DMA transfer count, and it was always identical, so then it was cleaner to just use the transfer count. This could probably be a steppingstone to implement RDM, but at the moment I don't have a need (or time) to implement RDM. Shameless plug, this is the product I'm building: https://dmxprosales.com/products/dmx-core-pico2

[michaelglass]

@HakanL thank you