Implement direct outputs for DigitalOut and subclasses

[rpanderson]

Amazing work on this highly functional psuedoclock, @philipstarkey! 🎉

I was wondering if it's possible to implement direct outputs on the Raspberry Pi Pico so that DigitalOut, Trigger, etc. can be derived from GPIO.

This would allow cheap cameras and other triggerable devices to require a very simple connection table (and budget!).

[philipstarkey]

It's on the todo list (and I've started modifying the firmware to this end). Basically you'll be able to replace any of the 4 pseudoclock cores with a set of direct outputs. I think the pins will have to be sequential, but I'm hoping that I'll be able to make the number of pins per each direct output core configurable (maybe in powers of 2 up to 2^4). Available instructions for the direct output cores should be 33% higher. There may, unfortunately, be some differences in how wait instructions are represented for direct output cores but that should be abstracted away inside the labscript class.

No ETA on this. Whenever I have a few weekends spare I guess!

[ehuegler]

Hi Phil, I was wondering if you have made any progress on this. I have been working with David Meyer, and we are very interested in making this feature happen. It sounds like you have already put some work in, in which case I would love to help you finish it. Otherwise, I would still be interested in discussing your thoughts on approaching this so I could start working on an implementation.

[philipstarkey]

Yes, I'm pretty sure I started working on it. I'll see if I can dig up the code!

[dihm]

Hey Phil, I have another summer student starting the first week of June that I think could help push this feature over the line (@pmiller2022). If not too much of a hassle, it would be nice to get a peak at any code you have in advance so he can hit the ground running.

On a related note, I've had some discussions with last year's student (@ehuegler) who has become quite expert with the Pico and had some interesting ideas I'd like to run past you as well. Both ideas leverage the idea that picos are cheap and plentiful to get a lot more digital outputs.

1. A dedicated, separate DO board that takes a prawnblaster pseudoclock as an input.
   • Pros: Doesn't require modification of existing firmware; should allow for a lot more outputs
   • Cons: Moves all of the clock+DO logic and timing headache onto labscript itself
2. Leverage the dedicated I2C cores to let a Prawnblaster directly control other daughter Pico boards that implement the DOs (or even more clocklines).
   • Commanding a digital out would be something like:
     • tell PB which sub-board, channel and timing
     • PB sends instructions to sub-board which saves runtime-endcoded commands
     • PB sends start trigger to all boards which can share a system clock so a timing pseudoclock isn't necessary.
   • Pros: I2C and system clock distribution can be quite scalable: could imagine a form-factor like the Pi HAT system (need more DOs, just stack another Pico on the headers); would have a single command interface making the labscript side of things easier
   • Cons: firmware that can do all of that is going to be complex

Ultimately, I'm hoping to have something by the end of the summer that is a full drop-in improvement over the standard pulseblaster.

[philipstarkey]

@dihm I have set myself an alarm for the weekend to turn on the relevant PC and dig up what I have. I tried over Christmas to get something going again and the main issues were (a) testing and (b) modifying the wrapping C++ code to handle a mixture of pseudoclock vs direct output PIO cores (which turned out to need to be spread over the 2 state machines because while each has 4 PIO cores, each block of 4 has to share the same program space which is only 32 lines long (and the existing pseudoclock PIO program takes up all of that).

Anyway, I can share what I have, but I actually really like idea 1 above. There is no reason why you couldn't feed the clock from a prawnblaster (or any pseudoclock) into another Pico based device that just clocked out a state to 8 digital lines from RAM on an input trigger. The PIO code for that would be dead simple. A lot of relevant functionality for DMA transfers could be copied from the prawnblaster. You don't need wait monitors or timing info. So you'd get a decent number of instructions in. You might even be able to fit 3 banks of 8 outputs each across 3 PIO cores depending on the available contiguous pins on the Pico (maybe only 2 🤷).

And then this new device would be an IntermediateDevice for labscript purposes. Which requires very minimal effort to write a device class. Labscript will take care of the timing (and it won't care what pseudoclock is driving it).

The hardest thing will be coming up with a name for the device 😁 I'm happy to create a repo for the firmware inside the labscript suite GitHub organisation if you like (once you have a name)

[carterturn]

I happened to write a brief implementation of approach 1 (the separate Pi Pico being used for digital outputs) for testing Labscript integration of devices that need a trigger. I have not had an opportunity to test it with a nice enough oscilloscope to verify that it is sufficiently accurate, precise, and repeatable for use in actual experiments, but it may already be good enough or at least a good place to start.

The Pi Pico code is in a repository here: https://github.mit.edu/zwierleingroup/prawn_do The Labscript device code is in the prawn_do folder here: https://github.mit.edu/Zwierleingroup/labscript_user_devices (I think those repositories should be read access for everyone; if not, please let me know and I will adjust.)

The name was decided on before I thought this might be useful to other people, hopefully it is not too uninformative, but it would not be too difficult to change.

[dihm]

@carterturn Unfortunately it does not appear that read access has been granted. I'm definitely interested in having a look and suspect it will be very helpful. My student will be starting in a couple weeks and between this and Phil's draft code, I think he'll be able to knock together something very reliable and well tested.

The name was decided on before I thought this might be useful to other people, hopefully it is not too uninformative, but it would not be too difficult to change.

To be honest, your name is much better than my initial suggestion. And as you say, changing project names is rather simple.

Final thought that crossed my mind this week. When using a separate board, I think we are going to suffer from the timing jitter on every trigger of +/-1 clock cycle (see #5 for a lot of details). I suspect that a useable device will need to share a clock with the master prawnblaster to ensure consistent results.

[philipstarkey]

Final thought that crossed my mind this week. When using a separate board, I think we are going to suffer from the timing jitter on every trigger of +/-1 clock cycle (see #5 for a lot of details). I suspect that a useable device will need to share a clock with the master prawnblaster to ensure consistent results.

I'm having trouble understanding everything I wrote in that thread, but is the jitter described only about the PrawnBlaster wait monitor due to the fact that we need to count the length of the wait as well as check for the trigger? (see here)

I think if you were using wait 1 pin 0 (which the PrawnBlaster does on the first trigger to start a shot) you don't see jitter?

[dihm]

My (perhaps incorrect) logic here is that registering a trigger can only happen synchronously with the clock. If the trigger arrives mid-clock period, the trigger won't register until the next clock tick. Honestly I barely remember what was going on in that thread as well. Ultimately will just need to check. Sharing clocks isn't that hard if it comes to it.

[philipstarkey]

My (perhaps incorrect) logic here is that registering a trigger can only happen synchronously with the clock. If the trigger arrives mid-clock period, the trigger won't register until the next clock tick. Honestly I barely remember what was going on in that thread as well. Ultimately will just need to check. Sharing clocks isn't that hard if it comes to it.

Yep, that's a concern. I think it would to half the jitter we saw, but I could very easily be wrong. I would also like to think that if both were synchronised to the same external reference, that the delays between reference clock tick and output of pseudoclock tick, would be sufficient to always occur clearly in-between the reference clock ticks, such that the second pico would be consistently running 1 reference clock tick behind the pseudoclock (which should lead to very minimal jitter). But as we found out several times, the PIO cores don't always behave as we expected! 😁

[carterturn]

It seems the MIT Github permissions are more complicated than I had anticipated. I have mirrored the repositories to normal Github here: https://github.com/carterturn/prawn_do https://github.com/carterturn/zwierlein_labscript_user_devices

Final thought that crossed my mind this week. When using a separate board, I think we are going to suffer from the timing jitter on every trigger of +/-1 clock cycle (see https://github.com/labscript-suite/PrawnBlaster/issues/5 for a lot of details). I suspect that a useable device will need to share a clock with the master prawnblaster to ensure consistent results.

Something of that order sounds very plausible; the issues outlined there are definitely not tested for or addressed in my implementation (just due to lack of time). I have not looked into external clocking of Picos yet, hopefully this is not too hard to add, or at least the existing code can be an inspiration.

I will also try to add more inline documentation to the code later this week.

[pmiller2022]

Hey Carter,

I am working with Dr. Meyer now to try and get this feature working with the help of the code you've made so far, however on the current public GitHub there is no inline documentation, making it a little harder to read. If you could add inline comments soon to the public GitHub that would be a great help to me.

[carterturn]

I apologize for the delay; the comments have now been committed and pushed to the public repository. Please let me know if there are points or sections I should clarify better.

[pmiller2022]

Thank you for the comments, they have been a great help. We have the firmware working on our end, we would like to develop off your code and test it. Would you prefer us to raise issues to you directly or for us to fork off of your code and develop it independently?

[carterturn]

That was quite fast! I think your testing environment is probably more thorough than mine, so I think it would make sense for you to have a fork so you can make the final decision on pull requests.

[pmiller2022]

Carter and Phil,

We've been testing the digital outputs and found them to be fairly accurate until ~100ns pulse width and we were trying to lower that but have run into problems trying to reduce the clock divide on the PIO to get the full 100 MHz where the DO board seems to miss a random collection of triggers. We have a second potential path from here with making the DO board function like the prawn blaster and only require the initial trigger and then having the DO board time itself. Let us know if either of you have any ideas or feedback.

[carterturn]

Interesting; the 100ns limit agrees with what I observed, but I had assumed it was an issue with noise pickup in my breadboard setup.

The trade-offs I see between internal timing and external triggering are simultaneity and precision. With the internal timing, the board can probably reach higher precision, but at the expense of simultaneity with other devices (unless external clocking is used). I have not actually tried to setup external clocking of a Pi Pico, so if sharing the Prawnblaster clock to the DO board is easy in practice, then strongly encouraging external clocking and using the internal timing option is probably the better option.

Without external clocking, I think the simultaneity issue might be non-trivial, so if external clocking is hard, making it the only option would not be great. The onboard oscillator of the Pi Pico has a specified stability of +/-50ppm, which implies a worst case drift of 3ms over a 60s shot. Some forums suggest that it is +/-1ppm in practice, which is much better, but that would still be enough drift to cause problems with (for example) absorption imaging if a camera and an AOM were triggered from different boards.

At least for my lab, I think the 100ns pulse width would be sufficient for most purposes (we currently use NI PXIe-6535 cards for digital outputs, which have a 10MHz clock). I am not sure how broadly applicable this is, but documenting the limitation may be enough.

The best functional option (at the expense of complexity and maintainability) would probably be to offer a switch between an internally clocked, triggered once mode and an externally clocked, many triggers mode. This might not be too difficult, as the internal triggering would probably be setup on a separate state machine, and thus be fairly well decoupled. I have not tried to setup inter-state machine triggers though, so I am not totally certain how complicated this is in practice.

[dihm]

Quick update on this.

First off, there is an even more fundamental reason the 100ns pulse width is the limit when externally triggering. I should have realized this earlier, but the prawnblaster cannot set rising edges faster than 100ns, so in the simplest external triggering mode where you only use rising edges for timing, that's what you get.

We also noticed some strangeness in external triggering based on the divide by 10 of the PIO clock, mostly that it limits the timing resolution of the external triggering to the same +/- 1 clock cycle (or 100ns). You can actually remove that divide by 10 and keep the PIO at full speed, but you have to be very careful about impedance matching between the boards to limit reflections getting interpreted as extra triggers. We've been able to completely fix that by making custom eval boards with unity-gain buffers on the prawnblaster output and the DO triggering input.

At least for my lab, I think the 100ns pulse width would be sufficient for most purposes (we currently use NI PXIe-6535 cards for digital outputs, which have a 10MHz clock).

I agree that this is generally true, but I also suspect that people will always want that little bit more and if there is a fairly simple way to get it, it is probably worth some effort. Using prawnblaster code, we can get the internally timed board down to the same 50ns pulses the prawnblaster can do. Given that I suspect we will need to externally clock no matter what so we don't get arbitrary +/- 10ns phase slips between the devices, we think the effort may be worth it. It's also nice that the number of external triggers to process goes down a lot, making inter-device connects a little less annoying.

@philipstarkey We are basically using your drafted PIO code with as much extra complexity as possible stripped out since the plan is to not incorporate directly into the prawnblaster itself. The thought is that we can insist on external start triggers and restart after wait triggers (so indefinite waits only as far as the DO board is concerned), leaving bit-banging of sequences of user-provided 16-bit words for user specified number of clock cycles in between. I vaguely recall you saying that you had a method for dropping the number of ASM instructions to 4 at one point. Since there are fewer branches to consider here, I wonder if it would be viable for us. Do you remember more details on what was involved? We'd be interested to workshop it a bit as that could get us to 25 MHz.

[philipstarkey]

@dihm no I don't remember what I tried to do. That said, the PIO code is probably sufficiently different that it wouldn't matter anyway. If you can share the PIO code, I'm happy to take a crack at optimising it down to less instructions.