Z-Stack_Issue_on_ASI_stage

There were 3 different types of situation:

While acquiring Z stack across large range with >1 step size, in the first half of the acquisition seems to be normal, but the second half stage stopped moving and showing:

Click me

``` ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range ```

While acquiring Z stack across small range with 0.2 step size, stage not moving, and showing:

Click me

``` ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range ``` 6. While acquiring Z stack with 0.5 step size, stage moved, but moved unevenly. ``` Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT ASI stage move axis absolute failed or is trying to move out of range: :N-4 -> Parameter Out of Range Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT Waiting for device... Waiting for device... Waiting for device... error message: DCAMERR.TIMEOUT error message: DCAMERR.TIMEOUT ```

Okay, so this is something that we should discuss as a team, and it is somewhat related to Issue TheDeanLab/ASLM#180. To the best of my knowledge, both Doug Shepherd and Adam Glaser both take the strategy where the stage is running continuously (albeit slowly), and they are grabbing images as it goes. Doug Shepherd's code is here, and Adam's figure is attached. Some of these tricks may also work for the multi scale microscope too.

Basically, tell the stage to start moving at a specific velocity for a specified distance. The ASI stage provides a TTL when it is at the desired velocity, which they use in the figure to trigger the camera. You then capture data as the scenery slowly moves through the detected volume. Now that we are acquiring data in a continuous loop, with deterministic timing between each image (at least with the perStack setting), this could work and it would eliminate the hardware-software triggering back and forth.

Regarding the different illumination modes, this could be considered one of them. I'm not sure what the best way to make things maximally abstract, or to add some of these features without cluttering up the code, but it might be nice. For example, our remote focus settings are pretty much only wired to work with sawtooth waveforms. Melissa will need sine waves for her TIRF system, and perhaps someday we want to try out the triangle wave for bidirectional ASLM. Presumably each one of these would then want to also have the ability to do a z-stack, operate in a continuous mode, etc.

I haven't looked into this in too much detail, so I may be incorrect, but to me this feels like two separate issues:

What @JinlongL posted looks like a bug, or at least mis-set stage bounds, and I think warrants further investigation. It's possible the bug was fixed in #294, not sure.
What @AdvancedImagingUTSW suggests would circumvent the bug, and is probably the long-term correct approach.

I think we could probably address (1) to fix the acute issue and deal with (2) over a longer time period.

Side note: for long tracebacks, such as the ones posted by @JinlongL, it can be convenient to use <details> and <summary> tags to create collapsable code blocks. See https://gist.github.com/scmx/eca72d44afee0113ceb0349dd54a84a2.

Reading into this a bit more this morning... After speaking with Adam Glaser, he recommends that we trigger everything off of the linear encoder, rather than the constant velocity signal. They will share with us some code so that we can see exactly how they are doing it.

In the meantime, while looking through Doug Shepherd's code...

The camera runs in an internal triggering mode, which is initiated by a +5V TTL start trigger. It then runs continuously, sending out +5V TTLs on output triggers 0:2. (line 228)
Sets up constant speed stage scanning (line 409). Function setup_asi_tiger not provided in repo, and could not find it elsewhere in his GitHub organization.
Creates binary waveform for triggering lasers (line 426).
The tiger controller is set to only send serial commands on change (line 469), then to not do this (line 493), then to do this again (line 504).
Then the part I am most interested in... Line 535

                        taskDI = daq.Task()
                        taskDI.CreateDIChan("/Dev1/PFI0","",daq.DAQmx_Val_ChanForAllLines)

                        ## Configure change detection timing (from wave generator)
                        taskDI.CfgInputBuffer(0)    # must be enforced for change-detection timing, i.e no buffer
                        taskDI.CfgChangeDetectionTiming("/Dev1/PFI0","/Dev1/PFI0",daq.DAQmx_Val_ContSamps,0)

                        ## Set where the starting trigger 
                        taskDI.CfgDigEdgeStartTrig("/Dev1/PFI0",daq.DAQmx_Val_Rising)

                        ## Export DI signal to unused PFI pins, for clock and start
                        taskDI.ExportSignal(daq.DAQmx_Val_ChangeDetectionEvent, "/Dev1/PFI2")
                        taskDI.ExportSignal(daq.DAQmx_Val_StartTrigger,"/Dev1/PFI1")

                        # ----- DIGITAL output ------   
                        taskDO = daq.Task()
                        taskDO.CreateDOChan("/Dev1/port0/line0:7","",daq.DAQmx_Val_ChanForAllLines)

                        ## Configure timing (from DI task) 
                        taskDO.CfgSampClkTiming("/Dev1/PFI2",DAQ_sample_rate_Hz,daq.DAQmx_Val_Rising,daq.DAQmx_Val_ContSamps,samples_per_ch)

                        ## Write the output waveform
                        samples_per_ch_ct_digital = ct.c_int32()
                        taskDO.WriteDigitalLines(samples_per_ch,False,10.0,daq.DAQmx_Val_GroupByChannel,dataDO,ct.byref(samples_per_ch_ct_digital),None)

                        ## ------ Start digital input and output tasks ----------
                        taskDO.StartTask()    
                        taskDI.StartTask()

He uses PyDAQmx, not the official nidaqmx library. This means some of the syntax is similar, but not exact to what we are used to. The documentation is not excellent, but it shows signs of usefulness here. He appears to be continuously screening PFI0 for TTLs (from the camera), and then when they are received, he uses these to trigger the lasers.
Then he acquires the data... line 584. No idea where the acq. object comes from, or where he actually sends the TTL to start the camera. My impression is that the digital input task is listening to the camera, and the digital output is waiting on the digital input.

Nonetheless, something like this, where the stage's linear encoder is sending signals to a digital input, which then triggers the waveform, is probably what is necessary for @JinlongL. Information on how to retrieve the various signals from the stage are provided here.

Working through the example, for our applications and with our hardware. We have a stage resolution of TPI4 (4 turns per inch), or 6.35 mm. The rotary encoder thus has a resolution of 45,396 counts/mm, or 22 nm/count. The minimum clock pulse is 4x the encoder resolution (it is a quadrature device), given us the minimum useful clock rate of 88 nm/pulse generated - we basically have to work with a multiple of this value.

In the diSPIM orientation, the stage movement is 45 degrees relative to the illumination direction. Thus to sample at ~160 nm in Z, we need to step 160 * sort(2) laterally, or or 226 nm. This is 2.5 pulses... If we were to go with 3 pulses, 264 nm, divided by sort(2), this would give us a z-sampling of 187 nm. Especially for aqueous solutions, this is probably good enough. We still have to provide the 22nm resolution to the command, as follows:

SCANR X=0.0 Y=1.0 Z=12 (X= start (mm), Y= stop (mm), Z=encoder_divide (desired step size, divided by 22 nm)

Then I think we specify the velocity as integration time / desired step size per integration.

If we have a 20 ms exposure time (the goal), desired step size of 264 nm (0.264 microns): 13.2 mm/s. The maximum speed is actually 7 mm/s for the TPI4 stage. So if we specify our exposure time based upon the max velocity, then we should be at 7 mm/s = 0.264/x... x = 38 ms.

More thoughts walking in this morning. In the configuration file, we and an encoder pinout to the stage device, or to the NI device. It kinda makes more sense for the latter, but it is a property of both. This will be the digital output from the ASI encoder.

We can then essentially make a specific version of the z-stack, where we program in the aforementioned stuff above. Pretty straight forward. And finally, rather than triggering the software with the digital signal we send from the DAQ to itself, we use the external signal to trigger a single finite acquisition.

Correction. 13.2 microns/second. The stage is not actually rate limiting.

@annie-xd-wang - This is the feature that I mentioned earlier on Slack. I started mentally thinking about it a little bit on the asi_scan_feature branch, but my progress so far is pretty pathetic. The file I added is the constant_velocity_acquisition under mode/devices/features.

The idea is that on @JinlongL's microscope, we need a special acquisition mode which hopefully we can implement as a feature. In short, for most of our software, the DAQ is set to run all of the devices upon receiving a voltage trigger from itself. Here, we want all of the data acquisition to be set up the same, but this time, instead of receiving a signal from itself, it will receive that signal from a different piece of hardware.

The way I imagine it is that we would use it like a z-stack. Thus, the start, finish, step size, and camera exposure, would be retrieved from the GUI. From this, we would calculate what velocity the stage should run at, and how often it should send us a signal. There are a few potentially confusing aspects - for example, the stage is canned at 45 degrees relative to the imaging system, so if we want a step size of 160 nm we need to move the stage 160 * sqrt(2). Also, we can't choose any step size that we want - it has to be a multiple of 88 nm (which is hardware specific).

Once this basic math is done, then you have to set the stage to move that distance, at that speed, with the number of multiples of the encoder divide (the 88 nm thing). The DAQ would calculate the waveform for the first channel, and run it in a finite mode every time it receives a trigger. I was able to get this working in lab the other day by using the scanr function in this library, followed by the start_scan function. Once I did this, the stage moved at the velocity I wanted and sent out the signal that I wanted. And, if I changed the DAQ wiring to that it could no longer trigger itself, but rather was triggered by this device, the entire system functioned.

So, for this to work we would need to...

Change the trigger source to some other channel (I can specify that later).
Calculate all of the stuff above and send the first command (scanr) to the stage.
Get the data acquisition system ready to run and start it
Start the stage scan with the start_scan function
Acquire until it is complete
Change the trigger source back to the original channel.

If there are multiple channels, we can start with everything running in a perStack mode. The only time a perZ acquisition even begins to make sense is if the camera settings and waveforms are identical for the channels, and there is no filter wheel. So, I would think that we only implement the perStack mode for now.

6259

M Series User Manual

Specifications

Good for the PCIe-6259 data acquisition card, which is what we were trying to send use for triggering. See section 5-4 of the manual. Earliest NI-DAQmx version support - 8.3 for PXIe

Finite sample mode generation refers to the generation of a specific, predetermined number of data samples. Once the specified number of samples has been written out, the generation stops.
Analog output supports two different triggering actions: • Start trigger • Pause trigger An analog or digital trigger can initiate these actions. All M Series devices support digital triggering, but some do not support analog triggering. To find your device’s triggering options, refer to the specifications document for your device. Refer to the AO Start Trigger Signal (section 5-6) and AO Pause Trigger Signal sections for more information about these triggering actions.

AO Start Trigger Signal

Use the AO Start Trigger (ao/StartTrigger) signal to initiate a waveform generation. If you do not use triggers, you can begin a generation with a software command.

Using a Digital Source To use AO Start Trigger, specify a source and an edge. The source can be one of the following signals: • A pulse initiated by host software • PFI <0..15> This is what we are using • RTSI <0..7> • AI Reference Trigger (ai/ReferenceTrigger) • AI Start Trigger (ai/StartTrigger) • PXI_STAR

Retriggerable Tasks

This is for analog input tasks that are externally triggered. Possibly relevant, but maybe not. Documentation Here

If you are not using an NI 63xx Multifunction I/O (X Series) DAQ board, analog input is not inherently retriggerable.
However; you can retrigger a board by starting and stopping the DAQmx task each time a trigger is received.

6738

Also used for some of our microscopes. FYI...

Now brainstorming in the nidaqmx documentation.

register_done_event() - This also seems like it should work. It will callback a function of our choosing when a finite task completes execution. So, perhaps we could stop and start the task rapidly in a callback function?

Quick discussion with @annie-xd-wang

Will either update our API, or adopt the Allen Institute's API.

If we do the latter, will consider either adopting a static version of their code, or adding it as a submodule in git.

I believe that we have solved this issue. Now @JinlongL just has to prove that it is a good strategy.

TheDeanLab / navigate