microscope-cockpit / cockpit

Cockpit is a microscope graphical user interface. It is a flexible and easy to extend platform aimed at life scientists using bespoke microscopes.
https://microscope-cockpit.org
GNU General Public License v3.0
35 stars 26 forks source link

Timelapse reps broken in status bar #886

Closed iandobbie closed 3 months ago

iandobbie commented 8 months ago

if I have a time lapse with eg 5 repeats of 6 slices. The status bar starts with cam: 0/30 this counts as images are received until we get to the end of the first stack of 6 images, then it vanishes. The time lapse appears to continue but there is no evidence that it is running.

iandobbie commented 8 months ago

On the red pitaya and with a picamera time lapse is totally broken. The first cycle happens but then cockpit returns and appears to think the experiment is finished. The Red Pitaya continues and completes the experiment, triggering cameras etc... but the data saver appears to have quit by then and the output file just stores the first cycle.

There is something broken the the logic, however I am certain this worked on the DSP so I assume is is something to do with some interactions between cockpit and the Red Pitaya.

iandobbie commented 8 months ago

And yet the entirely simulated system does the right thing and will delay between repeats and generates a complete stack with the correct number of frames. I guess this is an issue with the executor and how it is repeating the stack.

iandobbie commented 3 months ago

But only if you have no active light, as an active light produces an error that no executor knows how to handle the light.

iandobbie commented 3 months ago

And even when the simulated time lapse works (ie when we have no active light) the interval doesn't get the expected display in the status bar. I suspect these are all executor issues.

iandobbie commented 3 months ago

The time lapse repeat is fundamentally broken on the Red Pitaya.

1) Repeats are handled by the Red Pitaya so no return to cockpit between time lapse repeats.

2) The executor RunActions function calls trigCollect on the Red Pitaya that starts the C based runner program then immediately returns 'DSP done' and the current analog positons

3) The executor device then returns a cockpit event EXPERIMENT_EXECUTION that the experiment thread is waiting for, it then returns.

4) the only thing keeping the experiment running is the data saver in the loop

        while (time.time() - self.lastImageTime < 1.0
               or not self.imageQueue.empty())

So basically if the image queue is empty for more than 1s the experiment terminates and the file is closed etc...

We need some more checks here, long exposures, time lapse with more than 1s between exposures or something else slowing things down will kill this.

We need to check, 1) Have all the images arrived? 2) Are we in a lapse interval? If so update the status bar and wait for more images 3) What if we have longer exposures a 3s exposure would halt the imaging almost immediately.

iandobbie commented 3 months ago

If the handler returns for each rep (as I believe the DSP does) we get this code at the end of the loop.

            # Wait for the end of the rep.
            if rep != self.numReps - 1:
                waitTime = self.repDuration - (time.time() - startTime)
                time.sleep(max(0, waitTime))
iandobbie commented 3 months ago

Ok, I have put a fix in place that delays the timeout till a repDuration+1s. It also adds a timer to the status bar of time till next repeat. This required passing the repduration to the datasaver but seems like a nice UI upgrade that lets you know why nothing is happening.