StageAwareCamera does not pad images when stage/image is near border

[iandobbie]

The simulated camera, filterwheel and stage from a large multi channel image does not honor exposure time, meaning the image is returned extremely rapidly. In my findNuclei test script this brakes things as there is not enough time to process between arriving images.

The basic simulated camera does honor exposure time so I suggest we just add this to the camera/stage/filterwheel simulation so that we can add enough to delay to stop it breaking.

[carandraug]

I see that the StageAwareCamera._fetch_data does time.sleep(self._exposure_time). I see that it does that since the beginning (December 2020) which is one year before you opened this commit. Could the issue be somewhere else? How can one reproduce it?

[iandobbie]

I do remember investigating, then fudging it with an arbitrary sleep in my script as I was just trying to get it to work. I will go back and have another look to try and understand what the problem was and fix it properly.

[carandraug]

I took a look at the script uploaded to Zenodo and I couldn't find any sleep on it. Are we talking about the same thing?

[iandobbie]

I now can't get cockpit to load the script, don't know why I cant access the current working dir. I think this might be some of the mac access protection weirdness.

That said I think I night know what happened. If you don't have a light source then the image is returned as fast as possible. You need to activate a light source to get a faked image capture time. I will sort out the loading issue and check how it runs.

[iandobbie]

Ok, I fudged it by putting findNuclei on my path.

On the question of the actual issue, with no light active I get occasional assertion errors, presumably because the data is changing under the opencv code. With a light active and a 100ms exposure time, it runs fine.

[iandobbie]

Sorry I was wrong. Even at high speed it works fine until you reach the image edge. I presume there are issues with partial images returned by the simulated camera. Seems like my original issue is just rubbish so I am closing this.

[carandraug]

There's still an issue here, so I think we can continue discussion about it here.

As you guessed, the issue is the behaviour near the borders. Here's how to reproduce with Microscope only:

from microscope.simulators.stage_aware_camera import simulated_setup_from_image
from queue import Queue
import time

dev = simulated_setup_from_image("opencv2-display.png")
buf = Queue()

dev['camera'].set_client(buf)
dev['camera'].enable()
dev['stage'].enable()  # by default will move to the centre of the stage/image

dev['camera'].trigger()
time.sleep(dev['camera'].get_exposure_time() *2)    # proper size image awaits on buf

dev['stage'].move_to({'x': 0, 'y': 0, 'z': 0})  # move to lower limit
dev['camera'].trigger()  # bad image awaits on buf

good_image = buf.get()
print("good image shape is", good_image.shape) 
bad_image = buf.get()
print("bad image shape is", bad_image.shape)  # will be (0, 0)

[iandobbie]

So how about padding with zero's if the image fully filled. I'll have a look at it.

[iandobbie]

I have a suggested fix, it is a little involved but if any of the camera view is outside the image it pads it with zeros. To reduce additional processing it first checks if it need to do this. If not it just returns the subimage as previously. If this is needed it generates a new sub image filled with zeros, then works out what region of the main image to copy where in the subimage and does that.

The code is in branch stage-aware-cam-fix in my repository.

[carandraug]

Looks good, but would it be simpler if we just pad the image once when we read it? Then we wouldn't have to be constantly checking for out of bounds.

[iandobbie]

Maybe but that takes no account of weirdness like making the camera huge or some other trick that someone might do in testing or as a demonstration.

[carandraug]

That's a fair point. I've tested this but doesn't look like it works in rectangular images.

[iandobbie]

Yeah, that could be a problem, there is a weird x<>y transition that I probably aren't properly taking account of. I'll investigate.

[carandraug]

There were errors in other places. For an array with shape (2000, 100, 3) that's (y,x,z) and not (x,y,z) as the rest of the code expected. I have fixed your fix accordingly, added a test case, and pushed. Closing as fixed.