Camera sometimes 'misses' a shot

[timrowledge]

Several reports of the camera appearing to take a picture but nothing appearing in the costume list.

[timrowledge]

Oh good grief. This is one of those forest-for-trees things; I carefully check for the camera return code, which is 0 if there was an error, or crucially, no new frame from the camera hardware. Buuut… I left in a previous line that simply tells the camera to move data into the bitmap. So in some cases we’ll a) take a shot b) move the image into Scratch memory c) immediately take another shot d) be told there was no new data e) therefore skip making the new costume.

There must be some timing thing that can vary bit (maybe even the lighting conditions affecting the camera processing done?) that allows many cases to let the double-shot work and a few to ‘fail’. Easy enough to fix.

[timrowledge]

Removed the extraneous line from the code; it makes no difference on my machine since I wasn't having a problem, but it does seem a good candidate. Also added ability to broadcast "photo400@400" etc to get a (duh) 400 x 400 pixel image. Any number can be used and whether or not it produces an image is left to the lowest level v4linux2 driver, which means I make no promises about what will work. 32@32 seems to work; 16@20 seems to produce some odd artefacts; but 17@20 won't work at all. Obviously, negative numbers don't work. There's also an oddity about the driver in that if no new pixels have been processed then I don't get a n image. Nothing I can do about that.

[timrowledge]

And if you try very large sizes it may well take long enough process that you grow old and die first. I recommend nothing beyond ~800@600 for practical sizes. I've tested up to something like 3200@2000 and, given patience and coffee, it will work.

[timrowledge]

Argh. It gets sillier. The logic of operation is supposed to be- open camera connection, giving desired x& y size if it opened, loop a while to check if the camera is ready then ask what the actual x & y size are, else fail make a bitmap to suit that size ask the device to read pixels into the bitmap, returning a number frames processed if the number of frames processed was > 0 add a new sprite costume

The camera-is-ready test assumes that the camera answering a non-0 available width/height means it is ready - the plugin code fails if the camera is not open and the prim failure code returns 0. So far as I can see the unix code for 'get frame' doesn't actually return the number of frames since the last call, rather just 0 or 1. Which works ok for the simple case but probably prevents some other functionality.

A reasonably effective solution here is to loop around asking for a frame until the returned number is >0 or we run out of patience. The Scratch CameraDialog effectively does this in its #step method, only updating the cached form when there is 'new data' returned.

[dmoco]

Hi Tim, unless things have changed since I last looked at the code then the desired dimensions are specified only once and that's when attempting to open the camera, which will simply fail if the camera (or V4L2 shim) cannot supply those dimensions. If you are trying to dynamically specify dimensions, especially if opening/closing the camera on each request, then that would ring warning bells for me because of the V4L/2 set-up time. On first opening the camera there can be several black frames which by itself limits how fast you can open/capture/close. Yes, lighting level can significantly effect the capture time so I'd expect more getFrame failures, especially on faster machines (and probably more ENINTR's). Your second capture attempt may itself trigger a new low-level capture, possibly adding further delay. All guesswork because of the number of variables at play starting with the speed of the sensor (slow for the typically CMOS, faster for CCD then leapfrogged by CMOS-B IIRC).

More generally, I tried several things to ensure a frame is always captured and you might see a limited-loop in the C code that is currently set to 1 because, no surprise, any delay there hits Squeak/Scratch (hence no busy-waiting loops elsewhere). The number of frames since last call was never implemented to avoid changing the Squeak-side camera plugin interface.

An alternative would have been to take it all out-of-process and have a "camera server". I implemented that for Squeak/Etoys generally but not for Scratch at that time since everything had to be cross-platform.

[timrowledge]

Hi Derek; I was about to email you on this subject, so excellent timing. It makes sort of sense that there would be some delay between opening a device and getting anything useful out of it. It would be nice if we had a simple call to ask if anything is ready and I suspect that one might be able to extend the primFrameExtent/CameraExtent function to also check... something or other. If the v4l driver has anything suitable to check, which is way beyond my knowledge level.

So far as I can see I have to loop somewhere whilst awaiting useful pixels, whether that is in the generic camera code, your unix-specific code, the Squeak handler, or even the v4l driver. Since I can only easily control the Squeak level stuff today, I'll stick with that.

I imagine another approach might be a much more direct interface to the Pi camera, avoiding v4l altogether, but that is definitely a subject for another day since I have to get this out to release right now.

[dmoco]

To support broadcast requests you could try a small delay on opening the camera (to ensure there is a valid frame) and closing it on a timer instead of immediately (so it is ready for subsequent requests). Alternatively, put it under user control with CameraOn[w@h]/Off messages/blocks? That would mean the frame dim's are fixed for the duration but the scaling functions in John's "Scratch Plugin" could be used for mis-matched requests (I'd be tempted to set reasonably sized default dim's and always scale).

One problem you are likely to hit is when trying to get a current "snapshot" and ending up with a frame that could be several seconds old. That's due to inherent V4L/2 buffers which need exhausting to guarantee you have the latest frame. Most of all this should already be implemented for OLPC Etoys and on SqueakSource under my account as "WebCamMorph" and "Webcam Access Interface as used in MIT Scratch".

Btw, I always favoured moving video, audio and associated processing out of Squeak and handling it all via dynamically created GStreamer pipelines :-)

[timrowledge]

Fixed in Scratch-tpr-336/7/8/9 for May 10 2016 release