Request: Access to camera when using Vuforia

[khoyt5]

Last year our team experimented with using the camera and taking a picture of the beacon and then process the pixels to determine which button to push based on Red color or Blue color.

The Vuforia capability that is now the SDK looks promising for use but I could see using the raw pixel data in parallel. Is it possible to update the SDK calls to Vuforia to take a current snapshot and provide the pixel data to the running OpMode?

[rgatkinson]

This is a very reasonable and understandable request. Unfortunately, it's not technically possible to achieve, at least with the currently available version of Vuforia.

[rgatkinson]

Update: it turns out that this may be possible after all (thanks FTC 3491!). Investigation is proceeding...

[rgatkinson]

There are two solutions to this. First, a hack that will work with the existing release (but may not be supported long term), and second, an architected solution that should go out in the next release.

Both approaches ultimately provide access to a series of underlying Vuforia Frame objects from which camera image data can be accessed.

The first approach is to implement the following:

    /** {@link CloseableFrame} exposes a close() method so that we can proactively
     * reduce memory pressure when we're done with a Frame */
    static class CloseableFrame extends Frame {
        public CloseableFrame(Frame other) { // clone the frame so we can be useful beyond callback
            super(other);
        }
        public void close() {
            super.delete();
        }
    }

    // A hack that works around lack of access in v2.2 to camera image data when Vuforia is running
    // Note: this may or may not be supported in future releases.
    class VuforiaLocalizerImplSubclass extends VuforiaLocalizerImpl {

        class VuforiaCallbackSubclass extends VuforiaLocalizerImpl.VuforiaCallback {

            @Override public synchronized void Vuforia_onUpdate(State state) {
                super.Vuforia_onUpdate(state);
                // We wish to accomplish two things: (a) get a clone of the Frame so we can use
                // it beyond the callback, and (b) get a variant that will allow us to proactively
                // reduce memory pressure rather than relying on the garbage collector (which here
                // has been observed to interact poorly with the image data which is allocated on a
                // non-garbage-collected heap). Note that both of this concerns are independent of
                // how the Frame is obtained in the first place.
                CloseableFrame frame = new CloseableFrame(state.getFrame());
                RobotLog.vv(TAG, "received Vuforia frame#=%d", frame.getIndex());
                frame.close();
                }
            }

        VuforiaLocalizerImplSubclass(VuforiaLocalizer.Parameters parameters) {
            super(parameters);
            stopAR();
            clearGlSurface();

            this.vuforiaCallback = new VuforiaCallbackSubclass();
            startAR();

            // Optional: set the pixel format(s) that you want to have in the callback
            Vuforia.setFrameFormat(PIXEL_FORMAT.RGB565, true);
        }

        void clearGlSurface() {
            if (this.glSurfaceParent != null) {
                appUtil.synchronousRunOnUiThread(new Runnable() {
                    @Override public void run() {
                        glSurfaceParent.removeAllViews();
                        glSurfaceParent.getOverlay().clear();
                        glSurface = null;
                    }
                });
            }
        }
    }

Modify the callback to do whatever it is you wish to do instead of just writing to the log. Don't do lengthy operations in the callback. Instantiate this class (with new) rather than calling ClassFactory.createVuforiaLocalizer().

The second solution involves the following addition to the VuforiaLocalizer interface:

    /**
     * (Advanced) Returns a queue into which, if requested, Vuforia {@link Frame}s are placed
     * as they become available. This provides a means by which camera image data can be accessed
     * even while the Vuforia engine is running.
     *
     * <p>While the Vuforia engine is running, it takes exclusive ownership of the camera it is
     * using. This impedes the ability to also use that camera image data in alternative visual
     * processing algorithms. However, periodically (at a rate of tens of Hz), the Vuforia engine
     * makes available {@link Frame}s containing snapshots from the camera, and through which
     * camera image data can be retrieved. These can optionally be retrieved through the frame
     * queue.</p>
     *
     * <p>To access these {@link Frame}s, call {@link #setFrameQueueCapacity(int)} to enable the
     * frame queue. Once enabled, the frame queue can be accessed using {@link #getFrameQueue()}
     * and the methods thereon used to access {@link Frame}s as they become available.</p>
     *
     * <p>When {@link #setFrameQueueCapacity(int)} is called, any frame queue returned previously by
     * {@link #getFrameQueue()} becomes invalid and must be re-fetched.</p>
     *
     * @return a queue through which Vuforia {@link Frame}s may be retrieved.
     *
     * @see #setFrameQueueCapacity(int)
     * @see <a href="https://library.vuforia.com/sites/default/api/java/classcom_1_1vuforia_1_1Frame.html">Frame (Java)</a>
     * @see <a href="https://library.vuforia.com/sites/default/api/cpp/classVuforia_1_1Frame.html">Frame (C++)</a>
     * @see <a href="https://developer.android.com/reference/java/util/concurrent/BlockingQueue.html">BlockingQueue</a>
     */
    BlockingQueue<CloseableFrame> getFrameQueue();

    /**
     * Sets the maximum number of {@link Frame}s that will simultaneously be stored in the
     * frame queue. If the queue is full and new {@link Frame}s become available, older frames
     * will be discarded. The frame queue initially has a capacity of zero.
     *
     * <p>Note that calling this method invalidates any frame queue retreived previously
     * through {@link #getFrameQueue()}.</p>
     *
     * @param capacity the maximum number of items that may be stored in the frame queue
     * @see #getFrameQueue()
     * @see #getFrameQueueCapacity()
     */
    void setFrameQueueCapacity(int capacity);

    /**
     * Returns the current capacity of the frame queue.
     * @return the current capacity of the frame queue.
     * @see #setFrameQueueCapacity(int)
     * @see #getFrameQueue()
     */
    int getFrameQueueCapacity();

Note that in both cases it will be useful to make use of CloseableFrame for the following reason. The actual underlying image data is not allocated in the garbage collected Java heap but rather in manually allocated memory managed by Vuforia. The Frame is just a wrapper containing a handle to the non-GC-image-data and whose lifetime determines how long the image data should stick around: the finializer in Frame cleans it up. The trouble is that the two heaps don't interact as well as perhaps might be ideal: crashes have been observed that are almost certainly due to this interaction. CloseableFrame exposes a close() method that can be used to proactively release the underlying image data if it's known to no longer be needed by an application.

[gebailey]

I see in the release notes for v2.30 it says: "Fix deadlock / make camera data available while Vuforia is running." Does that fix basically allow us to use the second solution as listed above? I'm trying to square this with suggestions given by FIXIT3491 in their YouTube video explaining how they're able to capture the frames from Vuforia.

[rgatkinson]

The above described architected solution is now implemented in VuforiaLocalizer.

[cmacfarl]

Per @rgatkinson's comment, closing.