Frame.size is null

[coffeenahc]

I'm using 'com.otaliastudios:cameraview:2.0.0-beta04' and trying to get the frame's size and width by calling : frame.getSize().getWidth()/getHeight() but it throws a NPE. This is done on the onProcess(Frame frame) method.

[janlukes]

It's also happening to me. Mostly on the start of capture.

[natario1]

Can you post your onProcess method and CameraView logs?

@janlukes what does "Mostly on the start of capture" mean?

[natario1]

For reference #367

[coffeenahc]

[hanspeide]

This also happens to other fields in the Frame class, e.g. data. Seeing this on a Samsung Galaxy S7, but not seeing it on a Pixel 2.

[hanspeide]

Update: When the S7 is laying face down on a table the system logs output 2019-03-30 20:03:25.947 4745-30202/? D/SensorListener: Sensor(8) valid data is not coming yet!, and immediately after I get the Frame with fields set to null. In essence I don't think it's a bug that Frame has null fields as they're merely reflecting what's coming from the camera, but I do think it's a bug that these fields are annotated with @NonNull.

[natario1]

@hanspeide can you reproduce this with the demo app? What are the steps to reproduce?

We don't want to dispatch these updates at all, so the bug is dispatching in the first place... In which conditions does this happen? Is it the first frame(s)?

[hanspeide]

I'm trying to make a reproducible case now, and suspect that I was actually on the wrong track in previous post. Frames that do come in always seem to have data and size, but when they are run through e.g. FirebaseVisionBarcodeDetector.detectInImage(..) the frames will most of the time have data and size set to null when onSuccess(..) is called.

I believe the solution to this should be to call frame.freeze() before detectInImage(..), and then subsequently frame.release() as last line of onSuccess() or onFailure, but when I do I get an OOM exception in frame.freeze(). Any pointers?

[natario1]

@hanspeide this is expected and documented.

One solution is to freeze(). The other (better) is to do detection synchronously, which means do not return onFrame until the firebase task resolves. (You can use for example a CountDownLatch)

[hanspeide]

Yes, come to think about it it makes sense. Where can I find the documentation for this?? I'll look into CountDownLatch.

By the way - freeze() does a reference copy of size. Shouldn't it instead create a new Size object, reusing the width and hight from the original object?

Edit: I see that .freeze() is documented in the code

[natario1]

See https://natario1.github.io/CameraView/docs/frame-processing.html .

Sizes are immutable so there is no risk that the original Size would change, I think it's ok as is.

Instead of CountDownLatch, now that I think of, you can simply use Tasks.await().

[GuanacoDevs]

Personally I'm using Firebase MLKit to process Bingo Cards using OCR and QR codes. What I do is to have a set flags to scan and counters to avoid the user staring a the target without getting a positive result. In the case of Bingo Cards is kind of tricky because most time OCR won't give a good reliable result. But with Codes "CameraView" is faster than the speed of light (@natario1 I owe you a beer, heck I owe a case once I make something) So I have put MLKit logic into a runnable, whenever I get a positive result I clear the flag for the Frame Processor to stop, while MLKit is working the frames are not getting processed.

   if (scanningState == Keys.SCANNING_STARTED && isScanning) {
                scanningState = Keys.SCANNING_STOPPED;
                if (frame_count >= Keys.FAILED_READINGS) {
                    return;
                }
                ProcessBitmap processBitmap = new ProcessBitmap(bytes, onTaskFinished, freeSpaceRect, pictureFrameRect, frame.getRotation(), displaySize, overlayLeftRect, overlayRightRect);
            }

The processBitmap has an interface that tells the activity/fragment to get another frame to keep processing or to stop, depending on your needs that piece of code might mean nothing, but the idea is to process frames once MLKit has succeeded or failed. And in the case of Keys.FAILED_READINGS I just inform the user that the reading has failed and to try again or file a report(Ligthning conditions might have an effect). Whenever I get a positive result I stop the Frame Processor and move to the next step, or ask to process another frame until the counter has reached its maximum. EDIT: If you are doing realtime feedback you still have to wait for MLKit to be done.

[hanspeide]

@natario1 I'm not sure I get how .release() is supposed to be used. Using release in the below code will lead to an OOM exception in val frame = inputFrame.freeze(), but if I comment out the .release() part I get no OOM, and frame looks to be GC'd correctly.

val frame = inputFrame.freeze()

val firebaseVisionImage = FirebaseVisionImage.fromByteArray(frame.data, getMetadataForFrame(frame))
barcodeDetector.detectInImage(firebaseVisionImage)
    .addOnSuccessListener { firebaseVisionBarcodes ->
        Log.d(this.javaClass.name, "Frame data null? ${frame.data == null} Frame size null? ${frame.size == null}")
    }.addOnCompleteListener {
        frame.release()
    }

Regarding "do not return onFrame until the firebase task resolves.". Are you referring to the process() method?

[GuanacoDevs]

@hanspeide sorry if i missplaced my comment does this helps?

[natario1]

Yes, I meant the process() method. You can do as follows and do not freeze at all:

// During process()
Tasks.await(detector.detectInImage(inputFrame).onSuccessTask {
    // Extra code
})

Can you post the OOM trace in a new bug report? Thanks! @hanspeide

[natario1]

@GuanacoDevs

but the idea is to process frames once MLKit has succeeded or failed.

You could also use Tasks.await() or a CountDownLatch instead of your SCANNING flag, so the process() method returns when MLKit returns .

CameraView already does this kind of throttling, it skips updates if the processor is busy. And the flag it uses to do so, instead of your boolean, is whether the process() method is running or not.

Your solution is valid as well, just a bit less efficient (because CameraView will keep sending updates that you don't need).

[jgranleese953]

I've also been having issues with this and also the frame processor stops triggering the "process(Frame frame)" callback at all after the first 5 or so frames.

[natario1]

@jgranleese953 you can open a new issue with that and fill the bug information. Your process() implementation will also help

[jgranleese953]

https://github.com/natario1/CameraView/issues/430#issuecomment-478750985

[natario1]

I am closing this as it is not a bug. New version will give a better error message instead of throwing NPE, however the point is that you can not access a Frame after process() method has returned. [Unless you make a copy with freeze(), but this has a performance impact].

The OP screenshot is accessing the frame asynchronously during onSuccess() which is why Size becomes null. I encourage you to take a look at the suggestions I have given in this thread.

There was a bug about freeze/release described by @hanspeide , but that was fixed in #431 . Thank you!