Feature request: use FrameProcessors to do pre-processing (e.g. live effects)

[morejump]

it really difficult to start without sample!!

[morejump]

help me, plz

[morejump]

now i want to get frame to process in opencv c++, i have to get frame and use JNI then. But i dont now how to get instance of frame

[natario1]

We provide a java callback, then it's up to you. The only docs we can provide are already here https://github.com/natario1/CameraView#frame-processing . I know nothing about JNI but I am sure you can find a lot of info.

[aaronjrenfroe]

Is it possible to process the frames before they get displayed?

[natario1]

@aaronjrenfroe no, sorry! Not easy.

[morejump]

@natario1 I dont know how to use instance of frame before returning it to the preview. Can u give more detail about that, plz!!!

[Hrishi1999]

Well, I am not natario but ;) I've successfully implemented a frame processor in my app. You can ask me.

[natario1]

@morejump it is not supported. you can only do post-processing (barcodes, face detection, ...) @Hrishi1999 if your solution is open source somewhere we can take a look!

I'll change this issue into a feature request

[morejump]

@Hrishi1999 come on, can u tell me, how can it do it, plz :(

[Hrishi1999]

I am sorry if I misunderstood. Or maybe a few comments have been edited. What I meant is I can use the frames to do something, like maybe a realtime barcode scanner, or OCR?? Please explain me more, I implemented a frame processor, maybe I can help.

[natario1]

What he and some others are asking for, is to process the frames before they are displayed, so the preview displays the edited frame buffer instead of the real image. This is not possible with CameraView.

[abhisheksoni27]

@natario1 But if I want to Post-process the frame, how can I display the edited frame afterwards?

This is what I am currently doing:

cameraView.addFrameProcessor(new FrameProcessor() {
    @Override
    @WorkerThread
    public void process(Frame frame) {
        byte[] data = frame.getData();
        Size size = frame.getSize();

        // Process `data` using OpenCV
        // Now, how should I set set the changed data?
    }
});

As you know, the Frame class doesn't have setters. Does this mean a LivePostProcessing pipeline can't be implemented?

[JeremeLau]

@natario1 But if I want to Post-process the frame, how can I display the edited frame afterwards?

This is what I am currently doing:

cameraView.addFrameProcessor(new FrameProcessor() {
    @Override
    @WorkerThread
    public void process(Frame frame) {
        byte[] data = frame.getData();
        Size size = frame.getSize();

        // Process `data` using OpenCV
        // Now, how should I set set the changed data?
    }
});

As you know, the Frame class doesn't have setters. Does this mean a LivePostProcessing pipeline can't be implemented?

I have the same request, pls someone help

[JeremeLau]

@natario1 But if I want to Post-process the frame, how can I display the edited frame afterwards?

This is what I am currently doing:

cameraView.addFrameProcessor(new FrameProcessor() {
    @Override
    @WorkerThread
    public void process(Frame frame) {
        byte[] data = frame.getData();
        Size size = frame.getSize();

        // Process `data` using OpenCV
        // Now, how should I set set the changed data?
    }
});

As you know, the Frame class doesn't have setters. Does this mean a LivePostProcessing pipeline can't be implemented?

I have made my request done. If ur request is the same as mine. My request is 2 have the faces detected in scene and label the faces with square frames. Devious solution： 1、put a ‘SurfaceView’ just On Top of ‘CameraView’ 2、set it TRANSPARENT 3、get SurfaceHolder 4、here we have the method 'lockCanvas', do what u have 2 do(DrawRect etc),call 'unlockCanvasAndPost(canvas)', done. If ur request is more complex, then good luck

[natario1]

Please refer to #176 for updates about this

[aboozaid]

@natario1 But if I want to Post-process the frame, how can I display the edited frame afterwards? This is what I am currently doing:

cameraView.addFrameProcessor(new FrameProcessor() {
    @Override
    @WorkerThread
    public void process(Frame frame) {
        byte[] data = frame.getData();
        Size size = frame.getSize();

        // Process `data` using OpenCV
        // Now, how should I set set the changed data?
    }
});

As you know, the Frame class doesn't have setters. Does this mean a LivePostProcessing pipeline can't be implemented?

I have made my request done. If ur request is the same as mine. My request is 2 have the faces detected in scene and label the faces with square frames. Devious solution： 1、put a ‘SurfaceView’ just On Top of ‘CameraView’ 2、set it TRANSPARENT 3、get SurfaceHolder 4、here we have the method 'lockCanvas', do what u have 2 do(DrawRect etc),call 'unlockCanvasAndPost(canvas)', done. If ur request is more complex, then good luck

This will work only if a preview frame as same as a preview surface because when you draw the canvas it will be located at a different place as the preview surface is larger. Is there anyone who has a workaround?

I think the library already set functions to set the preview size and camera frame size as well but I'm so concerned with performance.