Camera View Over Image

[B1HD]

Hey Pedro, Your library is great and has really helped me with creating an app to livestream to Facebook. My question is if it's possible to somehow stream a scaled-down RTMP camera overlayed on an Image. The idea is that an image of a product would be full screen but the presenter still needs to be visible. Kind of like how on Zoom or FaceTime you have a smaller preview of your own camera overlayed with who you're chatting with. If you have any suggestions or ideas I'd really appreciate it! Thanks!

[pedroSG94]

Hello,

Do you mean with the product the stream device screen and add camera to the stream? In that case, yes you can do it. If you want open 2 cameras (front and back) this depend of the device model but most of device can't do it (hardware limitation)

To do the first case you can use SurfaceFilterRender like here: https://github.com/pedroSG94/RootEncoder/blob/master/app/src/main/java/com/pedro/streamer/openglexample/OpenGlRtmpActivity.java#L297 So using RtmpDisplay (this apply to RtspDisplay and SrtDisplay) you can add a the filter and then instead of use MediaPlayer to render the SurfaceTexture, you can use Camera2ApiManager or Camera1ApiManager:

                  Camera2ApiManager camera = new Camera2ApiManager(OpenGlRtmpActivity.this);
                  camera.prepareCamera(surfaceTexture, 1280, 720, 30);
                  camera.openCameraFront();
                  //this line must be called when you don't need camera anymore (like when you remove filter or activity is destroyed)
                  //camera.closeCamera();

Remember modify setScale and setPosition of SurfaceFiilterRender as you need to keep aspect ratio with the size that you want and the position.

[B1HD]

Thanks so much for the reply, that really helped me to get it working. The issue that I'm running into now is I'm transitioning to this Image with the camera overlayed on it from just the default RTMPCamera1 view and so when I add the surface texture as a filter everything appears correct and scaled properly but the camera preview is just black. I believe this is because the camera is still being used behind the filter? I can get the camera preview to show up if I switch the camera to the back camera. I'm guessing this is because when switchCamera is called it releases the camera it was using. My question is, is there anyway to release the camera that RTMPCamera1 is using without ending the preview or stream? Thanks so much in advance for any help you can provide.

[pedroSG94]

Hello,

Let me know if I understand you correctly. You have a stream with camera1 as video and then you want change to a stream with screen as video and using camera1 as filter. Right?

In that case, maybe use RtmpStream is the best way. You can change video and audio sources on realtime. And that sources are releases when you replace it for other source. You have a code example here that support change video and audio sources: https://github.com/pedroSG94/RootEncoder/tree/master/app/src/main/java/com/pedro/streamer/rotation In this example you can check that, the way to use RtmpStream is a bit different that others like RtmpCamera1. You have 3 main differences:

• You only need call prepareVideo and prepareAudio one time before start preview, stream or record so you can stop and start it multiples times after a success prepare. If you want change parameters you only need stop all and call prepare again.
• You can add a preview with startPreview but in this case the view is set in startPreview method allowing you to use a normal SurfaceView or TextureView instead of use OpenGlView.
• You need call release method when you finish all to make sure all is correctly released (normally you can use it in onDestroy method of an activity).

The way to add a filter is exactly the same that with RtmpCamera1

[BradenEgan]

Sorry for the delayed response, I appreciate your help. I've made some changes to the code and I'm using RTMPCamera2 now to handle everything. Is there a way to release the camera while using the OpenGLView without stopping the stream? What I'm trying to accomplish is using the toggleImageOverlay function to display an image with the camera view overlayed on the image. Here is a screenshot, I want the black square to display the camera feed but I just can't seem to get it. The stream works just fine and I can stream just the camera view with no issues. It's when I trigger the image overlay filter that I run into issues. I'm sure I'm missing something and I appreciate any help or suggestions you might have for me.

Screenshot of stream

Toggle Image Overlay Code...

private void toggleImageOverlay() {
        String cameraId;
        if(rtmpCamera2.getCameraFacing() == CameraHelper.Facing.FRONT){
            cameraId = camera2ApiManager.getCameraIdForFacing(CameraHelper.Facing.FRONT);
            Log.d(TAG, "toggleImageOverlay: Front facing cameraId = " + cameraId);
        } else {
            cameraId = camera2ApiManager.getCameraIdForFacing(CameraHelper.Facing.BACK);
            Log.d(TAG, "toggleImageOverlay: Back facing cameraId = " + cameraId);
        }
        //camera2ApiManager = new Camera2ApiManager(SecondActivity.this);
        camera2ApiManager.closeCamera();
                try {
                    if (imageList.isEmpty()) {
                        Toast.makeText(SecondActivity.this, "Please enter a SKU to load images!", Toast.LENGTH_SHORT).show();
                        return;
                    }

                    if (isImageOverlayVisible) {
                        // Hide the image overlay
                        if (imageObjectFilterRender != null) {
                            rtmpCamera2.getGlInterface().removeFilter(imageObjectFilterRender);
                            Log.d(TAG, "imageObjectFilterRender is not null");

                        }
                        if (surfaceFilterRender != null) {
                            Log.d(TAG, "surfaceFilterRender is not null");
                            camera2ApiManager.closeCamera();
                            rtmpCamera2.getGlInterface().removeFilter(surfaceFilterRender);
                            surfaceFilterRender = null;

                        }
                        isImageOverlayVisible = false;
                    } else {
                        Log.d(TAG, "Else statement was hit");
                        camera2ApiManager.getCameraCharacteristics();
                        imageObjectFilterRender = new ImageObjectFilterRender();
                        Bitmap freshBitmap = imageList.get(currentImageIndex); // Ensure this Bitmap is valid
                        imageObjectFilterRender.setImage(freshBitmap);
                        imageObjectFilterRender.setPosition(TranslateTo.BOTTOM_RIGHT);
                        rtmpCamera2.getGlInterface().addFilter(imageObjectFilterRender);

                        camera2ApiManager.closeCamera();

                        SurfaceFilterRender surfaceFilterRender =
                                new SurfaceFilterRender(new SurfaceFilterRender.SurfaceReadyCallback(){
                                    @Override
                                    public void surfaceReady(SurfaceTexture surfaceTexture) {
                                       camera2ApiManager.prepareCamera(surfaceTexture, 1280, 720, 30);
                                       camera2ApiManager.openCameraFront();

                                    }
                                });
                                rtmpCamera2.getGlInterface().addFilter(surfaceFilterRender);
                                spriteGestureController.setBaseObjectFilterRender(surfaceFilterRender);

                            // Set the scale and position according to your needs
                            surfaceFilterRender.setScale(25f, 25f); // Example scale for PiP
                            surfaceFilterRender.setPosition(TranslateTo.TOP_RIGHT); // Example position for Pi
                            surfaceFilterRender.setRotation(270);// P
                        }
                        isImageOverlayVisible = true;

                        //rtmpCamera2.stopPreview();

                } catch (Exception e) {
                    Toast.makeText(SecondActivity.this, "Error toggling image overlay", Toast.LENGTH_SHORT).show();
                }

    }

Catlog of the issue...

   2024-03-19 00:42:07.143  3227-3227  ImageStreamObject       com.blingiton.biolive                I  finish load image
2024-03-19 00:42:07.265  3227-3227  Camera2ApiManager       com.blingiton.biolive                I  optimal resolution set to: 1280x720
2024-03-19 00:42:07.274   988-4120  CameraService           cameraserver                         I  CameraService::connect call (PID -1 "com.blingiton.biolive", camera ID 1) for HAL version default and Camera API version 2
2024-03-19 00:42:07.485   988-4120  CameraService           cameraserver                         I  finishCameraOps: Finish camera ops, package name = com.blingiton.biolive, client UID = 10306
2024-03-19 00:42:07.487   988-4120  CameraFlashlight        cameraserver                         I  deviceClosed: device 1 is closed(package=com.blingiton.biolive)
2024-03-19 00:42:07.487   988-4120  CameraFlashlight        cameraserver                         I  prepareDeviceOpen: prepare for device open(cameraId=1, package=com.blingiton.biolive)
2024-03-19 00:42:07.488   988-4120  Camera2ClientBase       cameraserver                         I  Camera 1: Opened. Client: com.blingiton.biolive (PID 3227, UID 10306)
2024-03-19 00:42:07.489  3227-3708  CameraCaptureSession    com.blingiton.biolive                E  Session 0: Exception while stopping repeating: 
                                                                                                    android.hardware.camera2.CameraAccessException: CAMERA_DISCONNECTED (2): cancelRequest:458: Camera device no longer alive
                                                                                                        at android.hardware.camera2.CameraManager.throwAsPublicException(CameraManager.java:815)
                                                                                                        at android.hardware.camera2.impl.ICameraDeviceUserWrapper.cancelRequest(ICameraDeviceUserWrapper.java:95)
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl.stopRepeating(CameraDeviceImpl.java:1134)
                                                                                                        at android.hardware.camera2.impl.CameraCaptureSessionImpl.close(CameraCaptureSessionImpl.java:526)
                                                                                                        at android.hardware.camera2.impl.CameraCaptureSessionImpl$2.onDisconnected(CameraCaptureSessionImpl.java:737)
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl$7.run(CameraDeviceImpl.java:242)
                                                                                                        at android.os.Handler.handleCallback(Handler.java:873)
                                                                                                        at android.os.Handler.dispatchMessage(Handler.java:99)
                                                                                                        at android.os.Looper.loop(Looper.java:214)
                                                                                                        at android.os.HandlerThread.run(HandlerThread.java:65)
                                                                                                    Caused by: android.os.ServiceSpecificException: cancelRequest:458: Camera device no longer alive (code 4)
                                                                                                        at android.os.Parcel.createException(Parcel.java:1980)
                                                                                                        at android.os.Parcel.readException(Parcel.java:1934)
                                                                                                        at android.os.Parcel.readException(Parcel.java:1884)
                                                                                                        at android.hardware.camera2.ICameraDeviceUser$Stub$Proxy.cancelRequest(ICameraDeviceUser.java:402)
                                                                                                        at android.hardware.camera2.impl.ICameraDeviceUserWrapper.cancelRequest(ICameraDeviceUserWrapper.java:93)
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl.stopRepeating(CameraDeviceImpl.java:1134) 
                                                                                                        at android.hardware.camera2.impl.CameraCaptureSessionImpl.close(CameraCaptureSessionImpl.java:526) 
                                                                                                        at android.hardware.camera2.impl.CameraCaptureSessionImpl$2.onDisconnected(CameraCaptureSessionImpl.java:737) 
                                                                                                        at android.hardware.camera2.impl.CameraDeviceImpl$7.run(CameraDeviceImpl.java:242) 
                                                                                                        at android.os.Handler.handleCallback(Handler.java:873) 
                                                                                                        at android.os.Handler.dispatchMessage(Handler.java:99) 
                                                                                                        at android.os.Looper.loop(Looper.java:214) 
                                                                                                        at android.os.HandlerThread.run(HandlerThread.java:65) 

[pedroSG94]

Hello,

Currently the way to want use the library require create a custom filter. Also, you can use RtmpStream instead of RtmpCamera2 and add a custom VideoSource. I created an example here: https://github.com/pedroSG94/RootEncoder/pull/1440

Using this branch you can open Rotation example and select bitmap as VideoSource. This allow you use a Bitmap as the main source to render and then you can add a SurfaceFilterRender using camera2apimanager as you are doing. If you want change to normal camera you can stop and remove that SurfaceFilterRender and change the video source back to Camera2Source