Background Blur and Replacement freezing my video

[YumTheDeathGod]

Describe the bug Tried to implement the background blur functionality in react native project. So I have integrated all the necessary binaries into the project like amazon-chime-sdk, amazon-chime-sdk-media, amazon-chime-sdk-machine-learning. I have created a function to implement the background blur on click of button inside NativeMobileSDKBridge. Like this: @ReactMethod fun setBackgroundBlur(enabled: Boolean) { logger.info(TAG, "this is background blur functionality $enabled")

    if(enabled) {
        val surfaceTextureCaptureSourceFactory = DefaultSurfaceTextureCaptureSourceFactory(logger, eglCoreFactory)
        val cameraCaptureSource = DefaultCameraCaptureSource(reactApplicationContext, logger, surfaceTextureCaptureSourceFactory)
        val backgroundBlurVideoFrameProcessor = BackgroundBlurVideoFrameProcessor(
            ConsoleLogger(LogLevel.DEBUG),
            eglCoreFactory,
            reactApplicationContext,
            BackgroundBlurConfiguration(7.5f)
        )
        cameraCaptureSource.addVideoSink(sink = backgroundBlurVideoFrameProcessor)
        meetingSession?.audioVideo?.startLocalVideo(backgroundBlurVideoFrameProcessor)
    } else {
        meetingSession?.audioVideo?.startLocalVideo()
    }
}

Above function is imported inside bridge.js file like this: setBackgroundBlur: NativeModules.NativeMobileSDKBridge.setBackgroundBlur

And use this function on js: <BlurButton onPress={() => { NativeFunction.setBackgroundBlur(!isBlurEnabled); // Value is hard coded for demo purpose }} />

To Reproduce Steps to reproduce the behavior:

1. Launch meeting
2. Join with other participants. (atleast one)
3. Tap on button to activate blur effect for background
4. If you observe on participants side whenever you turn on the blur frame processing videos gets freeze and when you turn it off video will get back to normal.

Expected behavior It should have applied blur effect on background but its freezing the video

Logs If applicable, add logs from logcat to help explain your problem.

Screenshots If applicable, add screenshots to help explain your problem.

Test environment Info (please complete the following information):

• Device: Lenovo TB-8705F
• OS: Android 10
• Version amazon-chime-sdk: 0.17.0
• Version amazon-chime-sdk-media: : 0.17.0
• Version amazon-chime-sdk-media-machine-learning : 0.1.0

Hey folks first of all Is my implementation is correct if yes then why my video getting freeze on remote side as well as on local side also? and If it not a correct implementation then please help me to do so.

Logs for debugging

2022-05-26 10:50:00.815 11454-11725/com.app I/GED: ged_boost_gpu_freq, level 100, eOrigin 2, final_idx 2, oppidx_max 2, oppidx_min 0 2022-05-26 10:50:00.885 11454-11725/com.app I/GED: ged_boost_gpu_freq, level 100, eOrigin 2, final_idx 2, oppidx_max 2, oppidx_min 0 2022-05-26 10:50:00.941 11454-11652/com.appc I/ChimeReactNativeSDKDemoManager: this is background blur functionality true 2022-05-26 10:50:00.954 11454-11725/comapp I/GED: ged_boost_gpu_freq, level 100, eOrigin 2, final_idx 2, oppidx_max 2, oppidx_min 0 2022-05-26 10:50:00.960 11454-11652/com.app I/DefaultCameraCaptureSource: Stopping camera capture source 2022-05-26 10:50:00.969 11454-12496/com.app I/BackgroundBlurVideoFrameProcessor: Created BackgroundBlurVideoFrameProcessor 2022-05-26 10:50:01.093 11454-11708/com.app |/BufferQueueProducer: SurfaceTexture-24-11454-4 disconnect(P): api 4 2022-05-26 10:50:01.100 11454-12381/com.app I/SurfaceTextureCaptureSource: Setting on frame available listener to null 2022-05-26 10:50:01.100 11454-12381/com.app I/SurfaceTextureCaptureSource: Releasing surface texture capture source 2022-05-26 10:50:01.100 11454-11708/com.app I/DefaultCameraCaptureSource: Camera device closed for ID 1 2022-05-26 10:50:01.101 11454-11652/com.app I/DefaultVideoClientController: Setting external video source in media client to custom source 2022-05-26 10:50:01.105 11454-12381/com.app I/SurfaceTextureCaptureSource: Finished releasing surface texture capture source

Thanks

[georgezy-amzn]

Thank you for opening the issue. We’ll look into more.

[YumTheDeathGod]

I found the solution already. I just need to start the cameraCaptureSource whenever we want that blur effect. Thank you for your time @georgezy-amzn .

But still I need your help. Please assist. whenever I tried to add that blur effect then it is being applied correctly but when I observe the mobile video from web chime demo (on desktop) with the same room. I get distorted image of attendee (from mobile side) and the blurred background.

I am not able to get why is there is pixelated image of user after applying blurry effect. Do I have to change the frame rates for it? if yes then how should i do it?

[richhx]

Thanks for pointing this out. We're seeing the pixelated image from mobile device on the desktop side too. At first look, it appears to be a bug with how the image is downscaled and then upscaled during the processing. We're continuing to look into this and will keep you posted.

[richhx]

Resolving this as it should be merged now.

[AlkanV]

hello @YumTheDeathGod would you mind to share your solution on this here please? (especially for ios)

[G0ldsam]

@YumTheDeathGod it would be really helpful if you could provide the code