Chime video tiles getting freeze and disappear

[dharam-step]

Video Conferencing Bug

Description: We are encountering issues with our video conferencing implementation using AWS Chime. The conferencing system involves two roles: Instructor and Student. When the instructor turns on their video, the video sometimes gets stuck on the student's screen and vice-versa. Additionally, when we paginate the video tiles using the React Native Swiper component, the video tiles sometimes disappear randomly.

Steps to Reproduce:

1. Open the video conferencing application.
2. Join a meeting as an Instructor or Student.
3. Turn on video for either the Instructor or Student.
4. Observe the video freezing on the other participant's screen.
5. Attempt to paginate through the video tiles using the React Native Swiper component.
6. Notice that video tiles disappear randomly.

Expected Behavior:

• Video feeds should not freeze or get stuck on other participant's screen.
• Paginating through video tiles should display consistent behavior without any disappearing tiles.

Actual Behavior:

• Video feeds freeze on the other participant's screen intermittently.
• Paginating through video tiles sometimes leads to tiles disappearing randomly.

Component to render students wth swiper:

import { useState, useEffect } from 'react';
import { View, Text, Image, StyleSheet, } from 'react-native';
import { AppColors, FontSize, FontWeight } from '../../Themes';
import Swiper from 'react-native-swiper';
import StudentMenuSection from './MenuSection/studentMenuSection';
import { useSelector } from 'react-redux';
import SettingModal from './SettingModal';
import MemoizedFirstPageComponent from './SwipePageComponent/FirstPageComponent';
import MemoizedSecondPageComponent from './SwipePageComponent/SecondPageComponent';
import StudentSideUsersList from '../UsersList/studentSideUserList';

// Main render item
const StudentSideVideoStream = ({ videoTiles, joinersData, handleCamera, handleEndMeeting, selfVideoEnabled, handleMuteButton, handleSettingVideoIcon, handleSettingMuteIcon, handleSelectCameraItem, desfaultSelecetdCamera, listAudioDevice, handleOnPressMicItem, handleSettingButton, selectedMicrophone, selectedSpeaker, formattedTime, closeSettingModal, handleMenuUserIcon, onTap }) => {
    const [menuVisible, setMenuVisible] = useState(true);
    const [settingVisible, setSettingVisible] = useState(false);
    const [pageNumber, setPageNumber] = useState(0);
    const { userData } = useSelector((state) => state.userInfo);
    const [isLocalCameraOn, setIsLocalCameraOn] = useState()
    const [isLocalMicOn, setIsLocalMicOn] = useState()
    const [isUserListVisible, setIsUserListVisible] = useState(false)

    let length = videoTiles.length

    const checkForInstructorData = () => {
        const matchedElements = videoTiles.map(videoTile => {
            const matchingJoiner = joinersData.find(joiner => joiner.attendeeId === videoTile.attendeeId && joiner.isInstructor == true);
            return matchingJoiner ? { ...videoTile, ...matchingJoiner } : undefined;
        }).filter(matchedElement => matchedElement !== undefined);

        return matchedElements
    }

    const checkForSelfData = () => {
        const matchedElements = videoTiles.map(videoTile => {
            const matchingJoiner = joinersData.find(joiner => joiner.attendeeId === videoTile.attendeeId && (joiner.contactID == userData.contactID && joiner.isInstructor == false));
            return matchingJoiner ? { ...videoTile, ...matchingJoiner } : undefined;
        }).filter(matchedElement => matchedElement !== undefined);

        return matchedElements
    }

    const checkForOtherData = () => {
        const matchedElements = videoTiles.map(videoTile => {
            const matchingJoiner = joinersData.find(joiner => joiner.attendeeId === videoTile.attendeeId && (joiner.contactID !== userData.contactID && joiner.isInstructor == false));
            return matchingJoiner ? { ...videoTile, ...matchingJoiner } : undefined;
        }).filter(matchedElement => matchedElement !== undefined);
        return matchedElements
    }

    const instructorFilterData = checkForInstructorData();
    const selfFilterData = checkForSelfData();
    const otherAttendeeArray = checkForOtherData();
    // const otherAttendeeArray = Array.from({ length: 10 });
    let instructorData = instructorFilterData.length > 0 ? instructorFilterData[0] : null
    let selfData = selfFilterData.length > 0 ? selfFilterData[0] : null

    const maxArrayLength = 4;
    const otherAttendeeArray1 = otherAttendeeArray.slice(0, maxArrayLength);
    const otherAttendeeArray2 = otherAttendeeArray.slice(4, 10);
    const otherAttendeeArray3 = otherAttendeeArray.slice(10);

    useEffect(() => {
        let timer;
        if (menuVisible) {
            timer = setTimeout(updateStateAfterTimeout, 20000); // 20 seconds
        }
        return () => clearTimeout(timer);
    }, [menuVisible]);

    // Function to update the state
    const updateStateAfterTimeout = () => {
        setMenuVisible(false);
    };

    return (
        <View style={[styles.container]}>
            <View style={[styles.container]}
                onTouchStart={() => setMenuVisible(!menuVisible)}
            >
                <Swiper
                    loop={false}
                    showsButtons={false}
                    activeDotColor={'#FFFFFF'}
                    dotColor={'#505050'}
                    style={{ overflow: 'visible', marginTop: pageNumber == 0 ? 0 : -20 }}
                    automaticallyAdjustContentInsets={true}
                    onIndexChanged={(index) => {
                        setPageNumber(index)
                    }}
                >
                    <MemoizedFirstPageComponent
                        instructorData={instructorData}
                        otherAttendeeArray={otherAttendeeArray}
                        selfData={selfData}
                        pageNumber={pageNumber}
                        desfaultSelecetdCamera={desfaultSelecetdCamera}
                        settingVisible={settingVisible}
                        onTap={() => setMenuVisible(!menuVisible)}
                        userData={userData}
                    />
                    {
                        otherAttendeeArray.length > 0 &&
                        <MemoizedSecondPageComponent
                            instructorData={instructorData}
                            totalOtherArray={otherAttendeeArray}
                            otherAttendeeArray={otherAttendeeArray1}
                            otherAttendeeArray2={otherAttendeeArray2}
                            selfData={selfData}
                            selfFilterData={selfFilterData}
                            pageNumber={pageNumber}
                            desfaultSelecetdCamera={desfaultSelecetdCamera}
                            settingVisible={settingVisible}
                            userData={userData}
                        // onTap={() => setMenuVisible(!menuVisible)}
                        />
                    }

                </Swiper>

            </View>
            {
                menuVisible && selfData &&
                <StudentMenuSection
                    tilesLength={length}
                    handleCamera={() => {
                        handleCamera(selfData)
                        setIsLocalCameraOn(!isLocalCameraOn)
                    }}
                    handleEndMeeting={handleEndMeeting}
                    selfVideoEnabled={selfVideoEnabled}
                    handleMuteButton={() => {
                        handleMuteButton(selfData)
                        setIsLocalMicOn(!isLocalMicOn)
                    }}
                    selfData={selfData}
                    formattedTime={formattedTime}
                    handleSettingButton={() => {
                        if (selfData) {
                            setIsLocalCameraOn(selfData?.isCameraOn)
                            setIsLocalMicOn(selfData?.isMuted)
                            handleSettingButton()
                            setSettingVisible(true)
                        }
                    }}
                    handleMenuUserIcon={() => setIsUserListVisible(true)}
                />
            }
            {
                settingVisible &&
                <SettingModal
                    onRequestClose={() => setSettingVisible(false)}
                    closeSettingModal={() => {
                        setSettingVisible(false)
                        closeSettingModal(selfData, isLocalCameraOn, isLocalMicOn)
                    }}
                    selfData={selfData}
                    handleSettingVideoIcon={() => handleSettingVideoIcon(selfData)}
                    handleSettingMuteIcon={() => handleSettingMuteIcon(selfData)}
                    handleSelectCameraItem={handleSelectCameraItem}
                    desfaultSelecetdCamera={desfaultSelecetdCamera}
                    listAudioDevice={listAudioDevice}
                    handleOnPressMicItem={handleOnPressMicItem}
                    selectedMicrophone={selectedMicrophone}
                    selectedSpeaker={selectedSpeaker}
                    isLocalCameraOn={isLocalCameraOn}
                    setIsLocalCameraOn={setIsLocalCameraOn}
                    isLocalMicOn={isLocalMicOn}
                    setIsLocalMicOn={setIsLocalMicOn}
                />
            }

            {
                isUserListVisible &&
                <StudentSideUsersList
                    visible={isUserListVisible}
                    onRequestClose={() => setIsUserListVisible(false)}
                    usrsArray={[...otherAttendeeArray, selfData]}
                />
            }
        </View>
    );
};

export default StudentSideVideoStream;

const styles = StyleSheet.create({
    container: {
        flex: 1,
        flexDirection: 'row',
        paddingTop: 16,
        alignItems: 'center',
        justifyContent: 'center',
        marginTop: -20,

    },
    itemContainer: {
        padding: 8,
        height: 175,
        width: '100%',
    },
    imageStyle: {
        height: '100%',
        width: '100%',
        borderRadius: 6,
        // margin: 8
    },
    nameStyle: {
        fontSize: 11,
        fontWeight: '600',
        color: AppColors.black,
    },
    muteIconstyle: {
        height: 14,
        width: 14,
    },
    nameViewStyle: {
        flexDirection: 'row',
        alignItems: 'center',
        justifyContent: 'space-around',
        backgroundColor: AppColors.white,
        width: 75,
        height: 17,
        borderRadius: 2,
    },
    ratingText: {
        fontSize: 11,
        fontWeight: '900',
        color: AppColors.red,
    },
    innerView: {
        flexDirection: 'row',
        alignItems: 'center',
        justifyContent: 'space-between',
        width: 100,
        position: 'absolute',
        bottom: 10,
        left: 13,
    },
    ratingCircleView: {
        height: 15,
        width: 15,
        backgroundColor: AppColors.white,
        shadowColor: 'red',
        shadowOffset: {
            width: 10,
            height: 6,
        },
        shadowOpacity: 0.37,
        shadowRadius: 7.49,
        elevation: 10,
        borderRadius: 20,
        alignItems: 'center',
        justifyContent: 'center',
    },
    observerIndicatorIcon: {
        height: 18,
        width: 18,
    },
    observerIndicatorView: {
        position: 'absolute',
        top: 10,
        right: 10,
    },
    firstPageContainer: {
        flexDirection: 'row',
        justifyContent: 'space-around',
    },
    widgetMainView: {
        height: 120,
        width: 120,
        backgroundColor: '#333333D9',
        borderRadius: 90,
        position: 'absolute',
        right: 0,
        alignItems: 'center',
        justifyContent: 'center',
        top: 10
    },
    widgetInnerView: {
        flexDirection: 'row',
        alignItems: 'center',
        justifyContent: 'center'
    },
    heartCountTextStyle: {
        fontSize: FontSize.F32,
        fontWeight: FontWeight.FW300,
        color: AppColors.white
    },
    calTextStyle: {
        fontSize: FontSize.F14,
        fontWeight: FontWeight.FW500,
        color: AppColors.red,
        marginLeft: 5
    }, classBeginMainView: {
        height: '100%',
        width: '100%',
        backgroundColor: '#FFFFFFB2',
        alignItems: 'center',
        justifyContent: 'center'
    },
    classStartShortly: {
        fontSize: FontSize.F15,
        fontWeight: FontWeight.FW500,
        color: AppColors.darkBlack
    },
});

RNVideoRenderView Component:

/*
 * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
 * SPDX-License-Identifier: MIT-0
 */

import PropTypes from 'prop-types';
import React from 'react';
import { requireNativeComponent, findNodeHandle } from 'react-native';
import { NativeFunction } from '../../utils/Bridge';

export class RNVideoRenderView extends React.Component {

  componentDidMount() {
    // we need to delay the bind video 
    // Because "componentDidMount" will be called "immediately after the initial rendering occurs"
    // This is *before* RCTUIManager add this view to register (so that viewForReactTag() can return a view)
    // So we need to dispatch bindVideoView after this function complete
    setTimeout(() => {
      NativeFunction.bindVideoView(findNodeHandle(this), this.props.tileId);
    });
  }

  componentWillUnmount() {
    NativeFunction.unbindVideoView(this.props.tileId);
  }

  render() {
    return <RNVideoRenderViewNative {...this.props} />;
  }
}

RNVideoRenderView.propTypes = {
  /**
   * A int value to identifier the Video view, will be used to bind video stream later
   */
  tileId: PropTypes.number,
};

var RNVideoRenderViewNative = requireNativeComponent('RNVideoView', RNVideoRenderView);

Environment:

• React Native version: 0.71.10
• AWS Chime SDK version: 0.18.2
• Device: Samsung Galaxy M04
• Operating System: Android

This issue impacts the overall usability of our video conferencing feature. We appreciate your prompt attention to this matter.

[ashirkhan94]

Hi @dharam-step I think this is because the SDK pauses and resumes the video based on the network conditions. For IOS you can add new 2 events for pause and resume (here it is KEventonOnPauseVideoTile and KEventOnResumeVideoTile) and test it with the below observer functions , //For Pause

- (void)videoTileDidPauseWithTileState:(VideoTileState * _Nonnull)tileState
{
  // Not implemented for demo purposes
    [_bridge sendEventWithName:KEventonOnPauseVideoTile body:@{@"tileId":[NSNumber numberWithInt: (int)tileState.tileId],
                                                               @"attendeeId":[tileState attendeeId], @"isLocal":@(tileState.isLocalTile), @"isScreenShare":@(tileState.isContent),@"pauseState":@(tileState.pauseState)}];

}

// For Resume

- (void)videoTileDidResumeWithTileState:(VideoTileState * _Nonnull)tileState
{
  // Not implemented for demo purposes
  [_bridge sendEventWithName: KEventOnResumeVideoTile body:@{@"tileId":[NSNumber numberWithInt: (int)tileState.tileId],
@"attendeeId":[tileState attendeeId], @"isLocal":@(tileState.isLocalTile), @"isScreenShare":@(tileState.isContent), @"pauseState":@(tileState.pauseState)}];

}

https://aws.github.io/amazon-chime-sdk-android/amazon-chime-sdk/com.amazonaws.services.chime.sdk.meetings.audiovideo.video/-video-pause-state/index.html the above link explain the pauseState

For Android

pause:--

override fun onVideoTilePaused(tileState: VideoTileState) {
        // Not implemented for demo purposes
        logger.info(TAG, "Received event for VideoTilePaused: $tileState")
        eventEmitter.sendVideoTileEvent(RN_EVENT_VIDEO_TILE_PAUSED, tileState)

    }

Resume:---

    override fun onVideoTileResumed(tileState: VideoTileState) {
        // Not implemented for demo purposes
         logger.info(TAG, "Received event for VideoTileResumed: $tileState")
         eventEmitter.sendVideoTileEvent(RN_EVENT_VIDEO_TILE_RESUME, tileState)

    }

[ashirkhan94]

You can test the pause state in the react native side with pauseState value if the pause is due to poor network conditions the pauseState value ==2 ,

in react native side

   const OnPauseVideoTileSubscription = getSDKEventEmitter().addListener(MobileSDKEvent.OnPauseVideoTile, (tileState) => {
      console.log("OnPauseVideoTileSubscription=", tileState)
      if (tileState.pauseState == 2) {
       console.log("Video was paused due to a poor network connection") 
      }
    })

you can pause and resume video tiles based on the active page to reduce network issue

iOS:----

RCT_EXPORT_METHOD(setPauseTile:(NSNumber * _Nonnull)tileId) {
[meetingSession.audioVideo pauseRemoteVideoTileWithTileId:[tileId integerValue]];
}

RCT_EXPORT_METHOD(setResumeTile:(NSNumber * _Nonnull)tileId) {
[meetingSession.audioVideo resumeRemoteVideoTileWithTileId:[tileId integerValue]];
}

android:------

@ReactMethod
    fun setPauseTile(tileId: Int) {
logger.info(TAG, "setPauseTile: $tileId")
 meetingSession?.run {
            audioVideo.pauseRemoteVideoTile(tileId)
        }
}
@ReactMethod
    fun setResumeTile(tileId: Int) {
logger.info(TAG, "setResumeTile: $tileId")
 meetingSession?.run {
            audioVideo.resumeRemoteVideoTile(tileId)
        }
}