Video Streams show Black Screen sometimes (Android)

[stephanoparaskeva]

Bug report

• Reproduced on:
  • [x] iOS

Description

This only happens sometimes, and I can't find how to make it consistent. But it is a critical bug for myself. When call comes on Android device, sometimes both Remote and Local Stream show Black screen and I can't see any live video feed.

I use React Native CallKeep for handling calls.

Steps to Reproduce

• Call Android Device
• Display Incoming Call UI
• Accept Call and App is sent back to foreground
• See buttons and black screen instead of video stream (IE RTCView is displaying Black Video screen)

Versions

- ConnectyCube: Latest
- CallKeep: Latest
- React Native: 0.63.2
- Phone model: Android 7 Samsung S6

Code

const CallControl = ({ route }: { route: any }) => {
  const initSession = {
    ID: '',
    initiatorID: 0,
    currentUserID: 0,
    opponentsIDs: [],
    getUserMedia: () => {},
    accept: () => {},
    call: () => {},
    stop: () => {},
    mute: () => {},
    unmute: () => {},
    reject: () => {},
  };
  const initExtension = { busy: false };

  const sess = useRef<Session>(initSession);
  const ext = useRef<Extension>(initExtension);
  const callId = useRef<string>('');
  const currentUser = route?.params?.currentUser || {};

  const [localStream, setLocalStreams] = useState<any>([]);
  const [remoteStreams, setRemoteStreams] = useStateCallback<
    [] | Array<Stream>
  >([]);

  const [selectedUserId, setSelectedUser] = useState<null | number>(null);

  const [isAudioMuted, setIsAudioMuted] = useState(false);
  const [isFrontCamera, setIsFrontCamera] = useState(true);

  const cam = true;

  const reset = async () => {
    const id = oneTimeId();
    RNCallKeep.endCall(id);
    await ConnectyCube.videochat.clearSession(sess.current.ID);
    RNCallKeep.endAllCalls();

    InCallManager.stop();
    setRemoteStreams([]);
    setLocalStreams([]);
    setSelectedUser(null);
    setIsAudioMuted(false);
    setIsFrontCamera(true);
    sess.current = initSession;
    ext.current = initExtension;
    callId.current = '';
  };

  const oneTimeId = () => {
    callId.current = callId.current || uuid();
    return callId.current;
  };

  // Start/Stop onPress call handlers
  const startCall = async () => {
    if (!selectedUserId) {
      return;
    }

    const t = ConnectyCube.videochat.CallType.VIDEO; // AUDIO is also possible
    sess.current = ConnectyCube.videochat.createNewSession([selectedUserId], t);

    const stream = await sess.current?.getUserMedia?.(MEDIA_OPTIONS);
    await sess.current?.call({});

    setLocalStreams([{ userId: 'localStream', stream }]);
    setRemoteStreams([{ userId: selectedUserId, stream: null }]);

    if (isIOS) {
      const id = oneTimeId();
      const name = users.find(u => u.id === selectedUserId)?.name || '';
      RNCallKeep.startCall(id, name, name, 'generic', cam);
    }
  };

  const stopCall = async () => {
    await sess.current?.stop?.({});
    await reset();
  };

  // CallKeep button actions
  const acceptCall = async () => {
    const id = oneTimeId();
    RNCallKeep.setCurrentCallActive(id);

    const stream = await sess.current?.getUserMedia?.(MEDIA_OPTIONS);
    await sess.current?.accept?.({});

    setLocalStreams([{ userId: 'localStream', stream }]);
    setRemoteStreams([{ userId: selectedUserId, stream: null }]);
  };

  const rejectCall = async () => {
    await sess.current?.reject?.(ext.current);
    await reset();
  };

  // ConnectyCall Listeners
  const onCallListener: Listener = async (session, _userId, extension) => {
    ext.current = extension || initExtension;
    sess.current = session || initSession;

    const name =
      users.find(u => u.id === sess?.current?.initiatorID)?.name || '';
    const id = oneTimeId();
    RNCallKeep.displayIncomingCall(id, name, name, 'generic', cam);
  };

  const onAcceptCallListener: Listener = async (session, _, extension) => {
    ext.current = extension || initExtension;
    sess.current = session || initSession;
  };

  const onRemoteStreamListener = async (
    _session: Session,
    userId: number,
    stream: Stream
  ) =>
    setRemoteStreams([...remoteStreams, { userId, stream }], () => {
      InCallManager.start({ media: 'video', auto: false });
      InCallManager.setSpeakerphoneOn(true);
      RNCallKeep.backToForeground();
    });

  const onRejectCallListener = async () => {
    await reset();
  };

  const onStopCallListener = async () => {
    await reset();
  };

  const onUserNotAnswerListener = async () => {
    if (!remoteStreams[0]) {
      await reset();
    }
  };

  // Init
  const initializeConnectyCube = () => {
    ConnectyCube.videochat.onCallListener = onCallListener;
    ConnectyCube.videochat.onAcceptCallListener = onAcceptCallListener;
    ConnectyCube.videochat.onRemoteStreamListener = onRemoteStreamListener;
    ConnectyCube.videochat.onRejectCallListener = onRejectCallListener;
    ConnectyCube.videochat.onStopCallListener = onStopCallListener;
    ConnectyCube.videochat.onUserNotAnswerListener = onUserNotAnswerListener;
  };

  const initializeCallKeep = async () => {
    await RNCallKeep.setup(SETUP_OPTIONS);
    RNCallKeep.setAvailable(true);
    RNCallKeep.addEventListener('answerCall', acceptCall);
    RNCallKeep.addEventListener('endCall', rejectCall);
  };

  const cleanup = () => {
    RNCallKeep.removeEventListener('answerCall');
    RNCallKeep.removeEventListener('endCall');
    stopCall();
  };

  useEffect(() => {
    initializeConnectyCube();
    initializeCallKeep();
    return cleanup;
  }, []);

  // Actions
  const toggleMute = () => {
    const mute = !isAudioMuted;
    setIsAudioMuted(mute);
    mute ? sess.current?.mute?.('audio') : sess.current?.unmute?.('audio');
  };

  const toggleCamera = () => {
    setIsFrontCamera(!isFrontCamera);
    localStream[0]?.stream
      ?.getVideoTracks?.()
      .forEach((track: Track) => track._switchCamera());
  };

  // Buttons
  const renderCallStartStopButton = () => (
    <TouchableOpacity
      style={[
        styles.buttonContainer,
        remoteStreams[0] ? styles.buttonCallEnd : styles.buttonCall,
      ]}
      onPress={remoteStreams[0] ? stopCall : startCall}>
      <MaterialIcon
        name={remoteStreams[0] ? 'call-end' : 'call'}
        size={32}
        color='white'
      />
    </TouchableOpacity>
  );

  const renderMuteButton = () => (
    <TouchableOpacity
      style={[styles.buttonContainer, styles.buttonMute]}
      onPress={toggleMute}>
      <MaterialIcon
        name={isAudioMuted ? 'mic-off' : 'mic'}
        size={32}
        color='white'
      />
    </TouchableOpacity>
  );

  const renderSwitchVideoSourceButton = () => (
    <TouchableOpacity
      style={[styles.buttonContainer, styles.buttonSwitch]}
      onPress={toggleCamera}>
      <MaterialIcon
        name={isFrontCamera ? 'camera-rear' : 'camera-front'}
        size={32}
        color='white'
      />
    </TouchableOpacity>
  );

  return (
    <SafeAreaView style={{ flex: 1, backgroundColor: 'black' }}>
      <StatusBar backgroundColor='black' barStyle='light-content' />
      {remoteStreams[0] ? (
        <View style={styles.inColumn}>
          <VideoScreen
            mirror={false}
            stream={remoteStreams?.[0]?.stream}
            userId={remoteStreams?.[0]?.userId}
          />
          <VideoScreen
            mirror={isFrontCamera}
            stream={localStream?.[0]?.stream}
            userId={localStream?.[0]?.userId}
          />
        </View>
      ) : (
        <UsersSelect
          currentUser={currentUser}
          selectedUserId={selectedUserId}
          setSelectedUser={setSelectedUser}
        />
      )}
      <SafeAreaView style={styles.container}>
        <View style={styles.toolBarItem}>
          {remoteStreams[0] && renderMuteButton()}
        </View>
        {(selectedUserId || remoteStreams[0]) && (
          <View style={styles.toolBarItem}>{renderCallStartStopButton()}</View>
        )}
        <View style={styles.toolBarItem}>
          {remoteStreams[0] && localStream && renderSwitchVideoSourceButton()}
        </View>
      </SafeAreaView>
    </SafeAreaView>
  );
};

[ccvlad]

Hi, @stephanoparaskeva Were you meeting the issue in our sample? Or could you give any steps to reproduce? Did you test with other Android devices (not Sams Galaxy)?

[stephanoparaskeva]

Hi, @stephanoparaskeva Were you meeting the issue in our sample? Or could you give any steps to reproduce? Did you test with other Android devices (not Sams Galaxy)?

Not in the example, I provided the above code. You can use that to test. The issue comes after integrating ConnectyCube with RN CallKeep.

I've tried on S6 and Google Note. Same problem. To reproduce you can play with the above code ^ and move items around and you'll see that sometimes a black stream is the result which is quite problematic.

[whenmoon]

I am also experiencing this issue although I'm not using RN CallKeep. I have seen it on a Google Pixel 2XL and Samsung A10 and seems to happen at random times, not very often to it is hard to reproduce.

I'm using react-native-webrtc to render video, could it be related to any of these issues? There are a lot relating to black screens on Android.

[ccvlad]

The log will be helpful. Anyway we will update you if we catch the problem or fix it.

I'm using react-native-webrtc to render video, could it be related to any of these issues? There are a lot relating to black screens on Android.

@whenmoon , thanks for your message. Our SDK also uses the react-native-webrtc as import to exports the RTCView.

[whenmoon]

Thanks @ccvlad , I'm going to see if I can recreate the issue.

[stephanoparaskeva]

Thanks @ccvlad , I'm going to see if I can recreate the issue.

Is there any progress?

[whenmoon]

Thanks @ccvlad , I'm going to see if I can recreate the issue.

Is there any progress?

I haven't been able to to consistently recreate yet, but I did find this comment which passes a zOrder prop to RTCView. I've also read comments about it happening to people when using cellular networks instead of wifi, and it is true, it has always happened to me when on 4G networks.

Will keep trying to debug but am working on other bugs rn.

[stephanoparaskeva]

Thanks @ccvlad , I'm going to see if I can recreate the issue.

Is there any progress?

I haven't been able to to consistently recreate yet, but I did find this comment which passes a zOrder prop to RTCView. I've also read comments about it happening to people when using cellular networks instead of wifi, and it is true, it has always happened to me when on 4G networks.

Will keep trying to debug but am working on other bugs rn.

This happens to me over wifi

[ccvlad]

Hi, @stephanoparaskeva We have released SDK v.3.6.2. There are you are able to set only TURN server instead of STUN. Set CONFIG.videochat.alwaysRelayCalls as true in your config.

We are testing TURN servers now. I did not find any issue with EU TURN server (RN Android with RN iOS, RN Android with the sample videochat web). We are going to test US TURN. Will let you know.

Could you attach logs with the issue starting from init a call? Will be helpful

[stephanoparaskeva]

Hi, @stephanoparaskeva We have released SDK v.3.6.2. There are you are able to set only TURN server instead of STUN. Set CONFIG.videochat.alwaysRelayCalls as true in your config.

We are testing TURN servers now. I did not find any issue with EU TURN server (RN Android with RN iOS, RN Android with the sample videochat web). We are going to test US TURN. Will let you know.

Could you attach logs with the issue starting from init a call? Will be helpful

I am from the EU. And what will this do?

[whenmoon]

Hi, @stephanoparaskeva We have released SDK v.3.6.2. There are you are able to set only TURN server instead of STUN. Set CONFIG.videochat.alwaysRelayCalls as true in your config.

We are testing TURN servers now. I did not find any issue with EU TURN server (RN Android with RN iOS, RN Android with the sample videochat web). We are going to test US TURN. Will let you know.

Could you attach logs with the issue starting from init a call? Will be helpful

I will try this too. As I can't recreate right now @stephanoparaskeva if you can get any logs that would be great.

[stephanoparaskeva]

@whenmoon

How does one obtain logs?

[ccvlad]

Guys, any logs or special behavior will be great as info. I can't reproduce to catch the problem.

[whenmoon]

@whenmoon

How does one obtain logs?

logcat in Android studio should catch it?

[ccvlad]

@whenmoon

How does one obtain logs?

Metro server for RN iOS/Android, Android Studio logcat, XCode, browser console for web

[stephanoparaskeva]

@whenmoon How does one obtain logs?

Metro server for RN iOS/Android, Android Studio logcat, XCode, browser console for web

I doubt this is a problem with STUN/TURN as I still receive stream and IOS always works. It seems to be a problem on android after I converted my code to React Native CallKeep and changed all of ConnectyCube example to use React Hooks.

The above code might reveal the real issue

I can also provide an adapted version of the repo if need be?

[whenmoon]

I think this could likely be a STUN/TURN issue but need to test

[stephanoparaskeva]

Any updates on this?

[ccvlad]

Hi, @stephanoparaskeva We have not reproduced the issue. I cannot say to which project this problem belongs until I clearly met. Also we do not use the RN CallKeep UI. The CallKeep is not our product, you may need to refer to them. It would also be useful to see the log at the time of the problem, then we can find out whose problem.

Do you hear the sound of stream when it shows just black screen?

[stephanoparaskeva]

Hi, @stephanoparaskeva We have not reproduced the issue. I cannot say to which project this problem belongs until I clearly met. Also we do not use the RN CallKeep UI. The CallKeep is not our product, you may need to refer to them. It would also be useful to see the log at the time of the problem, then we can find out whose problem.

Do you hear the sound of stream when it shows just black screen?

Yes we hear the sound of the stream, and the stream seems to be working just that there is a black screen. I believe the Stream may still think app is not in foreground or something?

[ccvlad]

Then it makes no sense to watch the log cause everything will be fine there. This is a UI problem. Something is wrong with the RTCView or with the RN CallKeep. Did you set the prop zOrder in the RTCView?

[stephanoparaskeva]

Then it makes no sense to watch the log cause everything will be fine there. This is a UI problem. Something is wrong with the RTCView or with the RN CallKeep. Did you set the prop zOrder in the RTCView?

I remember setting zOrder which didn't solve my issue but I can try again now. For reference this is my VideoScreen.js: While getting the black screen, the UI is displaying RTCView correctly. It all works, it just has RTCView showing a black screen.

import React from 'react';
import { View, StyleSheet } from 'react-native';
import { RTCView } from 'react-native-connectycube';
import CallingLoader from './call_loader';
import { Stream, users } from '../config';

const VideoScreen = ({
  userId,
  stream,
  mirror,
}: {
  mirror: boolean;
  userId: number;
  stream: Stream;
}) => {
  return (
    <View style={styles.blackView}>
      {stream ? (
        <RTCView
          mirror={mirror}
          objectFit='cover'
          style={styles.blackView}
          key={userId}
          streamURL={stream.toURL()}
        />
      ) : (
        <View style={styles.blackView}>
          <CallingLoader name={users.find(u => u.id === userId)?.name || ''} />
        </View>
      )}
    </View>
  );
};

export default VideoScreen;

const styles = StyleSheet.create({
  blackView: { flex: 1, backgroundColor: 'black' },
});

[ccvlad]

I use zOrder={1}. You can try to use bigger value - 999 or more...

[stephanoparaskeva]

I use zOrder={1}. You can try to use bigger value - 999 or more...

zOrder has no effect for me

[ccvlad]

I remember a similar issue for the cordova-plugin-iosrtc. Then I set z-index: -1 for other elements and positive for the webrtc view to make a stream visible. You can try to set backgroundColor as transparent:

const styles = StyleSheet.create({
  blackView: { flex: 1, backgroundColor: 'transparent' },
});

Your issue with black screen would related to the react-native-wertc or react-native-callkeep if it would not help

[whenmoon]

I've just updated my config to this: CONFIG.videochat.alwaysRelayCalls as true.

Will le t you know if I have any issues.

[stephanoparaskeva]

I've just updated my config to this: CONFIG.videochat.alwaysRelayCalls as true.

Will le t you know if I have any issues.

How is this going?

[ccvlad]

Hi @stephanoparaskeva Are you still having the black screen? Seems the CONFIG.videochat.alwaysRelayCalls = true won't fix your issue, because you already have a stream. You have problem with displaying your stream. I mean that you have UI issue.

[stephanoparaskeva]

Hi @stephanoparaskeva Are you still having the black screen? Seems the CONFIG.videochat.alwaysRelayCalls = true won't fix your issue, because you already have a stream. You have problem with displaying your stream. I mean that you have UI issue.

I am still having this issue. I am not sure what is causing it and how to fix it but it can happen randomly where I see a black screen instead of the stream. If you can try to replicate this using my code above it will be very helpful.

[PRRugebregt]

Hi! Is there any progress on this? I'm having the same issue in my iOS app. The audio is streaming and the users are connected, but it shows a black videoscreen. Hopefully someone can shed some light on this. Thanks.

[ccvlad]

Hi, @patturik

We heard about the issue on Android and still have no idea how to fix it. We were trying to reproduce, but w/o any results. I believe that the black screen problem on Android is related to CallKeep. Please provide to us any helpful info (steps to reproduce, logs). Can we reproduce the problem on our VideoChat sample? Also check the setting zOrder that discussed above.

[PRRugebregt]

Hi @ccvlad,

Thanks for the reply. I'm sorry I didn't see this was for android specifically.

However I managed to get it working eventually. For me the key was to configure and initialize the audiosession of the videosession. When I didn't set this specifically, I ended up with the occasional black screen for one user.. When I set the audiosession and configured it for the videochat, everything just runs smoothly! I don't know if this is any help, but just thought I'd put it out here. Maybe it's not the video itself that it causing the problem, but setting the correct audio device for the localstream. Good luck in solving this problem! And thanks again for the reply.

[ccvlad]

Closed due to inactivity. Please create a new issue if needed.