Switching components causes the video element to be interrupted

[LRYMND]

I'm getting the following error when trying to unmount the carplay component with React Router:

The play() request was interrupted because the media was removed from the document.

I'm getting this error after switching back and forth a few times.

I tried the following cleanup function, also I changed the running state to a variable, because the state won't get updated in the first render which caused the play() function to run idefinitely.

useEffect(() => {
    console.log("opening carplay", settings);
    let videoElement = null;
    let isRunning = false;

    jmuxer = new JMuxer({
      node: 'player',
      mode: 'video',
      maxDelay: 30,
      fps: 30,
      flushingTime: 100,
      debug: false
    });

    const height = ref.current.clientHeight;
    const width = ref.current.clientWidth;
    setHeight(height);
    setWidth(width);

    if (status)
      setShowNav(false);

    if (type === 'ws') {
      ws.onmessage = event => {
        if (!isRunning) {
          videoElement = document.getElementById('player');
          if (videoElement != null)
            videoElement.play().then(() => console.log("Play Promise"));
          isRunning = true;
        }

        let buf = Buffer.from(event.data);
        let video = buf.slice(4);
        jmuxer.feed({
          video: new Uint8Array(video)
        });
      };
    } else if (type === "socket.io") {
      ws.on('carplay', data => {
        let buf = Buffer.from(data);
        let duration = buf.readInt32BE(0);
        let video = buf.slice(4); //console.log("duration was: ", duration)

        jmuxer.feed({
          video: new Uint8Array(video),
          duration: duration
        });
      });
    }

    return function cleanup() {

      var thePromise = videoElement.play();
      console.log("promise: ");

      if (thePromise != undefined) {

        thePromise.then(function () {
          if (videoElement != null) {
            videoElement.pause();
            videoElement.currentTime = 0;
          }
        }).catch((error) => console.log(error));
      }
    }    
  }, []);

[rhysmorgan134]

Not a functional component but it should get you where you need to get to.

https://github.com/rhysmorgan134/jaguar-xf-canbus-app/blob/rework/app/src/components/carplay/carplayWindow.js

I will make it so if no settings are passed the button is hidden

[LRYMND]

I see that socket.io is being used in this case but it's difficult to understand the structure and get an overview. I assume that you are connecting/disconnecting from the socket as needed but could you eventually dumb it down for me a bit in two or three sentences?

[rhysmorgan134]

The carplay component from this library is the one you should use, it handles all the data feeding to jmuxer by its self. All you need to do is pass it reference to the websocket or the socket.io instance. I am unsure as to why you have that above function since that is everything that this component handles?

If you have your entire code I can help further, but the above code looks to just be a snippet that is replicating this library?

https://github.com/rhysmorgan134/jaguar-xf-canbus-app/blob/rework/app/src/components/carplay/carplayWindow.js

This file handles the connection to the socket, and also this disconnection on unmount, it implements the carplay component from this library. This is handled likely elsewhere in your app, if the socket connection is at the top level, then the instance of that socket just needs to passed all the way down.

https://github.com/rhysmorgan134/react-js-carplay/blob/main/src/Carplay.js

This file then checks whether it is a socket.io instance or a websocket instance and attaches the relevant listener.

[LRYMND]

I copied your structure and wrapped the Carplay component inside the carplayWindow.js. I tried setting up an io.socket client and connect it to the ws server. It didnt work for me. I'm getting CORS errors and also I don't understand why the server is running on port 3000 while the io.client is using port 3002? I went back then and used ws client side but cannot seem to disconnect it once I'm leaving the page.

This is my carplayWindow, the Carplay component looks like yours.

import React, { useState, useEffect } from 'react';
import JMuxer from 'jmuxer';
import io from 'socket.io-client'
import Carplay from './Carplay'
import { useNavigate } from 'react-router-dom';

const { ipcRenderer } = window;
let socket = null;

const useConstructor = (callBack = () => {}) => {
    const [hasBeenCalled, setHasBeenCalled] = useState(false);
    if (hasBeenCalled) return;
    callBack();
    setHasBeenCalled(true);
  }

function CarplayWindow({ settings, setShowNav }) {
    const [status, setStatus] = useState(false);
    const [playing, setPlaying] = useState(false);
    const [connected, setConnected] = useState(false);
    const [start, setStart] = useState(null);

    const navigate = useNavigate();

    //const socket = io("ws://localhost:3001", {transports: ['websocket'], upgrade: false}).connect("ws://localhost:3001");

    useConstructor(() => {
        socket = new WebSocket("ws://localhost:3001");
        socket.binaryType = "arraybuffer";
      });

    useEffect(() => {

    console.log("Socket: ", socket);
        //socket.on('connect', () => setConnected(true))

        ipcRenderer.on('plugged', () => { setStatus(true); console.log("connected") });
        ipcRenderer.on('unplugged', () => { setStatus(false); console.log("disconnected") });
        ipcRenderer.on('quitReq', () => { leaveCarplay() });
        ipcRenderer.send('statusReq');

        return function cleanup() {
            //socket.off('connect', () => setConnected(false));
            ipcRenderer.removeAllListeners();
            socket.close();
            //socket.disconnect(true);
        };
    }, []);

    useEffect(() => {
        if (status) {
            if (window.location.hash === "#/" || window.location.hash === "") {
                setShowNav(false);
            }
        } else {
            setShowNav(true);
        }
    }, [status]);

    const touchEvent = (type, x, y) => {
        ipcRenderer.send('click', { type: type, x: x, y: y });
        console.log("touch event type: ", +type + " x: " + x + " y:" + y);
      };

    function leaveCarplay() {
        setShowNav(true);
        navigate("/dashboard");
        console.log("leaving carplay");
    }

    return (
        <Carplay 
            touchEvent={touchEvent}
            ws={socket}
            type={"ws"}
            settings={settings}
            status={status}   
        />

    );
} 

 export default CarplayWindow;

When I'm closing the socket connection I'll get an error once I switched the page "Websocket is not open: readyState2". When I'm not closing the socket connection, I'll get the error I mentioned in my first post. I'd like to use your approach but again, I don't understand how youre using io.socket along with websocket on different ports especially when the documentation says that a socket.io client is not compatible to a standard web socket server.

[rhysmorgan134]

give this a try

import Carplay from 'react-js-carplay'
import { useNavigate } from 'react-router-dom';

const { ipcRenderer } = window;
let socket = null;

function CarplayWindow({ settings, setShowNav }) {
    const [status, setStatus] = useState(false);
    const [playing, setPlaying] = useState(false);
    const [connected, setConnected] = useState(false);
    const [start, setStart] = useState(null);

    const navigate = useNavigate();

    useEffect(() => {
        socket = new WebSocket("ws://localhost:3001");
        socket.binaryType = "arraybuffer";
        //socket.on('connect', () => setConnected(true))

        ipcRenderer.on('plugged', () => { setStatus(true); console.log("connected") });
        ipcRenderer.on('unplugged', () => { setStatus(false); console.log("disconnected") });
        ipcRenderer.on('quitReq', () => { leaveCarplay() });
        ipcRenderer.send('statusReq');

        return () => {
            //socket.off('connect', () => setConnected(false));
            ipcRenderer.removeAllListeners();
            socket.close();
            //socket.disconnect(true);
        };
    }, []);

    useEffect(() => {
        if (status) {
            if (window.location.hash === "#/" || window.location.hash === "") {
                setShowNav(false);
            }
        } else {
            setShowNav(true);
        }
    }, [status]);

    const touchEvent = (type, x, y) => {
        ipcRenderer.send('click', { type: type, x: x, y: y });
        console.log("touch event type: ", +type + " x: " + x + " y:" + y);
    };

    function leaveCarplay() {
        setShowNav(true);
        navigate("/dashboard");
        console.log("leaving carplay");
    }

    return (
        <Carplay
            touchEvent={touchEvent}
            ws={socket}
            type={"ws"}
            settings={settings}
            status={status}
        />

    );
}

export default CarplayWindow;

[LRYMND]

Doesnt work... Now I'm getting the error Cannot set properties of null (setting 'onmessage') as soon as I open the carplay page. It seems like the socket is not connected in time.

I moved

socket = new WebSocket("ws://localhost:3001");
socket.binaryType = "arraybuffer";

outside the function again but now I'm getting "Websocket is not open: readyState2" again when moving away from carplay.

[rhysmorgan134]

then I would set a listener to onconnect of websocket and conditionally render carplay based on that. If you do a push of your current code I can take a look when I get a chance, hard to help with snippets!

[LRYMND]

I pushed the current code to the experimental branch of my volvo-rtvi repo.

[rhysmorgan134]

import React, { useState, useEffect } from 'react';
import Carplay from 'react-js-carplay'
import { useNavigate } from 'react-router-dom';

const { ipcRenderer } = window;
let socket = null;

function CarplayWindow({ settings, setShowNav }) {
    const [status, setStatus] = useState(false);
    const [playing, setPlaying] = useState(false);
    const [connected, setConnected] = useState(false);
    const [start, setStart] = useState(null);

    const navigate = useNavigate();

    useEffect(() => {
        socket = new WebSocket("ws://localhost:3001");
        socket.binaryType = "arraybuffer";
        socket.addEventListener('open', () => setConnected(true))
        ipcRenderer.on('plugged', () => { setStatus(true); console.log("connected") });
        ipcRenderer.on('unplugged', () => { setStatus(false); console.log("disconnected") });
        ipcRenderer.on('quitReq', () => { leaveCarplay() });
        ipcRenderer.send('statusReq');

        return () => {
            //socket.off('connect', () => setConnected(false));
            ipcRenderer.removeAllListeners();
            socket.close();
            //socket.disconnect(true);
        };
    }, []);

    useEffect(() => {
        if (status) {
            if (window.location.hash === "#/" || window.location.hash === "") {
                setShowNav(false);
            }
        } else {
            setShowNav(true);
        }
    }, [status]);

    const touchEvent = (type, x, y) => {
        ipcRenderer.send('click', { type: type, x: x, y: y });
        console.log("touch event type: ", +type + " x: " + x + " y:" + y);
    };

    function leaveCarplay() {
        setShowNav(true);
        navigate("/dashboard");
        console.log("leaving carplay");
    }

    return (
        <div style={{width: '100%', height: '100%'}}>
    {connected ? <Carplay
            touchEvent={touchEvent}
            ws={socket}
            type={"ws"}
            settings={settings}
            status={status}
        /> : <div></div>}
    </div>
    );
}

export default CarplayWindow;

[rhysmorgan134]

tested although with no dongle and limited functionality, it should set you on the right path, if you don't use this library then you will likely run into issues as newer version of node-carplay get released. The latest react-js-carplay will always be compatible with the latest node-carplay

[LRYMND]

Ok so I removed my own carplay.js now and installed react-js-carplay and copy-pasted your changes above. With the socket.close() in the return I'm still getting this error when leaving the carplay page:

Websocket is not open: readyState2

When I'm removing the line I'm getting the following errors when I'm NOT on the carplay page and the video component is not being rendered:

Uncaught (in promise) DOMException: The play() request was interrupted because the media was removed from the document. https://goo.gl/LdLk22

Uncaught TypeError: Cannot read properties of null (reading 'play')
    at ws.onmessage (bundle.js:108985:17)  //This error is thrown over and over again, probably on every frame.

So basically I'm still where I started but I'm using 100% your code now. Except the CarplayWindow component of course. Pushed the changes to the branch.

[LRYMND]

I tried the method from the link in the log message again and implemented it this way in my custom carplay.js (which is almost identical to the one from react-js-carplay):

    let videoElement
    let playPromise

    if (type === 'ws') {
      ws.onmessage = (event) => {
        if (!running) {
          videoElement = document.getElementById('player')
          if(videoElement != null) {
            playPromise = videoElement.play()
            if(playPromise !== undefined) {
              setRunning(true)
              playPromise.then(_ => {

              })
            .catch(error => {
            });
            }
          }
        }
        let buf = Buffer.from(event.data)
        let video = buf.slice(4)
        jmuxer.feed({ video: new Uint8Array(video) })
      }
    }

From what I read, the video is stopping automatically, once you remove the component and this way the promise from the video is handled and errors are being caught when you remove it.

So far I didnt got any more error messages. Occasionally carplay is not showing up when I switch back to the page, however the touch input still works and when I click where the auto icon should be, the "quitReq" gets registered and I can leave the page again. When I move back to the carplay page it usually shows up normal again. Not sure though if this might be related or some states are just not properly updated though.

Maybe this is something you want to implement in react-js-carplay since it resolves the issue with an unhandled promise?

//Edit: should be noted though that this is not helping with not being able to disconnect from the socket.

//Edit 2: I resolved the socket issue by moving it one layer up to app.js. This way the socket stays just active the whole time but I'm not creating multiple instances.