search

rhysmorgan134 / react-carplay

MIT License
423 stars 65 forks source link

Audio distorted issue tracker #23

Closed rhysmorgan134 closed 1 year ago

rhysmorgan134 commented 1 year ago

Please use the format below to list if you have audio issues

LRYMND commented 1 year ago
rhysmorgan134 commented 1 year ago

Good news, I have figured it out, hoping to get a new release done tomorrow. Newer dongles send out the audio at a different sample rate, so technically your audio was playing at 44100hz when it was being sent at 48000hz, the same with the seriously distorted audio, that was getting sent at 48000hz and being played back at 16000 hz

rhysmorgan134 commented 1 year ago

24 should resolve this, please try V1.0.6

LRYMND commented 1 year ago

Still having issues on node-carplay 2.0.11. Tested with react-js-carplay as well as my lightly modified spin-off. If I can log or test anything specific let me now!

rhysmorgan134 commented 1 year ago

If you could let me know which type of audio you are getting distortion on and which are good that would be great. Off the top of my head, there is media, nav, Siri implemented right now. If you could let me know and attached a decent length log file from when the audio is getting played that would be great

LRYMND commented 1 year ago

Tested again today.

I'm openenong the app and start music playback. When I launch the navigation then and the voice starts giving directions the audio gets distorted.

Log:

> export ELECTRON_START_URL=http://localhost:3000 && electron . --no-sandbox

spawning carplay:  {
  dpi: 240,
  nightMode: 0,
  hand: 0,
  boxName: 'nodePlay',
  width: 799,
  height: 479,
  fps: 60
}
directory found
sending dpi
device connected and ready
starting projection
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer e8 03 00 00>
Carplay message:  Support Wifi
unkown type:  13  with data::  <Buffer aa 55 aa 55 0d 00 00 00 0d 00 00 00 f2 ff ff ff>
unkown type:  14  with data::  <Buffer aa 55 aa 55 0d 00 00 00 0e 00 00 00 f1 ff ff ff>
unkown type:  18  with data::  <Buffer aa 55 aa 55 82 00 00 00 12 00 00 00 ed ff ff ff>
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer e9 03 00 00>
Carplay message:  undefined
test message i
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer 07 00 00 00>
Carplay message:  undefined
test message 
parsing type:  204  for length:  32 <Buffer aa 55 aa 55 20 00 00 00 cc 00 00 00 33 ff ff ff>
parsing 204 <Buffer 32 30 32 31 2e 31 32 2e 32 32 2e 31 36 34 37 43 41 59 00 00 00 00 00 00 00 00 00 00 00 00 00 00>
version number:  2021.12.22.1647CAY
unkown type:  25  with data::  <Buffer aa 55 aa 55 7e 00 00 00 19 00 00 00 e6 ff ff ff>
unkown type:  1  with data::  <Buffer aa 55 aa 55 1c 00 00 00 01 00 00 00 fe ff ff ff>
window ready
socket connected - sending data...
enabling wifi
auto connecting
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer eb 03 00 00>
Carplay message:  undefined
test message k
unkown type:  35  with data::  <Buffer aa 55 aa 55 11 00 00 00 23 00 00 00 dc ff ff ff>
unkown type:  36  with data::  <Buffer aa 55 aa 55 11 00 00 00 24 00 00 00 db ff ff ff>
unkown type:  18  with data::  <Buffer aa 55 aa 55 82 00 00 00 12 00 00 00 ed ff ff ff>
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer e9 03 00 00>
Carplay message:  undefined
test message i
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer ef 03 00 00>
Carplay message:  undefined
test message o
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer ec 03 00 00>
Carplay message:  undefined
test message l
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer f1 03 00 00>
Carplay message:  undefined
test message q
parsing type:  2  for length:  8 <Buffer aa 55 aa 55 08 00 00 00 02 00 00 00 fd ff ff ff>
parsing 2 <Buffer 03 00 00 00 01 00 00 00>
wifi avail, phone type:  3  wifi:  1
data received:  { status: true }
parsing type:  8  for length:  24 <Buffer aa 55 aa 55 18 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer 04 00 00 00 41 34 3a 43 33 3a 33 37 3a 30 43 3a 36 44 3a 31 46 0c f0 74>
Carplay message:  undefined
test message A4:C3:37:0C:6D:1F
                              pt
non-audio found
unknown audio type:  10 false false
non-audio found
unknown audio type:  1 false false
unkown type:  3  with data::  <Buffer aa 55 aa 55 04 00 00 00 03 00 00 00 fc ff ff ff>
unkown type:  37  with data::  <Buffer aa 55 aa 55 00 00 00 00 25 00 00 00 da ff ff ff>
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer f0 03 00 00>
Carplay message:  undefined
test message p
non-audio found
unknown audio type:  11 false false
non-audio found
unknown audio type:  10 false false
non-audio found
unknown audio type:  10 false false
non-audio found
setting audio to nav
non-audio found
unknown audio type:  1 false true
non-audio found
setting audio to pending media
non-audio found
setting audio to media now
non-audio found
setting audio to nav
non-audio found
unknown audio type:  1 false true
non-audio found
setting audio to pending media
non-audio found
setting audio to media now
14 0.24246231155778894 0.6586433260393874
16 0.24246231155778894 0.6586433260393874
parsing type:  8  for length:  4 <Buffer aa 55 aa 55 04 00 00 00 08 00 00 00 f7 ff ff ff>
parsing 8 <Buffer 03 00 00 00>
Carplay message:  undefined
test message 
received 0 event
undefined
received 0 {}
parsing type:  0  for length:  0 <Buffer aa 55 aa 55 00 00 00 00 00 00 00 00 ff ff ff ff>
starting background worker
/home/pi/Development/volvo-rtvi/node_modules/electron/dist/resources/scripts/dashboard.py
python: can't open file '/home/pi/Development/volvo-rtvi/node_modules/electron/dist/resources/scripts/dashboard.py': [Errno 2] No such file or directory
closing background worker
FATAL ERROR:  
/home/pi/Development/volvo-rtvi/node_modules/electron/dist/electron exited with signal SIGABRT
pi@raspberrypi:~/Development/volvo-rtvi $
rhysmorgan134 commented 1 year ago

Thanks for the detail, I believe I have figured it out, however I am also looking to implement audio via the browser rather than a bunch of ffplay instances, so will look to complete the two in the same merge!

Thanks for details, also will look into the PR on react-js-carplay!

rhysmorgan134 commented 1 year ago

Please give this one ago, v3 should have resolved these issues. This won't be backwards compatible with your custom set up, but I will get instructions sorted on how to implement it. It is worth the aggro as the performance is much better, and audio hugely more stable.

LRYMND commented 1 year ago

All in all the performance seems to have improved a lot!

In react-carplay you still need to remove the old websocket code from App.js so it actually executes properly.

One thing I found is that a click on the auto-icon is not triggering the quitReq event anymore.

One more thing that would be really helpful for multipage apps and that I implemented in my spinoff is having a state that checks, if the video stream contains any content which can be checked from App.js. This way you can bridge the time it takes between the "phone plugged" event and the actual start of the video stream with a loading message. (Or keep showing the "connect phone" message until the stream started.) I could look into this and make a proposal.

Another thing I did is adding app.commandLine.appendSwitch('disable-gpu'); at the top of main.js so I dont get the vsync errors anymore.

//Edit: I understand now how you use the socket to receive messages from the carplay module. This is really elegant. This way I can also check, whether the stream has started by just listening to it.

//Edit 2: I implemented everything into my app now. When Carplay was launched and I switch to another window and then back to carplay, the stream is hidden behind the "connect iphone to begin carplay" div, probably because I cannot pass the status prop to the carplay component anymore and carplay is initiated with status=false when rerendered. Would be awesome if you could change this up again! :)

Thank you very much for your ongoing efforts!

rhysmorgan134 commented 1 year ago

ah, glad you got it going, I will add a socket message on load to request a status update as to whether it is active or not!

There is a small update on react-js-carplay now which re-enabled the quit functionality from the button if you would rather receive the event from react-js-carplay itself rather than connecting manually, that will keep the carplay socket isolated (if that's what you would wish) I need to go through and do a clean up of all the unused props, but that can wait for now.

LRYMND commented 1 year ago

I think it might be convenient to be able to change the status remotely. Then I could initialize it with true and hide the "Connect iPhone...." div right away in order to display a custom message for the sake of modularity?

As a side note: You also mentioned in the other issue that you can hide the settings button by not passing a settings prop right? When doing this with the current version it throws an error that it's missing 'settings.fps'.

Again, thanks a lot!

rhysmorgan134 commented 1 year ago

It should be implemented now, if you grab the latest react-js-carplay and node-carplay you should be fine to navigate away and back.

The settings button is something I will implement to make it visible or not. However I may just allow a user to pass a react component as a prop to customise the screen. Only reason I am against it being external, is that it is a layer of complexity that will need support outside of the two packages. If a user wants to link in and grab status from socket.io then it is open for users to do that.

LRYMND commented 1 year ago

Implemented the new react-js-carplay 3.0.17 now and it seems like its not working. Here is what I'm doing:

I'm launching the app and the index is the carplay page. After the app launched, I don't see any content at all although Carplay was loaded. When I connect my phone and the stream starts, it switches to the "Connect iphone..." div. By listening to the socket I can see, that the video stream already arrives. Touch events are also registered. When I now turn carplay in the phonesettings off and back on again, it updates the Carplay component and the stream becomes visible, however only with poor performance. When I then navigate away and come back, I can see that the component is now making a new status req but in my case, it just switches from a black or hidden div to the "Connect iPhone..." div again. I'd have to turn carplay off an back on again in my phone settings so it then requests a new status again to reveal the video stream.

Hope it makes sense to you.

rhysmorgan134 commented 1 year ago

Take a look here, it is a skeleton project with nav working feel free to pull and run npm start electron-dev to test, I am guessing you have something wrong with the window load, I guess you are laying something over the top of carplay which is messing it up. Which is why it's easiest to leave carplay socket comms contained

https://github.com/rhysmorgan134/landrover-freelander2-can-app

LRYMND commented 1 year ago

Thank you very much for this example. It gives me a bit of headache though since my app is structured a bit different, but I also dont see why it wouldn't work. The behaviour your example shows is exactly what I would expect.

You have a home component in which you render the other components based on a switch. In my case, I'm using react-router to render the different components, one of which is CarplayWindow, in which I nest the carplay component. From my point of view it's as straight forward as it can be?

App.js:

const App = () => {

  const [showNav, setShowNav] = useState(true);
  const [settings, setSettings] = useState(null);
  const [startedUp, setStartedUp] = useState(false);

  function loadSettings(data) {
    console.log('loading settings...')
    if (data != null) {
      setSettings(data);
    }
  }

  useEffect(() => {
    if (settings != null) {
      console.log('settings loaded: ', settings);
      setStartedUp(true);
    }
  }, [settings])

  useEffect(() => {
    ipcRenderer.send('getSettings');
    ipcRenderer.send('wifiUpdate');
    ipcRenderer.on('allSettings', (event, data) => { loadSettings(data) });

    return function cleanup() {
      ipcRenderer.removeAllListeners('allSettings');
    };
  }, []);

  return (
    <>
    <CarplayAudio />
    {startedUp && <HashRouter>
      {
        showNav && <>
          <TopBar />
          <NavBar />
        </>
      }
        <Routes>
          <Route path='/dashboard' element={<Dashboard
            settings={settings}
          />} />
          <Route path='/' element={
          <CarplayWindow
            settings={settings}
          />} />
          <Route path='/settings' element={<Settings
            settings={settings}
            setSettings={setSettings}
          />} />
        </Routes>
    </HashRouter>}
    </>

  );
};

export default App;

CarplayWindow.js:

function CarplayWindow({ settings }) {

    const { ipcRenderer } = window;
    const navigate = useNavigate();

    const [status, setStatus] = useState(false);

    useEffect(() => {
        ipcRenderer.on('plugged', () => { setStatus(true); console.log('phone connected'); });
        ipcRenderer.on('unplugged', () => { setStatus(false); console.log('disconnected'); });
        ipcRenderer.on('quitReq', () => { leaveCarplay() });

        ipcRenderer.send('statusReq');
        return () => {
            ipcRenderer.removeAllListeners();
        };
    }, []);

    function leaveCarplay() {
        navigate('/dashboard');
        console.log('leaving carplay');
    }

    const reload = () => {
            console.log('reqReload test')
            //ipcRenderer.send('reqReload')
        }

        const touchEvent = (type, x, y) => {
            ipcRenderer.send('click', {type: type, x: x, y: y})
        }

        const openCarplay = (e) => {
            setStatus(true);
        }

        const dummy = (e) => {
            console.log('openModalReq')
            navigate('/settings');
        }

    return (
        <div className={`carplayWindow ${settings.theme}`} style={{ height: '100%', width: '100%'}}>
                <Carplay
                    settings={settings}
                    status={status}
                    touchEvent={touchEvent}
                    type={"ws"}
                    openModal={false}
                    openModalReq={dummy}
                    closeModalReq={dummy}
                />
        </div>
    );
}

export default CarplayWindow;
LRYMND commented 1 year ago

Got it running now. I got rid of the Hashrouter and used the switch approach from your app. It's still weird that it wasn't working before. I even moved the Carplay component up a level onto App.js but it still wouldnt work and everytime I navigate back I was locked behind the div. I wonder if it actually has something to do with the Router component but I'm just glad now that it works and that I don't have to fiddle around with your code anymore.

Thanks for your efforts!

rhysmorgan134 commented 1 year ago

No Problem, glad you got it working, when I get time I will set up a project with react-router and give it a test, it should work, if the component is unmounting and then mounting it should behave the same as launching the project.

Going to close this issue out now!