search

Brandawg93 / homebridge-nest-cam

View your Nest cams in HomeKit using Homebridge.
https://www.npmjs.com/package/homebridge-nest-cam
GNU General Public License v3.0
211 stars 27 forks source link

No Audio #1

Closed Brandawg93 closed 4 years ago

Brandawg93 commented 4 years ago

I have looked into the audio issue and have two choices:

  1. I can add audio support and force everyone with ffmpeg to recompile with libfdk_aac codec.

  2. I can leave audio support out for a much easier setup.

For now, I am going with #2. Audio may work if you uncomment out this line of code, but it doesn't work for everyone. If anyone can find a better way, please let me know here or submit a pull request.

jake-at-work commented 4 years ago

We’re you successful in getting audio working with that line? I get an error from ffmpeg that there is no output on that stream. I feel like there a few things missing. First in the PlaybackBegin the audio codec in the stream I’m getting is 0 (speex). Then in the PlaybackPacket I feel like there should be a line to feed the audio packet to ffmepg.stdin.

Brandawg93 commented 4 years ago

I was not successful getting audio working with that line on my pi4 because it doesn’t have libfdk_aac. I’m trying to figure out a way to not have to encode at all. I know that there is an iOS stream profile for the cameras which is what the iOS app uses. If I can get that setup correctly, audio would just work because it’ll be just passing everything straight through to HomeKit.

Brandawg93 commented 4 years ago

@pivotal-jbarrett you're definitely right. Something is wrong here. It's never even sending audio to ffmpeg.

rollinnerdy commented 4 years ago

@pivotal-jbarrett you're definitely right. Something is wrong here. It's never even sending audio to ffmpeg.

Correct. I added a code to write the audio to ffmpeg.stdio[3] and when spawning to include a 4th pipe. The in the audio flags -i pipe:3. ffmpeg sees bytes on that pipe but can't figure out what they are.

Do you have any insight into why the video appends [0, 0, 0, 1] to the buffer and if we need to do the same with audio?

Another interesting thing, on the StartPlayback no matter what profile I specify the response is always a redirect to h264 with speex. If I change the profile not found action to next then a I get h264 and opus. I haven't found any combinations yet that give h264 and aac.

Do you have any idea where the specs for the the Nest camera streaming service (nexus?) are? Perhaps those can shed some light.

Brandawg93 commented 4 years ago

Unfortunately, I'm not an expert on ffmpeg and audio/video codecs, but I am good at javascript, so I was able to reverse engineer the authentication protobufs from the javascript on https://home.nest.com. The specs for streaming are in there as well. I just don't know what to do with the streaming specs. Here is how to find them.

  1. Go to https://home.nest.com
  2. Open Chrome dev tools and go to the network tab
  3. Change the network filter from All to WS
  4. Find the first nexustalk packet and select initiator
  5. Select the first main-....js file.

Somewhere around there is all the protobufs for authentication, streaming, and even other things like the hello doorbell.

Brandawg93 commented 4 years ago

I've added better code readability with more stuff from the protocol buffers in cb6880190275db8e2268ee508421e2dc6ccab2d5. I was eventually able to get a stream of OPUS audio to write to ffmpeg stdin, but ffmpeg doesn't know what it is and I suspect it has something to do with this line. The first problem is to get ffmpeg to recognize the OPUS input and the second is to make ffmpeg distinguish between raw H264 packets and OPUS packets because right now, they're all being written to stdin with no distinguishment.

Brandawg93 commented 4 years ago

I was able to get static to play with the cameras. The code is on the audio tree. It's using a s16le audio format for input which is not correct, but it proves that audio may work. I haven't found out how to get the OPUS decoder to work yet, but that's the last piece of the puzzle.

github-actions[bot] commented 4 years ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

dgreif commented 4 years ago

Hey @Brandawg93, I'm the maintainer of homerbidge-ring. Glad to see another dev actively working on homebridge cameras! Since it looks like you hit a lot of the same roadblocks as I did on audio, I wanted to let you know I found a fairly good solution for the Ring implementation. @oznu was kind enough to publish an npm package with pre-built binaries of ffmpeg with the aac-eld codec included. This means your users don't have to install ffmpeg manually, and don't have to worry about compiling it with aac-eld (which seems to really trip most users up). You can check out how I'm using it here: https://github.com/dgreif/ring/blob/dc4218dd0e1a5a3ef2bdd7e40bc74e19f3a31ba2/homebridge/ring-platform.ts#L148. I'm currently only supporting 1-way audio, but hopefully that helps get you up and running!

Brandawg93 commented 4 years ago

For future reference, I've learned a few things while converting the plugin to typescript:

I'm still trying to figure out how to make ffmpeg understand the aac packets, but that is the last piece of the puzzle.

Brandawg93 commented 4 years ago

Audio is now supported in 2.0.0.