search

Samfox2 / homebridge-videodoorbell

Video-Doorbell plugin based on homebridge-camera-ffmpeg for Homebridge.
Apache License 2.0
95 stars 17 forks source link

Two way audio ? #25

Open Partizan7676 opened 5 years ago

Partizan7676 commented 5 years ago

Is it possible? No one camera makers even "Works with homekit" logo dont have two way audio feature. Somfy is announce, but sales is not starting yet. Is it homekit problem or not? Seems that happened when apple removes doorbell category from list...

llemtt commented 5 years ago

I have my video doorbell currently working.

https://github.com/KhaosT/homebridge-camera-ffmpeg/issues/174

Partizan7676 commented 5 years ago

@llemtt can you describe your hardware and software config?

llemtt commented 5 years ago

Production environment is:

Raspberry PI2 B

Linux TeoRasp 4.14.70-v7+ #1144 SMP Tue Sep 18 17:34:46 BST 2018 armv7l

node v8.11.1

npm -g -depth=0 list /usr/lib ├── gpio@0.2.10 ├── homebridge@0.4.45 ├── homebridge-camera-ffmpeg@0.1.8 ├── homebridge-videodoorbell@0.1.9 ├── npm@6.4.1 └── rpi-gpio@1.0.0

libasound2-dev

ffmpeg version N-91008-ge6114d21ac Copyright (c) 2000-2018 the FFmpeg developers built with gcc 6.3.0 (Raspbian 6.3.0-18+rpi1+deb9u1) 20170516 configuration: --arch=armel --target-os=linux --enable-gpl --enable-omx --enable-omx-rpi --enable-nonfree --enable-mmal --enable-libfdk-aac

        {
            "platform": "Video-doorbell",
             "cameras": [
                {
                    "name": "videodorbell test",
                    "port": 5005,
                    "button": true,
                    "videoConfig": {
                                            "source": "-f alsa -i hw:CARD=usbtv -f v4l2 -i /dev/video0",
                                            "stillImageSource": "-nostats -nostdin -timelimit 2 -analyzeduration 1 -f v4l2 -i /dev/video0 -frames:v 1",
                                            "maxStreams": 2,
                                            "maxWidth": 640,
                                            "maxHeight": 480,
                                            "maxFPS": 10,
                                            "vcodec": "h264_omx",
                                            "debug": true,
                                            "additionalCommandline": " ",
                                            "audio": "2way -filter:a volume=12dB -f alsa hw:CARD=ALSA,0"
                                      }
                }
            ]
        }

Clients (iPhones, iPad) all run iOS 12.0.1.

Partizan7676 commented 5 years ago

@llemtt what is usbtv?

llemtt commented 5 years ago

Fushicai USBTV007 Video Grabber [EasyCAP]

my doorbell hw is plain old analog video doorbell...

so video input is grabbed from /dev/video0, audio input is grabbed form (alsa) hw:CARD=usbtv, and audio ouput is (alsa) hw:CARD=ALSA,0 to the PI phone jack then connected to doorbell audio input

paqpaqpaq commented 5 years ago

@llemtt So, if audio is working on the ffmpeg stream, and with"audio": "2way" the intercom pops up without crashing, it would suffice to install libasound2, and set "audio": "2way -filter:a volume=12dB -f alsa hw:CARD=ALSA,0"? Or is your setup actually hardwired?

llemtt commented 5 years ago

@paqpaqpaq

In order to use alsa, ffmpeg must be compiled with libasound2-dev installed, so first install the lib then compile ffmpeg.

Next I suggest to try first that ffmpeg works by sending some audio file to the pi audio jack like this:

ffmpeg -i AUDIOFILE -f alsa hw:CARD=ALSA,0

If everything is ok then with my POC ffmpeg.js and this configuration:

"audio": "2way -f alsa hw:CARD=ALSA,0"

the intercom from the iPhone/iPad will be sent to the pi audio jack.

paqpaqpaq commented 5 years ago

thanks.

So, there needs to be a hardwired connection between the audio jack from the pi to the cam, regardless?

llemtt commented 5 years ago

@paqpaqpaq

I don't know, it depends on your cam!

My doorbell is an old "analog" type so I connected the audio jack to his "analog" audio input.

For instance I have an old version D-Link DCS-942L, but it supports 2way audio only through a proprietary audio format (ACAS) so it actually works only with their app...

If your cam accept an http or rtp or any "standard" input stream, you can put the necessary ffmpeg "output" configuration into "audio" property in place of "-f alsa hw:CARD=ALSA,0".

RavenSystem commented 5 years ago

@llemtt I am trying to setup a video doorbell. Video works well, audio input works but it sounds a little fast than normal, and audio output (from iPhone/iPad to RPi) is not good, it sounds slow and very distorted. I have not idea where is the problem.

I am using a Raspberry Pi 3 Model B+, with its camera under /dev/video0, audio input with an USB device and audio output to RPi audio jack.

I installed homebridge-videodoorbell@0.1.9 and replaced ffmpeg.js with your PoC file from here https://github.com/KhaosT/homebridge-camera-ffmpeg/issues/174.

My config.json is:

{
    "bridge": {
        "name": "DoorBell-Bridge",
        "username": "AC:54:3D:B3:12:34",
        "port": 51826,
            "pin": "123-45-678"
    },

    "platforms": [
            {
            "platform": "Video-doorbell",
            "cameras": [
                {
                    "name": "doorbell",
                    "port": 5005,
                    "button": true,
                    "videoConfig": {
                        "source": "-f v4l2 -i /dev/video0 -f alsa -i hw:CARD=AK5371,0",
                        "stillImageSource": "-nostats -nostdin -timelimit 2 -analyzeduration 1 -f v4l2 -i /dev/video0 -frames:v 1",
                        "maxStreams": 2,
                        "maxWidth": 640,
                        "maxHeight": 480,
                        "maxFPS": 15,
                        "vcodec": "h264_omx",
                        "maxBitrate": 90,
                        "debug": true,
                        "additionalCommandline": " ",
                        "audio": "2way -f alsa hw:CARD=ALSA,0"
                    }
                }
            ]
        }
    ]
}

Many thanks for your help.

dosordie commented 5 years ago

@llemtt Im trying to get Audio Output working. ffmpeg -i test.mp3 -f alsa hw:0,0 works fine. But i got an Error when using Home App (Adress already in use)

Feb 11 01:40:19 debian homebridge[1653]: ffmpeg -nostats -nostdin -re -i rtsp://192.168.10.37:554/Streaming/channels/101 -map 0:0 -vcodec libx264 -pix_fmt yuv420p -r 10 -f rawvideo -loglevel verbose -vf scale=1280:720 -b:v 299k -bufsize 299k -maxrate 299k -payload_type 99 -ssrc 13533811 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params vPbTvvYUrwRrn1qHY/ByxUtpUzzPXZfrR5A8VMhM srtp://192.168.10.140:54220?rtcpport=54220\&localrtcpport=54220\&pkt_size=1316 -map 0:1 -acodec libfdk_aac -profile:a aac_eld -flags +global_header -f null -ar 16k -b:a 24k -bufsize 24k -ac 1 -payload_type 110 -ssrc 15163982 -f rtp -srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params BNWdkG7KPZkEVCCfbN4W36jrS79j2155Y1SxUVwt srtp://127.0.0.1:59118?rtcpport=59118\&localrtcpport=9998\&pkt_size=188 Feb 11 01:40:19 debian homebridge[1653]: [Speaker] ffmpeg -v error -nostats -nostdin -max_ts_probe 0 -protocol_whitelist file,udp,rtp -i Tür-Cam_speaker.sdp -f speaker.mp4 Feb 11 01:40:19 debian homebridge[1653]: proxy listening 0.0.0.0:59118 Feb 11 01:40:19 debian homebridge[1653]: ffmpeg version N-93107-g6de396c216 Feb 11 01:40:19 debian homebridge[1653]: Copyright (c) 2000-2019 the FFmpeg developers Feb 11 01:40:19 debian homebridge[1653]: built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516 Feb 11 01:40:19 debian homebridge[1653]: configuration: --enable-gpl --enable-nonfree --enable-libfdk-aac --enable-libgsm --enable-libmp3lame --enable-libtheora --enable-libvorbis --enable-libx264 --enable-libxvid --enable-zlib --enable-postproc --enable-swscale --enable-pthreads --enable-libdc1394 --enable-version3 --enable-libopencore-amrnb --enable-libopencore-amrwb Feb 11 01:40:19 debian homebridge[1653]: libavutil 56. 26.100 / 56. 26.100 Feb 11 01:40:19 debian homebridge[1653]: libavcodec 58. 47.100 / 58. 47.100 Feb 11 01:40:19 debian homebridge[1653]: libavformat 58. 26.101 / 58. 26.101 Feb 11 01:40:19 debian homebridge[1653]: libavdevice 58. 6.101 / 58. 6.101 Feb 11 01:40:19 debian homebridge[1653]: libavfilter 7. 48.100 / 7. 48.100 Feb 11 01:40:19 debian homebridge[1653]: libswscale 5. 4.100 / 5. 4.100 Feb 11 01:40:19 debian homebridge[1653]: libswresample 3. 4.100 / 3. 4.100 Feb 11 01:40:19 debian homebridge[1653]: libpostproc 55. 4.100 / 55. 4.100 Feb 11 01:40:19 debian homebridge[1653]: [tcp @ 0x55668eafba80] Feb 11 01:40:19 debian homebridge[1653]: [Speaker] [udp @ 0x556bff622040] Feb 11 01:40:19 debian homebridge[1653]: [Speaker] bind failed: Address already in use Feb 11 01:40:19 debian homebridge[1653]: [Speaker] [udp @ 0x556bff6222c0] Feb 11 01:40:19 debian homebridge[1653]: [Speaker] bind failed: Address already in use Feb 11 01:40:19 debian homebridge[1653]: [Speaker] Tür-Cam_speaker.sdp: Invalid data found when processing input

Is there a Problem with : proxy listening 0.0.0.0:59118 ?

Greeds Dominik

RavenSystem commented 5 years ago

I found my problem with distorted audio. RPi audio jack and camera don't work well when using kernel module to get /dev/video0. I disconnected camera, unloaded kernel module and deactivated in rasp-config; then I created a dummy video input, and now 2-way audio works. Actually, I am waiting for a new USB camera from Amazon. I hope with the new cam, all works fine.

RavenSystem commented 5 years ago

@dosordie I suggest to change in config.json "Tür-Cam" name for other without any special character, like "mycam". Then, you can rename it into Home App.

dosordie commented 5 years ago

@RavenSystem Changed the Name in Config.json. Same Problem

srtp://192.168.10.140:57575?rtcpport=57575&localrtcpport=57575&pkt_size=1316

In my network there is no device with the address 192.168.10.140 - Could this be a Problem? But I do not know where to change that

llemtt commented 5 years ago

@dosordie

It's very strange because in my code the proxy is hard wired to listen on port 9999.

Anyway I never tried it on debian (pc?) so nodejs udp sockets can behave a little different there.

192.168.10.140 must be your iPhone/iPad/AppleTV address!

RavenSystem commented 5 years ago

@llemtt Your PoC is amazing! Have you an improved version or are you working on it?

RavenSystem commented 5 years ago

@llemtt Another question: is there any way to increase 10 seconds speaker timeout?

Thanks!

llemtt commented 5 years ago

@llemtt Your PoC is amazing! Have you an improved version or are you working on it?

Many thanks. Sorry but nowadays I don't have time to work on it, I have my videodoorbell working so I will get back to it only if I would need to change/fix something.

Consider also that code is no longer up to date to latest version of ffmpeg-camera and requires some rework to eventually merge.

@llemtt Another question: is there any way to increase 10 seconds speaker timeout?

Not an easy way, that timeout (rtp/udp timeout) is hardcoded into ffmpeg and cannot be passed as a parameter so one can:

RavenSystem commented 5 years ago

@llemtt

Not an easy way, that timeout (rtp/udp timeout) is hardcoded into ffmpeg and cannot be passed as a parameter so one can:

  • modify own instance of ffmpeg
  • ask ffmpeg maintainers to let timeout be passed as a parameter (I have already done that anyway... but don't think it will see any future soon)
  • generate dummy rtcp packets in the proxy while the true ones are not received (just an idea)

Do you know where is hardcoded into ffmpeg?

Update: https://trac.ffmpeg.org/ticket/2415

dosordie commented 5 years ago

@llemtt

@dosordie

It's very strange because in my code the proxy is hard wired to listen on port 9999.

Anyway I never tried it on debian (pc?) so nodejs udp sockets can behave a little different there.

Is there a way to change the port (by variable or in the script File?) Yes it´s a Debian on a VM. Ok, the IP is from my iPhone...

llemtt commented 5 years ago

@RavenSystem

Do you know where is hardcoded into ffmpeg?

define READ_PACKET_TIMEOUT_S 10

in file rtsp.c

Update: https://trac.ffmpeg.org/ticket/2415

exactly! voted (+1) on that ticket

@dosordie

port used are 9998 and 9999, search ffmpeg.js for "999" you'll find all occurrences

port 9999 is used inside the .sdp speaker file that ffmpeg uses as the speaker input description

dosordie commented 5 years ago

@llemtt I do not think the port is the problem, right? The error message is "bind failed: Address already in use" Can the IP? Address be set anywhere in config or ffmpeg.js?

llemtt commented 5 years ago

@dosordie Actually is a port problem, systemwise you can have just one socket bound to the same port although, depending on OS, you can bound the same port on different IP addresses (for instance if you have more than one interface or an interface with multiple IPs).

I can think two things

dosordie commented 5 years ago

@llemtt I found the problem. It was "Webmin" which belonged to the port 10000. Probably that has the port 9999 reserved.

Now it works good. Top work! Thank you!

burnbrigther commented 5 years ago

How could this work with a camera like the Axis 3343 which supports 2-way audio. From the documentation, it shows: QuickTimeTM supports G.711 and AAC audio encoding. The following paths can be used:

RavenSystem commented 5 years ago

@llemtt Please, I hope you can help me. I have an USB camera that works under /dev/video0, an USB mic that works with ALSA, and the RPi audio jack to use the speaker. I am using second version of your working PoC, that I think is the last.

But I have 2 problems with audio: a) Audio from RPi to iPhone: The audio is heard with a little higher frequency. b) Audio from iPhone to RPi: The audio is heard with a big lower frequency.

I don't know if it's necessary to customize ALSA setup in any way, or do other things.

And another question: I only use homebridge-videodoorbell@0.1.9, but I see that you use homebridge-camera-ffmpeg@0.1.8 too in your setup. Is it necessary?

Many thanks for your help!

llemtt commented 5 years ago

@RavenSystem I remember that audio one octave higher or lower was something I experienced at the beginning but it disappeared once I moved on with configurations. I think it was related to an issue of 8 vs 16 bits pcm between ffmpeg and alsa.

I went directly to the hw card also because using default + alsa configuration wasn't completely ok.

What happens if you record the speaker audio on a file with "2way speaker.mp4" and play back to phone jack with ffmpeg? Is the pitch correct?

For my doorbell I use homebridge-videodoorbell@0.1.9 but I was testing another two-way cam with homebridge-camera-ffmpeg@0.1.8, it's not necessary.

RavenSystem commented 5 years ago

@llemtt I am going to do some records but, can you tell me how to manipulate hardware audio card directly, please?

burnbrigther commented 5 years ago

@llemtt I don't see the 2way code getting activated with video-doorbell from ffmpeg.config. I also don't see anything to make outbound communication from the phone. Am I missing something?

toshibochan commented 5 years ago

I have this error please help.

[NULL @ 0x2c88330] Requested output format 'alsa' is not a suitable output format hw:CARD=ALSA,0: Invalid argument

pi@iDomoPi3Plus:~$ ffmpeg -i dingdong.mp3 -f alsa hw:CARD=ALSA,0

ffmpeg version N-93878-g81acc9adbf Copyright (c) 2000-2019 the FFmpeg developers built with gcc 6.3.0 (Raspbian 6.3.0-18+rpi1+deb9u1) 20170516 configuration: --prefix=/usr/local --arch=armel --target-os=linux --enable-omx-rpi --enable-nonfree --enable-gpl --e nable-libfdk-aac --enable-mmal --enable-libx264 --enable-decoder=h264 --enable-network --enable-protocol=tcp --enable- demuxer=rtsp libavutil 56. 28.100 / 56. 28.100 libavcodec 58. 52.101 / 58. 52.101 libavformat 58. 27.103 / 58. 27.103 libavdevice 58. 7.100 / 58. 7.100 libavfilter 7. 53.101 / 7. 53.101 libswscale 5. 4.101 / 5. 4.101 libswresample 3. 4.100 / 3. 4.100 libpostproc 55. 4.100 / 55. 4.100 [mp3 @ 0x2c85210] Skipping 417 bytes of junk at 45. [mp3 @ 0x2c85210] Estimating duration from bitrate, this may be inaccurate Input #0, mp3, from 'dingdong.mp3': Metadata: title : http://www.freesfx.co.uk Duration: 00:00:01.85, start: 0.000000, bitrate: 129 kb/s Stream #0:0: Audio: mp3, 44100 Hz, stereo, fltp, 128 kb/s [NULL @ 0x2c88330] Requested output format 'alsa' is not a suitable output format hw:CARD=ALSA,0: Invalid argument

llemtt commented 5 years ago

@toshibochan check first that libasound2-dev is installed, if not install

sudo apt-get install libasound2-dev

and recompile ffmpeg

superbraz commented 4 years ago

How could this work with a camera like the Axis 3343 which supports 2-way audio. From the documentation, it shows: QuickTimeTM supports G.711 and AAC audio encoding. The following paths can be used:

* rtsp:///axis-media/media.amp
  I have been able to use applications on the ipad which support 2 way audio in the past.  And now I only have 1 way audio from the camera, listening.
  Thank you guys for your work on this.  Will be great to put together a real two-way audio/intercom system.

i am also interested - i have an AXIS A8207-VE and i wand to speak with people in front of the door

superbraz commented 4 years ago

nobody?

llemtt commented 4 years ago

@superbraz the feature is still experimental and has not been merged yet

I successfully run it and I have recently fixed some codec details and maybe one day I'll try a merge, in the meantime one has to substitute the ffmpeg.js file with my version.

Anyway first preliminary step to configure for two way is to reproduce whatever audio file on your camera using ffmpeg, can you do that?

superbraz commented 4 years ago

how can i do that?

llemtt commented 4 years ago

@superbraz can you post a link to your camera documentation?

superbraz commented 4 years ago

https://www.axis.com/products/axis-a8207-ve/support-and-documentation

this one?

llemtt commented 4 years ago

sorry no, you need VAPIX documentation, as a registered user you should obtain that

anyway your ffmpeg command should be like this

ffmpeg -i {an audio file to play} -f mulaw http://{servername}/axis-cgi/audio/transmit.cgi

burnbrigther commented 4 years ago

Log in to your MyAxis account, enable developer access, then look here: https://www.axis.com/vapix-library/

burnbrigther commented 4 years ago

@superbraz It has been some time since I was playing around with audio with my 3343. I have since moved to a Pi 4b with my P3375-VE and use h264_omx for video. I've learned a couple of things along the way - you will have better results if your request sends sizing.

       {
                    "name": "FRONT DOOR",
                    "manufacturer": "Axis",
                    "model": "P3375-VE",
                    "motion": false,
                    "doorbell": false,
                    "videoConfig": {
                        "source": "-rtsp_transport tcp -r 18 -i rtsp://XXXX:XXXX@192.168.1.231:554/axis-media/media.amp?streamprofile=Media?tcp&resolution=1280x720",
                        "stillImageSource": "-i http://root:n2trips@192.168.1.231/axis-cgi/jpg/image.cgi",
                        "maxStreams": 3,
                        "maxWidth": 1280,
                        "maxHeight": 720,
                        "maxFPS": 20,
                        "vcodec": "h264_omx",
                        "videoFilter": "none",
                        "additionalCommandline": "-loglevel verbose",
                        "audio": false,
                        "debug": true
                }

This is what I last used for audio and could only hear what was being said, I couldn't get the mic icon and talking to other side to fully work. This was with a previous version of @llemtt ffmpeg.js:

"audio": "2way -c:a pcm_mulaw -ac 1 -ar 16000 -b:a 128k -vcodec h264_omx -multiple_requests 1 -reconnect_at_eof 1 -reconnect_streamed 1 -content_type audio/basic -f mulaw http://XXXX:XXXXX@192.168.1.231/axis-cgi/audio/transmit.cgi -report",

I still have not yet figured out how to get bi-directional working. Maybe @llemtt 's new code fixes things? Maybe it is my POE-analog speaker/mic set up which gets the audio from the camera itself. Let me know how you get on. I will be experimenting more with this. The goal is to be able to have two-way conversation on my ipad or iphone with someone at the door through my analog set up. In theory, this set up shouldn't be much different than running this through the Pi. Just in case I decide to experiment with that route, I did pick up a USB input for the Pi with audio in and out.

burnbrigther commented 4 years ago

@llemtt is your ffmpeg.js.txt dated Aug 28, 2018 from #174 your latest one? If not, could you link your latest one please? Thank you.

superbraz commented 4 years ago

@llemtt ok, i am on the vapix doc right now - it is a very long page, where can i send it to you?

@burnbrigther my axis cams are working finde in homebridge, still image, video and audio (one way, from the cam)

{ "name": "Indoor", "manufacturer": "AXIS", "model": "M1065-LW", "serialNumber": "ACCXXXXXXXX", "videoConfig": { "source": "-re -i rtsp://USER:PASSWORD@192.168.1.XXX/axis-media/media.amp?resolution=1280x720", "stillImageSource": "-i http://USER:PASSWORD@192.168.1.XXX/axis-cgi/jpg/image.cgi", "maxStreams": 3, "maxWidth": 1280, "maxHeight": 720, "maxFPS": 20, "maxBitrate": 200, "vcodec": "libx264", "packetSize": 1316, "audio": false, "debug": false }

llemtt commented 4 years ago

@llemtt is your ffmpeg.js.txt dated Aug 28, 2018 from #174 your latest one? If not, could you link your latest one please? Thank you.

ffmpeg.js.txt

attached my current version

llemtt commented 4 years ago

@llemtt ok, i am on the vapix doc right now - it is a very long page, where can i send it to you?

There should be a document called "vapix audio" (I have a very old outdated version), the relevant part (in my version) is only chapter "4.4 Transmit audio data".

Have you tried the ffmpeg command?

superbraz commented 4 years ago

i dont know how... my homebridge is running in a docker container on synology nas

Transmit audio data

Transmit a singlepart audio data stream: Check what audio formats your Axis product can transmit. For a complete list of audio formats supported by VAPIX® see Audio compression formats.

http:///axis-cgi/param.cgi?action=list&group=Properties.Audio.Decoder

Access control: viewer

Method: POST

Syntax: Copyhttp:///axis-cgi/audio/transmit.cgi

Content-Type:

llemtt commented 4 years ago

@superbraz sorry but I have no direct experience with dockers and NAS, but for testing purposes you can use ffmpeg from any unix command line (even win but not recommended...)

burnbrigther commented 4 years ago

@llemtt Thank you for your latest code. I am using the latest version of homebridge-ffmpeg which has changed significantly with lots more code. I just did a comparison of the newest ffmpeg.js and yours and there are lots of differences. Would it be possible to merge your code with latest ffmpeg.js? I could maybe try to do this myself, if I know exactly what is different with your code only. Are your changes only everything marked "2way"?

llemtt commented 4 years ago

@burnbrigther I'm sorry but I don't think my code is completely compatible with the newest homebridge-ffmpeg-camera, 2 way audio requires following Apple guidelines better than they do

anyway major changes are lines 317-339, 377-469 and 492-494

burnbrigther commented 4 years ago

@llemtt Thanks - is your ffmpeg.js merged with latest homebridge-videodoorbell?

llemtt commented 4 years ago

I currently use my ffmpeg.js with latest version of homebridge-videodoorbell but it isn't merged