Camera accessory

[gerswin]

is possible?

[ignalex]

all fixes applied - same config as was working before. stream starts. it is not copied into log after stopping python but its there:

[ignalex]

@aptonline The authentication is taken care of for you. Does the ffmpeg process start?

So it looks like its starting but no output:

ffmpeg version 3.2.12-1~deb9u1 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
  configuration: --prefix=/usr --extra-version='1~deb9u1' --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --enable-gpl --disable-stripping --enable-avresample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libebur128 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libmp3lame --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-omx --enable-openal --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libopencv --enable-libx264 --enable-shared
  libavutil      55. 34.101 / 55. 34.101
  libavcodec     57. 64.101 / 57. 64.101
  libavformat    57. 56.101 / 57. 56.101
  libavdevice    57.  1.100 / 57.  1.100
  libavfilter     6. 65.100 /  6. 65.100
  libavresample   3.  1.  0 /  3.  1.  0
  libswscale      4.  2.100 /  4.  2.100
  libswresample   2.  3.100 /  2.  3.100
  libpostproc    54.  1.100 / 54.  1.100
Unknown input format: 'avfoundation'
[hap_server] Request PUT from address '('192.168.1.18', 49284)' for path '/characteristics'.
[hap_server] Set characteristics content: {'characteristics': [{'aid': 1, 'iid': 15, 'value': 'ARUCAQABEOZy2nWOE0YXnNH6hL2pYmQ='}]}
[characteristic] client_update_value: SelectedRTPStreamConfiguration to ARUCAQABEOZy2nWOE0YXnNH6hL2pYmQ=
[camera] Set stream config request: 0
[camera] [e672da75-8e13-4617-9cd1-fa84bda96264] Stopping stream.
[hap_server] 192.168.1.18 - "PUT /characteristics HTTP/1.1" 204 -

Also seeing Unknown input format: 'avfoundation' in red if thats significant?

same was the case for me - you probably from raspberry pi = I coulnd't find how to install avfoundation on it I had to change to another codecs (look above in this issue - there was working replacement of avfoundation with video4linux2) but no good as of right now :)

[ikalchev]

@aptonline Can you copy the ffmpeg command and start it on its own - but change the address to your local address and remove the local port http param. Try to make the command work on its own for your platform, e.g. the avfoundation is for mac OS, try video2linux2 (see ignalex's earlier post where he got it working)

[ignalex]

@ikalchev trying to reduce resolution and bitrate etc with no luck...

can you confirm - i noticed the ffmpeg syntax changed quite significantly from the previous (not only v_port notation) but also a few parameters related to codecs. did you changed it cause it is must or it is a copy from another implementation source? also localrtcpport is now using v_port but before it was the another local_video_port (although could be same value?)

[ikalchev]

Yes, I made changes to it, but they shouldn't be that many. I am currently going through the old command and trying to find the diffs - so far:

• An extra -g 300

Let me go through the pull request for camera_base.

[ikalchev]

I that's the one that worked: https://github.com/ikalchev/HAP-python/pull/98/commits/7612785f10fa7390037f08db1ccb36addca49d8c

I think the only difference is the pix_fmt. @ignalex Can you try with the referenced cmd?

[ignalex]

I that's the one that worked: 7612785

I think the only difference is the pix_fmt. @ignalex Can you try with the referenced cmd?

nope, no change. I also had it out of my working config so it can't affect.

[ikalchev]

What fresh hell is this...

@ignalex Do you happen to keep the setup, i.e. the code form camera_base at the working commit?

[ignalex]

nope :( got the config ffmpeg (in the issue above :) ) ... give me 5, I will try to dig it out from snapshot ...

[ikalchev]

In the meantime, I managed to get the normal 30 fps, but still not streaming. It's almost 1 am here, will catch up tomorrow. We will get over this eventually :)

[ignalex]

ok... I hard reset to the last working commit ( 7612785f10fa7390037f08db1ccb36addca49d8c ) in camera_base. added my fixes for ffmpeg and made it work. HAP-python.zip

attaching zip of all the code. gotta be not too easy to check the differences :) ) also don't know if raspi's ffmpeg codec will work for your mac :)

let me know any progress. cheers

[cdce8p]

@ignalex Can you open a PR with your changes? That's what Github is for in the end 😄

[ignalex]

@ignalex Can you open a PR with your changes? That's what Hithub is for in the end 😄

not yet. it is not 'progressive' changes but rather 'regressive'. when we understand what is the cause of non streaming, I will add extra config options for raspi ...

[ignalex]

hey @ikalchev I got 2 instances comparing side by side (both modified for ffmpeg). one from old (working) camera_base, another - from updated dev (not working one).

the very interesting thing I see is the ip address is different.

this is dev - bad

this is camera_base - old working my iPhone sits on ip 13, HAP-server (RPi) is on ip 18 the ip 5 (from log of camera_base - which works) is my apple-TV. this makes me really confused. I know for sure I haven't configured ip 5 anywhere. no idea where HAP server picked it up (successfully).

i tried to force inject ip 5 into ffmpeg config line (dev branch) but no luck. maybe should try somewhere else, but you have address configured and inherited in many places ...

hope it gives you some crumbles :) night time for me now cheers A

[ignalex]

any updates ?

[ikalchev]

Sorry, didn’t have the time but I will work on this in the morning (which is after 8 hours).

The debug you did should definitely help.

[ikalchev]

@ignalex The ip 5 you see in the working case is sent from iOS, are you sure that's not the address of the iPhone? (i.e. the IP in the srtp URL is sent from iOS as the address to which to stream)

Another question: I see that in the non-working case the picture format is way higher (1280x720) - used to be 640x360. Have you tried lowering these in the camera_main.py?

[ikalchev]

These are the difference between the two commands (I redacted the ports, addresses and crypto):

ivankalchev@IK-Pro:~/Dev/github/HAP-python$ icdiff dev.txt camera_base.txt 
dev.txt                                            camera_base.txt                                           
-f                                                 -f                                                
video4linux2                                       video4linux2                                      
-input_format                                      -input_format                                     
h264                                               h264                                              
-video_size                                        -video_size                                       
1280x720                                           640x360              <<<        
-framerate                                         -framerate                                        
20                                                 20                                                
-i                                                 -i                                                
/dev/video0                                        /dev/video0                                       
-vcodec                                            -vcodec                                           
---                                                ---                                               
-ssrc                                              -ssrc                                             
1                                                  1                                                 
-f                                                 -f                                                
rtsp                                               rtsp                                              
-b:v                                               -b:v                                              
299k                                               132k          <<<                                   
                                                   -bufsize          <<<
                                                   132k               <<<
-payload_type                                      -payload_type                                     
99                                                 99                                                
-f                                                 -f                                                
rtp                                                rtp                                               
-srtp_out_suite                                    -srtp_out_suite

Can you try with decreased resolution settings and, if that doesn't work, try removing -bufsize 132k

Edit: I meant "try adding -bufsize 132k"

[ignalex]

@ikalchev I had 2 runs in within 5 min. iPhone won't change IP. and 5th on the router has been registered for the TV. [I will double check again]

What is the streaming mechanics? does ffmpeg make the stream to the iPhone IP (requesting device) directly? I was thinking... - the Home Kit requires apple server (home-Pod, apple-TV, MAC(?) ... apple device). if raspi itself will create a stream - it has to handle routing / firewall etc. (and my feeling, in the general case the stream will work only when iPhone only in the local wifi in this case???) but I tried streaming when wifi was turned off on iphone, and it could find the route. it means something helps raspi to pass firewalls and redirecting ports (I don't have any specific ports open on my router or redirecting ports). which potentially could mean that by the design it could require to direct the stream to the apple device (server, in my case AppleTV), which will handle the authentication within your apple-account and will allow iPhone to rich stream even when it is outsider the local network. So the question - are you sure the stream should be directed to iphone, not to the apple-server device ?

with resolution > I will make sure it is downgraded + will check the buf size.

cheers A

[ikalchev]

This is a good point - Unfortunately I can't test remote HAP(or however it is called).

But either way, this address is the destination address for the stream. It could well be that in case you have set up an iPad or Apple TV as a home hub, iOS will pass the address of the hub itself. This is so the hub will act as a gateway to the outside world. In can you don't have a hub, then the communication is directly from the pi to the iPhone, because Apple's assumption is that you are in a Home LAN network.

About the resolution: there may be an issue there as I just tried to change mine to a lower setting and it kept at 1280x.. so I guess I am not setting this correctly. Will debug some more now.

[ignalex]

just ran the working camera_base and it now streams to 13 (iPhone wifi) or 10 (which is Home-Pod - if from outside (4G). so it looks really smart - picking local IP when within one, or apple-server's as a gateway if it is from outside. will hardcode resolution into ffmpeg of dev branch and will come back to you...

[ikalchev]

If you change the settings of the camera you will need to re-add the accessory because iOS will not fetch the new available settings. Or you can call driver.config_changed to notify iOS the device capabilities have changed, but iOS will still need some time to fetch these anew.

[ignalex]

If you change the settings of the camera you will need to re-add the accessory because iOS will not fetch the new available settings. Or you can call driver.config_changed to notify iOS the device capabilities have changed, but iOS will still need some time to fetch these anew.

always do - delete old accessory.state file and pair again. - even switching branches won't allow to use it from another's

[ignalex]

changed resolution > no luck (it uses 13 IP as from another's branch) deleted buffer - still no luck. stream is there.

[ikalchev]

Similar issue btw: https://github.com/gozoinks/homebridge-camera-ffmpeg-ufv/issues/8

So based on your experiments, the command itself seems OK - something else is messed up. I will rebase and debug. Thanks mate!

[ignalex]

not exactly same - they said the service crashes. in our case it goes well :)

made a few changes / tests > no results. I ran out of ideas mate :)

[ikalchev]

In the worst case I will rollback and start from the last working state. Will do that later today.

[ikalchev]

Folks, I won't be able to work on this today, continuing tomorrow.

[ikalchev]

@ignalex Unfortunately I cannot make it work on my machine even on the commit that is working for you (pairs OK, starts ffmpeg OK, but stays at Loading in the Home app). This makes debugging very tricky for me.

So here is my plan:

• Will rebase to the commit that is working for you
• Apply only minor improvements (mainly the option for the user to give a command)
• You test if it works and if so
• I will release a version to master and pip

In this way we can get it to more folks.

What do you think?

[ignalex]

@ikalchev there was one important change even on top of that commit - it was in the history of this issue. make sure you change this and test before you rebase.

and if still need to rebase- lets do it on camera_base repo and do small changes testing on your side and my side a few iterations until found the glitch and only after that PR to dev. cheers

[ikalchev]

Yes exactly - will do it on camera_base.

BTW the rtsp flag is not needed - the spec requires RTP or SRTP. I have tried it though - no luck.

Finally, the working version is at 7612785f10fa7390037f08db1ccb36addca49d8c right?

[ignalex]

Commit version Looks right although can’t check now from my phone - I put it in the comments somewhere up. Without rtsp is didn’t work on my Pi. Maybe some others paeans interfere with that one

Sent from my iPhone

On 15 Oct 2018, at 08:01, Ivan Kalchev notifications@github.com wrote:

Yes exactly - will do it on camera_base.

BTW the rtsp flag is not needed - the spec requires RTP or SRTP. I have tried it though - no luck.

Finally, the working version is at 7612785 right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ikalchev]

Folks, awesome news - I managed to show the stream on my iPhone (finally!) This will make things much much easier, because I can actually test if I have regressed something.

[ikalchev]

https://github.com/ikalchev/HAP-python/pull/161

[ignalex]

Is there a way to update paired accesssory if changing steam config? I have only ssh to home next days, if pairing again I would need to be in the same wifi...

[ikalchev]

You can find the state file of pyhap and change the config_version field to +=1

[ignalex]

no luck on my side. with modified ffmpeg command (to working one) and (on 2nd attempt) reducing resolution - stream starting but iphone with 'loading' status and disconnects in 30 sec. ** camera_stable branch

[ikalchev]

@ignalex The screenshot ffmpeg command has the parameter -payload_type 99 repeated two times and the message Executing ... is shown - both are part of the current version on the dev branch.

Check again that you are on the camera_stable - I am able to start the stream consistently.

[ignalex]

gotcha! got it working from camera_stable with modified ffmpeg on pi3. streaming to outside - iPhone is in 4G, streaming server IP is my apple home-pod (selected automatically). well done @ikalchev ! any idea where was a glitch the last time? I see you merged it to dev - should switch to there prob.

[ikalchev]

Still no idea 😅 I haven’t merged to dev - I merged dev to camera_stable to cherry pick some changes tot he camera I made. I have a few more things to finish and I am merging to dev.

[aptonline]

Ready, willing and able to test again 👍🏼. Think this will be ready for the next release of Home Assistant (0.81.0)?

[ikalchev]

It depends on when is that :). In the meantime anyone preparing to use that can play with ffmpeg and make it work (ie start a stream locally) on their device.

[aptonline]

Well HA releases are on a 2 week cycle and 0.80.x is out now so... 😉

[nashlib]

I’ve been waiting in this as well. Final frontier for HomeKit

On 18 Oct 2018, at 4:15 pm, aptonline notifications@github.com wrote:

Well HA releases are on a 2 week cycle and 0.80.x is out now so... 😉

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[cdce8p]

@aptonline It woun't be in 0.81, because the deadline is tomorrow already. Since I'm quite busy at the moment, I believe 0.83 (so about one month) would be the earliest. However, if you want to take the charge on this one, let me know. I would be happy to help.

[aptonline]

@cdce8p I think you are in a better position than me :). I look forward to testing.

[ikalchev]

Folks, how would you like to give me your opinion on the client hooks for the camera:

Right now you can pass a start_stream_cmd which can be a string containing the stream configuration. For example (this is the default cmd):

'ffmpeg -re -f avfoundation -i 0:0 -threads 0 '
    '-vcodec libx264 -an -pix_fmt yuv420p -r {fps} -f rawvideo -tune zerolatency '
    '-vf scale={width}:{height} -b:v {v_max_bitrate}k -bufsize {v_max_bitrate}k '
    '-payload_type 99 -ssrc {v_ssrc} -f rtp '
    '-srtp_out_suite AES_CM_128_HMAC_SHA1_80 -srtp_out_params {v_srtp_key} '
    'srtp://{address}:{v_port}?rtcpport={v_port}&'
    'localrtcpport={v_port}&pkt_size=1378'

The placeholders will be formatted before the command is executed. You can see the start_stream method of Camera for a list of all the properties that you can {insert} in your command.

Alternatively, if a simple cmd string does not fit the bill, you can subclass Camera and implement the start_stream, stop_stream and reconfigure_stream methods - the same stream configuration as above will be passed as a dict.

You can try all of the above on the camera_stable branch right now.

[ikalchev]

Merged to dev again. Will push to master tomorrow probably.

[cdce8p]

@ikalchev Great work :1st_place_medal: I haven't been able to test it yet, but I definitely will in the future. Would you mind taking a quick look at #162 before you release the next version. Would be great to get this in as well, if you think that it should be merged.

Anyway, looking forward to the next release. I have some open issues in Home Assistant that you probably have fixed already.

[ikalchev]

Merged to master. GitHub release 2.3.0. Pushed to pip - version 2.3.0

As always, open issues for anything that doesn't behave.

Thanks everyone and sorry for the long delay!