Frigate clips do not play on remote devices in Home Assistant

dermotduffy commented 2 years ago

Describe the problem you are having

I started exploring implementation of https://github.com/dermotduffy/frigate-hass-card/issues/439 , and realized that I cannot get HA to play any Frigate clips on remote media players.

Steps to reproduce:

Open the HA media player (sidebar -> 'Media')
Choose 'Frigate'
Choose 'Clips'
In the bottom right use the device selector to choose any device other than 'Browser'
Click play on any clip.

Expected behavior:

HA should play the media on the remote device.

Observed behavior:

The chromecast logo will appear on the target device, but the media will not play. The HA logs contain:

2022-05-01 16:23:44 ERROR (Thread-14) [homeassistant.components.cast.media_player] Failed to cast media https://[HA]/api/hls/0c7cf72cdcf07935f13b933daa682bc501338a1340553c29ff35ec389849766/master_
playlist.m3u8?authSig=[AUTH] from internal_url ([HA]). Please make sure the UR
L is: Reachable from the cast device and either a publicly resolvable hostname or an IP address

Initially I thought this was either an error on my end, or a problem with the integration. However:

Other media (e.g. random HA cameras) work just fine casting to the same devices from HA via the same steps above. It is Frigate event media only that will not play.
In chatting with @NickM-27 on discord, he observes the same -- but found some clips for some cameras that do work, which make it less likely this is an integration issue vs a Frigate or video issue.

Related/similar old bug: https://github.com/blakeblackshear/frigate/issues/2400

Version

2.2.2/0.10.1-83481af

Frigate config file

mqtt:
  host: [MQTT]
ffmpeg:
  global_args:
    - -hide_banner
    - -loglevel
    - info
  hwaccel_args:
    - -hwaccel
    - vaapi
    - -hwaccel_device
    - /dev/dri/renderD128
    - -hwaccel_output_format
    - yuv420p

snapshots:
  enabled: True
cameras:
  [many more cameras elided]
  sitting_room:
    ffmpeg:
      inputs:
        - path: [PATH]
          roles:
            - detect
            - rtmp
            - record
    detect:
      enabled: True
      width: 1920
      height: 1080
      fps: 4
    record:
      enabled: True
      retain:
        days: 0
      events:
        retain:
          default: 1
    motion:
      mask:
        - 1868,1058,1868,1018,1392,1018,1392,1058

detectors:
  coral:
    type: edgetpu
    device: usb
objects:
  track:
    - person
    - car
  filters:
    person:
      # Optional: minimum score for the object to initiate tracking (default: shown below)
      min_score: 0.6
      # Optional: minimum decimal percentage for tracked object's computed score to be considered a true positive (default: shown below)
      threshold: 0.75

Relevant log output

Log output shown above.

FFprobe output from your camera

(~) $ ffprobe rtsp://[WYZE-CAMERA]
ffprobe version 4.2.4-1ubuntu0.1 Copyright (c) 2007-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
Input #0, rtsp, from '[PATH]:
  Metadata:
    title           : Session streamed by the WYZE Media Server
    comment         : live
  Duration: N/A, start: 0.000625, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 15 fps, 15 tbr, 90k tbn, 30 tbc
    Stream #0:1: Audio: pcm_alaw, 8000 Hz, 1 channels, s16, 64 kb/s

Frigate stats

No response

Operating system

Other Linux

Install method

Docker Compose

Coral version

USB

Network connection

Wired

Camera make and model

Wyze cam v2

Any other information that may be helpful

No response

NickM-27 commented 2 years ago

I've seen some cameras that are more problematic than others and cameras that have some clips that work and some that don't. All are H.264 encoded.

The error I got in HA was:

Failed to cast media 
http://192.168.50.106:8123/api/frigate/frigate/vod/event/1651447110.007839-lp4unp/index.m3u8?authSig=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiI2NTdhZTk1M2YyZmM0YjM2ODFkNDczODNlMTEzOTIzNyIsInBhdGgiOiIvYXBpL2ZyaWdhdGUvZnJpZ2F0ZS92b2QvZXZlbnQvMTY1MTQ0NzExMC4wMDc4MzktbHA0dW5wL2luZGV4Lm0zdTgiLCJpYXQiOjE2NTE0NDcyMzYsImV4cCI6MTY1MTUzMzYzNn0.uHjU2tK05wccLOEIfWwl5KzsT4NgljnIi_DsnY9z3UM from internal_url (http://192.168.50.106:8123). 
Please make sure the URL is: Reachable from the cast device and either a publicly resolvable hostname or an IP address

felipecrs commented 2 years ago

Have you tried to use the Play Media action instead?

There it works for me.

PS: this is just out of curiosity; it does not mean that this isn't an issue.

dermotduffy commented 2 years ago

Hey @felipecrs , that is calling the play_media service, which (for me!) exhibits the exact same behavior as the in-built HA media-player (that's actually what I was trying first, for use in https://github.com/dermotduffy/frigate-hass-card/issues/439 ).

Does this & the HA media player work reliably for you across different cameras?

felipecrs commented 2 years ago

Yep, it works nicely for me!

https://github.com/blakeblackshear/frigate/issues/2896#issuecomment-1058802957

felipecrs commented 2 years ago

Some checks are:

Is your HA externally accessible with a valid HTTPS certificate?
Do you have the "Media Source" integration active? (it is active in default_config)

NickM-27 commented 2 years ago

Might be somewhat different for @dermotduffy but in my case where some clips work but some don't, may be something very nuanced

felipecrs commented 2 years ago

My clips work, but rarely they don't work too. I thought it was some kind of instability of HA itself.

NickM-27 commented 2 years ago

My clips work, but rarely they don't work too. I thought it was some kind of instability of HA itself.

Is it the same ones that always don't work, or does one work if you try later?

felipecrs commented 2 years ago

Is it the same ones that always don't work, or does one work if you try later?

I'll try to verify this later.

dermotduffy commented 2 years ago

Is your HA externally accessible with a valid HTTPS certificate? Do you have the "Media Source" integration active? (it is active in default_config)

Yes to both. Can cast any other media types (including live Frigate cameras) -- just not Frigate clips.

dermotduffy commented 2 years ago

Small update: Casting snapshots from Frigate also works fine/every-time, it's exclusively clips that have a problem (I've tried all my cameras -- cannot find one that works). If there were networking or certificate issues here I would expect snapshots to also fail.

blakeblackshear commented 2 years ago

In my experience, casting is very picky about both the video and audio codecs. I believe the audio must be AAC.

felipecrs commented 2 years ago

That's true. My audios are AAC (-c:a aac).

NickM-27 commented 2 years ago

All my cams are set to aac, and same camera has some work and some not. I'll pull the clips into VLC and see if there's any difference

felipecrs commented 2 years ago

Actually I found that I cannot reproduce the clips for one of my cameras (which is indeed different than the others). The behavior is exactly as @dermotduffy describes.

This clip works:

vlc_4wEKaFhp4y

But this one not:

vlc_230u6IFmBY

(That's a camera without a microphone)

Apart from the framerate (15FPS in the first, 30FPS in the second) and resolution, the only difference is that the second lacks audio.

I will try to add -c:a aac to it too, and see if it makes any difference (would Frigate create an empty audio layer, or would it crash? Let's see).

felipecrs commented 2 years ago

Yep, that solves the issue. Now the clips for the camera that were not playable before are playable in the Chromecast. I simply added:

output_args:
  record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
  rtmp: -c:v copy -c:a aac -f flv

To the <camera>.ffmpeg. PS: there wasn't anything before, and I don't have a global output_args config.

The clip now has a blank audio layer in AAC format.

dermotduffy commented 2 years ago

Thanks @felipecrs -- good find. That also worked for me and I can now cast a clip from a camera that previously did not work. That doesn't really explain why @NickM-27 would have some clips that did and did not work on the same camera, though.

In any case, I don't know the background behind disabling audio (-an vs c:a aac) being in the default record flags -- what drives that choice? One the one hand, casting a clip feels like something that we'd expect and want to just work out of the box, but on the other hand -- no-one seems to have complained about it much ...

felipecrs commented 2 years ago

I think the strongest reason behind -an is due to CPU usage.

NickM-27 commented 2 years ago

Here is some info, left DOES NOT work while the right DOES work:

The only difference that I can see is a slightly higher framerate on the one that does not work 🤷‍♂️

felipecrs commented 2 years ago

@NickM-27, perhaps in your case, the video file has some small corruption. One that VLC handles, but Chromecast does not.

It happens a lot with my wi-fi cameras... in that case, Chromecast isn't the only place it's not playable. They are usually not playable from the Frigate UI too.

NickM-27 commented 2 years ago

@NickM-27, perhaps in your case, the video file has some small corruption. One that VLC handles, but Chromecast does not.

It happens a lot with my wi-fi cameras... in that case, Chromecast isn't the only place it's not playable. They are usually not playable from the Frigate UI too.

Yeah, all of these are perfectly playable from Frigate UI.

felipecrs commented 2 years ago

Ok. So, I think everything we just said about AAC is true. But @NickM-27 may have something else even.

NickM-27 commented 2 years ago

Yeah, honestly not too worried about mine as it works mostly and could be some network unhappiness or something.

I'll add a PR to add a section in the docs with a note that audio is required for using casting with the integration for others that desire the functionality

dermotduffy commented 2 years ago

Think this is ~resolved.

iamnikv commented 2 years ago

Hello, this is my output args config for my camera:

      output_args:
        rtmp: -vf transpose=2,transpose=2 -c:v libx264 -c:a aac -preset ultrafast -tune zerolatency -b:v 600k -f flv
        record: -vf transpose=2,transpose=2 -c:v libx264 -c:a aac -preset ultrafast -tune zerolatency -b:v 600k -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1
        detect: -vf transpose=2,transpose=2 -f rawvideo -pix_fmt yuv420p

However my clips are still not playing on my Android device neither in the HA app nor in the browser.

felipecrs commented 2 years ago

Does it work with:

ffmpeg:
  output_args:
    record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac

https://docs.frigate.video/faqs/#audio-in-recordings

NickM-27 commented 2 years ago

@nvvasilev What error are you getting inside hass logs when trying this?

I am assuming this is with the USB camera and those other args are required for record?

iamnikv commented 2 years ago

@nvvasilev What error are you getting inside hass logs when trying this?

I am assuming this is with the USB camera and those other args are required for record?

That is correct. The args are specific to the USB cam. Logs:

Logger: homeassistant
Source: custom_components/frigate/views.py:294
Integration: Frigate ([documentation](https://github.com/blakeblackshear/frigate), [issues](https://github.com/blakeblackshear/frigate-hass-integration/issues))
First occurred: 5:11:32 PM (1 occurrences)
Last logged: 5:11:32 PM

Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/frigate/views.py", line 294, in _proxy_msgs
    await ws_out.send_bytes(msg.data)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_ws.py", line 315, in send_bytes
    await self._writer.send(data, binary=True, compress=compress)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 688, in send
    await self._send_frame(message, WSMsgType.BINARY, compress)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 659, in _send_frame
    await self.protocol._drain_helper()
  File "/usr/local/lib/python3.9/site-packages/aiohttp/base_protocol.py", line 87, in _drain_helper
    await asyncio.shield(waiter)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 528, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/local/lib/python3.9/asyncio/sslproto.py", line 199, in feed_ssldata
    chunk = self._sslobj.read(self.max_size)
  File "/usr/local/lib/python3.9/ssl.py", line 888, in read
    v = self._sslobj.read(len)
ssl.SSLError: [SSL: SSLV3_ALERT_BAD_RECORD_MAC] sslv3 alert bad record mac (_ssl.c:2633)

iamnikv commented 2 years ago

I am not sure if this error is related to the playback issue..

iamnikv commented 2 years ago

This error does not appear every time I try to play a clip, that is why I think it is not related.

iamnikv commented 2 years ago

Ok, got it working. I setup an rtsp-server and stream the USB cam live feed as an rtsp stream using this ffmpeg command: ffmpeg -f v4l2 -i /dev/video0 -vf "transpose=2,transpose=2" -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -pix_fmt yuv420p -c:v libx264 -c:a aac -preset ultrafast -tune zerolatency -b:v 600k -s 1280x720 -f rtsp -rtsp_transport tcp rtsp://localhost:8554/frontdoor

Then I use the rtsp stream in the Frigate configuration like this:

cameras:
  frontdoor:
    ffmpeg:
      inputs:
        - path: rtsp://192.168.1.139:8554/frontdoor
          roles:
            - detect
            - record

And now every new clip recording with this new config is playable on my Android devices. I wonder if in the output args for the detect role from the previous config had to be configured with the -c:v libx264 -c:a aac in order to fix the problem as it was the only role not configured to use audio..

      output_args:
        rtmp: -vf transpose=2,transpose=2 -c:v libx264 -c:a aac -preset ultrafast -tune zerolatency -b:v 600k -f flv
        record: -vf transpose=2,transpose=2 -c:v libx264 -c:a aac -preset ultrafast -tune zerolatency -b:v 600k -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1
        detect: -vf transpose=2,transpose=2 -f rawvideo -pix_fmt yuv420p

blakeblackshear / frigate