search

roleoroleo / yi-hack-Allwinner-v2

Custom firmware for Yi 1080p camera based on Allwinner platform
MIT License
752 stars 90 forks source link

WebRTC stream fails after a while #860

Closed robbinjanssen closed 3 weeks ago

robbinjanssen commented 2 months ago

First of all, thanks for all your hard work! Really appreciate the time and effort you are putting into this.

The problem

When my cam has been running for a while, at some point the stream will no longer load. Not in HA using the WebRTC component, and neither in VLC using the RTSP stream. After a reboot it works again for a limited time. Seems like a memory issue so I'm hoping we can solve this.

The details:

The settings:

Logs:

Please let me know what logs I can provide to you, the output of top at the moment the camera stream is no longer working:

  PID  PPID USER     STAT   VSZ %VSZ CPU %CPU COMMAND
 1382     1 root     S    58220 95.2   0 67.9 ./rmm
 2175     1 root     S<    5420  8.8   0  4.4 rRTSPServer -m h51ga -r high -a pcm -p 554 -u admin -w REDACTED
  602     2 root     SW       0  0.0   0  0.5 [RTW_XMIT_THREAD]
 9676  8568 root     R      996  1.6   0  0.3 top
 8548  1799 root     S      920  1.5   0  0.3 /tmp/sd/yi-hack/bin/dropbearmulti /tmp/sd/yi-hack/sbin/dropbear -R -B -p 0.0.0.0:22 -2 7
  595     2 root     SW       0  0.0   0  0.3 [ksdioirqd/mmc1]
  604     2 root     SW       0  0.0   0  0.3 [RTWHALXT]
  616     1 root     S     1260  2.0   0  0.2 ./dispatch
    7     2 root     RW       0  0.0   0  0.2 [rcu_preempt]
    3     2 root     SW       0  0.0   0  0.2 [ksoftirqd/0]
  603     2 root     SW       0  0.0   0  0.2 [RTW_CMD_THREAD]
12448     2 root     SW       0  0.0   0  0.2 [kworker/u2:2]
 1414     1 root     S     1488  2.4   0  0.0 /backup/tools/wpa_supplicant -c/tmp/wpa_supplicant.conf -g/var/run/wpa_supplicant-global -Dnl80211 -iwlan0 -B
 1750     1 root     S     1216  1.9   0  0.0 ./cloud
14152   616 root     S     1048  1.7   0  0.0 [sh]
 1472     1 root     S     1048  1.7   0  0.0 udhcpc -i wlan0 -b -s /backup/tools/default.script -x hostname:yi-camera
 2181     1 root     S      992  1.6   0  0.0 {wd_rtsp.sh} /bin/sh /tmp/sd/yi-hack/script/wd_rtsp.sh
 8568  8548 root     S      984  1.6   0  0.0 -ash
    1     0 root     S      980  1.6   0  0.0 /sbin/init
14128  2181 root     S      980  1.6   0  0.0 sleep 10
 2372     1 root     S      964  1.5   0  0.0 {crond} /tmp/sd/yi-hack/bin/busybox /tmp/sd/yi-hack/usr/sbin/crond -c /var/spool/cron/crontabs/
 2086     1 root     S      964  1.5   0  0.0 {ntpd} /tmp/sd/yi-hack/bin/busybox /tmp/sd/yi-hack/usr/sbin/ntpd -p pool.ntp.org
 1778     1 root     S      964  1.5   0  0.0 {httpd} /tmp/sd/yi-hack/bin/busybox /tmp/sd/yi-hack/usr/sbin/httpd -p 80 -h /tmp/sd/yi-hack/www/ -c /tmp/httpd.conf
 1799     1 root     S      892  1.4   0  0.0 /tmp/sd/yi-hack/bin/dropbearmulti /tmp/sd/yi-hack/sbin/dropbear -R -B -p 0.0.0.0:22
 1869     1 root     S      880  1.4   0  0.0 mqttv4
 2269     1 root     S      792  1.3   0  0.0 onvif_notify_server --conf_file /tmp/onvif_simple_server.conf
 1938     1 root     S      776  1.2   0  0.0 mqtt-config
 2350     1 root     S      688  1.1   0  0.0 wsd_simple_server --pid_file /var/run/wsd_simple_server.pid --if_name wlan0 --xaddr http://%s/onvif/device_service -m yi-camera -n Yi
 2264     1 root     S      676  1.1   0  0.0 ipc2file
  533     2 root     SW       0  0.0   0  0.0 [mmcqd/0]
 3967     2 root     SW       0  0.0   0  0.0 [kworker/0:0]
  528     2 root     SW<      0  0.0   0  0.0 [kworker/0:1H]
  454     2 root     SW       0  0.0   0  0.0 [irq/302-sunxi-m]
  298     2 root     SW       0  0.0   0  0.0 [kswapd0]
24732     2 root     SW       0  0.0   0  0.0 [kworker/u2:0]
16887     2 root     SW       0  0.0   0  0.0 [kworker/0:2]
  214     2 root     SW       0  0.0   0  0.0 [kcompactd0]
  549     2 root     SWN      0  0.0   0  0.0 [jffs2_gcd_mtd4]
13733     2 root     SW       0  0.0   0  0.0 [kworker/0:1]
  457     2 root     SW       0  0.0   0  0.0 [irq/303-sunxi-m]
  456     2 root     SW       0  0.0   0  0.0 [irq/166-sdc0 cd]
  422     2 root     SW<      0  0.0   0  0.0 [bioset]
  412     2 root     SW<      0  0.0   0  0.0 [bioset]
  407     2 root     SW<      0  0.0   0  0.0 [bioset]
  417     2 root     SW<      0  0.0   0  0.0 [bioset]
  530     2 root     SW<      0  0.0   0  0.0 [bioset]
  387     2 root     SW<      0  0.0   0  0.0 [bioset]
  382     2 root     SW<      0  0.0   0  0.0 [bioset]
  392     2 root     SW<      0  0.0   0  0.0 [bioset]
  288     2 root     SW<      0  0.0   0  0.0 [spi0]
  267     2 root     SW<      0  0.0   0  0.0 [cfg80211]
  259     2 root     SW       0  0.0   0  0.0 [sys_user]
  273     2 root     SW<      0  0.0   0  0.0 [watchdogd]
  397     2 root     SW<      0  0.0   0  0.0 [bioset]
  215     2 root     SW<      0  0.0   0  0.0 [crypto]
  212     2 root     SW<      0  0.0   0  0.0 [writeback]
roleoroleo commented 2 months ago

It could be a memory issue. Try to disable kernel tuning.

robbinjanssen commented 2 months ago

I will try and let you know, thanks!

robbinjanssen commented 2 months ago

Hi @roleoroleo, the stream seems more stable now. Also disabled the snapshot feature which seems to help. I've encountered another problem, when looking at the stream it goes black after a while, then it takes about 50secs (this varies) and the screen comes back on, you can see the cam adjusting the light levels.

Not sure if this is a soft- or hardware issue, do you have any idea/suggestions? Is there some more info I can provide?

The black screen starts at 20:33:32 but i've trimmed this video to only show the relevant part:

https://github.com/roleoroleo/yi-hack-Allwinner-v2/assets/120077/f858e6fe-b710-45da-af98-33e0855540f6

roleoroleo commented 2 months ago

When the stream disappears, check if the rtsp daemon (rRTSPServer) is running. Maybe the daemon dies and the watchdog restarts it.

robbinjanssen commented 2 months ago

The RTSPServer is still running while this happens, is there anything else I can check?

roleoroleo commented 2 months ago

You should try to run the daemon with extended log. kill the process and run it again:

killall wd_rtsp.sh
killall rRTSPServer
/tmp/sd/yi-hack/bin/rRTSPServer -m h51ga -r high -a pcm -p 554 -u admin -w REDACTED -d 15

And send me the output.

robbinjanssen commented 2 months ago

Thanks, will do tonight, doesn't really seem to happen during the day

robbinjanssen commented 2 months ago

@roleoroleo it's kind of hard to tell when the black screen starts exactly because the output is a lot. If there's anything I can do to improve that please let me know.

Attached is the log of the stream: https://gist.github.com/robbinjanssen/307994efba0926d394507f24eb28d983

I noticed the black screen at around 1713990366305.

roleoroleo commented 2 months ago

The log is perfect: no errors, no lost frames in the grabber and no lost frames in the delivery. Around 1713990366305 nothing strange. Please try a network capture with wireshark/tcpdump. You can download tcpdump for Allwinner-v2 here: https://github.com/roleoroleo/yi-hack-utils/tree/main/Allwinner-v2 tcpdump -i wlan0 -w /tmp/capture.pcap

robbinjanssen commented 2 months ago

@roleoroleo here is the output, it happens a couple times in this capture. I tried analyzing it and saw some batched groups of messages containing "Interleaved channel", not sure what it means though.

./tcpdump -i wlan0 -w /tmp/sd/yi-hack/tmp/capture.pcap
tcpdump: listening on wlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C7143 packets captured
7198 packets received by filter
0 packets dropped by kernel

capture.pcap.zip

robbinjanssen commented 2 months ago

When the frames turn black, I actually hear the camera clicking and I think its turning the IR LED on/off.. is this faulty hardware or broken (yi) firmware?

I think I read in another issue that you cannot manually turn on/off the IR LED right? Only toggle it entirely?

roleoroleo commented 2 months ago

Sorry but I have to ask a new capture. You should start the rtsp client after starting the capture. Otherwise wireshark is not able to detect the rtp/rtcp stream.

Meanwhile, I found some strange thing: the traffic is fragmented, a lot of reassembled packets with strange sizes. Is there something strange in your network?

When the frames turn black, I actually hear the camera clicking and I think its turning the IR LED on/off.. is this faulty hardware or broken (yi) firmware?

Check if it's just a click (the shutter) or if the cam is rebooting. About hardware failure, it could be a power issue. Are you using the factory adapter and cable? Has the cam enough current?

robbinjanssen commented 1 month ago

I do hear a click sometimes, but more frequently the cam is just trying to "adjust" the light levels. The cam is not rebooting as I do see the time/date increasing on the stream.

I'm starting to think this is a faulty hardware issue, but I'll try to make a new capture this week. I've also ordered a Eufy E220 cam as an alternative to experiment with. If that one works fine and I don't need this one anymore, I'll gladly sent it to you (if you are in EU) for you to experiment with.

robbinjanssen commented 3 weeks ago

@roleoroleo (un)fortunately it was a hardware issue and the cam has been replaced. This one works :-) Thanks for your help