home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.85k stars 30.1k forks source link

Seven Segments Display on Docker hangs Hass #8206

Closed MartyTremblay closed 6 years ago

MartyTremblay commented 7 years ago

Home Assistant release (hass --version): 0.47.1 (Docker)

Python release (python3 --version): 3.6.1

Component/platform: Seven Segments Display

Description of problem: When I enable image_processing.seven_segments, my Docker container becomes irresponsive after a few minutes

Problem-relevant configuration.yaml entries and steps to reproduce:

image_processing:
  - platform: seven_segments
    digits: 2
    height: 160
    width: 128
    threshold: 87
    extra_arguments: "-d 2 -b black"
    source:
      - entity_id: camera.basement_dehumidifier

Traceback (if applicable): Unable to get a traceback since the container hangs before anything is printed from ssocr to stdout.

Additional info: Looked at code to double-check that we weren't using Popen.wait(), which could cause deadlocks. Turns out that the script uses Popen.communicate() as hoped. Not sure what's going on...

michaelarnauts commented 7 years ago

You can still get the logs of a running or even crashed container, even if you can't connect to it.

If it's still running, try docker ps to get the container id of your hass container (first column). You can get the logs trough docker logs -f *id*. It can be useful to increase your loglevels to debug before doing this. You might see some useful information in the few moments before the lockup.

If the container is already stopped (but I don't think that this lockup will kill the container), you can use the same steps as above, but use docker ps -a to also see your stopped containers.

MartyTremblay commented 7 years ago

Hey thanks for the tip @michaelarnauts . here are the last few relevant lines after a crash.

2017-06-27 17:45:49 DEBUG (MainThread) [homeassistant.components.websocket_api] WS 140357295413344: Sending {'id': 2, 'type': 'event', 'event': {'event_type': 'state_changed', 'data': {'entity_id': 'sensor.ups_battery_voltage', 'old_state': <state sensor.ups_battery_voltage=1190.0 ; unit_of_measurement=V, friendly_name=UPS Battery Voltage, icon=mdi:flash @ 2017-06-27T17:44:47.221392-04:00>, 'new_state': <state sensor.ups_battery_voltage=1209.0 ; unit_of_measurement=V, friendly_name=UPS Battery Voltage, icon=mdi:flash @ 2017-06-27T17:45:49.263966-04:00>}, 'origin': 'LOCAL', 'time_fired': datetime.datetime(2017, 6, 27, 21, 45, 49, 264020, tzinfo=<UTC>)}}
2017-06-27 17:45:49 DEBUG (MainThread) [homeassistant.components.websocket_api] WS 140357295413344: Sending {'id': 2, 'type': 'event', 'event': {'event_type': 'state_changed', 'data': {'entity_id': 'sensor.fixed_ups_battery_voltage', 'old_state': <state sensor.fixed_ups_battery_voltage=119.0; unit_of_measurement=V, friendly_name=Mains Voltage, icon=mdi:power-plug @ 2017-06-27T17:44:47.230290-04:00>, 'new_state': <state sensor.fixed_ups_battery_voltage=120.9; unit_of_measurement=V, friendly_name=Mains Voltage, icon=mdi:power-plug @ 2017-06-27T17:45:49.283571-04:00>}, 'origin': 'LOCAL', 'time_fired': datetime.datetime(2017, 6, 27, 21, 45, 49, 283620, tzinfo=<UTC>)}}
2017-06-27 17:45:49 INFO (MainThread) [homeassistant.components.http] Serving /api/camera_proxy/camera.basement_dehumidifier to 192.168.101.151 (auth: False)
2017-06-27 17:45:49 DEBUG (MainThread) [haffmpeg.core] Start FFmpeg with ['ffmpeg', '-i', 'http://user:pass@192.168.101.57/img/video.asf', '-an', '-frames:v', '1', '-c:v', 'mjpeg', '-pred', '1', '-r', '1', '-f', 'image2pipe', '-']

The configured camera:

camera:
  - platform: ffmpeg
    name: basement dehumidifier
    input: http://user:pass@192.168.101.57/img/video.asf
    extra_arguments: '-pred 1 -r 1'

Should also state that running the camera without image_processing works and is stable.

pvizeli commented 7 years ago

Can you see if the ffmpeg process is still running?

MartyTremblay commented 7 years ago

According to this, it is not.

Before it hangs:

docker top b89527a8be58
PID                 USER                COMMAND
991                 root                python -m homeassistant --config /config

After a few refreshes:

PID                 USER                COMMAND
991                 root                python -m homeassistant --config /config
2926                root                ffmpeg -i http://user:pass@192.168.101.57/img/video.asf -an -frames:v 1 -c:v mjpeg -pred 1 -r 1 -f image2pipe -

After it hangs:

docker top b89527a8be58
PID                 USER                COMMAND
991                 root                python -m homeassistant --config /config
7581                root                python -m homeassistant --config /config

Strange that the container is now running two instances of homeassistant?

MartyTremblay commented 7 years ago

Or perhaps more accurate:

docker exec -it b89527a8be58 ps -aux       
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  4.2  2.4 1416208 98928 ?       Ssl+ 21:37   0:48 python -m homeassistant --config /config
root       329  0.0  2.1 1416208 86804 ?       S+   21:43   0:00 python -m homeassistant --config /config
root       339  0.0  0.0  19192  2240 ?        Rs+  21:57   0:00 ps -aux
MartyTremblay commented 7 years ago

Interesting, kill -9 7581 (second process) on the host revives the homeassistant web application.

However, after a few minutes, the web application hangs and there are two python -m homeassistant --config /config running again.

michaelarnauts commented 7 years ago

My guess is that the 2nd process is a fork of the real home assistant process due to the subprocess.Popen. That process will actually be running the ssocr command, but the command still tells us it's running homeassistant, since it's forked.

Is there a way to increase verbosity of the ssocr binary?

pvizeli commented 7 years ago

@michaelarnauts I think the problem is the ffmpeg process. But I don't see any mistake ha-ffmpeg code :worried:

MartyTremblay commented 7 years ago

Indeed @pvizeli Anecdotally, I'm experiencing the same freeze-up with 2 homeassistant python processes even with image_processing turned-off, except that it takes longer to happen. I'll remove the ffmpeg camera to see if it stabilizes my container.

MartyTremblay commented 7 years ago

Hmm, removing the ffmpeg camera has stabilized the container and seems to have made the application generally more responsive.

I'll be using ffmpeg from within ZoneMinder for ssocr processing and see how that goes.

MartyTremblay commented 7 years ago

Using ZoneMinder seems to have done the trick. ssocr has been running for ~51hrs without a glitch. Certainly seems to be an issue with the ffmpeg component

pvizeli commented 7 years ago

Yeah, the move to async break ffmpeg. I will try to change some things but my task list is high

balloobbot commented 7 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment :+1: