search

gazoodle / gecko-home-assistant

Home Assistant integration for spas equipped with Gecko Alliance in.touch2 modules
MIT License
67 stars 22 forks source link

Leaks a UDP socket once per minute during disconnect (Cannot get file, protocol retry count exceeded) #80

Open rct opened 1 year ago

rct commented 1 year ago

Version

0.1.9

Configuration

  1. Originally Config flow through GUI
  2. Integration disabled but still getting this error.

Describe the bug

My spa is shut down for the winter.

My home assistant ran out of available ports and could no longer resolve names (no ports available for DNS lookups, pings would fail). Home assistant was up for a few weeks without a restart. There were 28,000+ UDP sockets. There are only ~28,200 port numbers available in a default system configuration (debian buster) - https://access.redhat.com/solutions/2039103. So it will only take about 19.5 days to exhaust the available ports.

After some troubleshooting and process of elimination it seems that a UDP socket is being leaked once per minute. Timing seems to coincide with the logging of the error Cannot get file, protocol retry count exceeded

To make matters worse, disabling the integration through the Hass GUI and restarting doesn't stop the problem. The error still occurs and the socket keeps leaking.

I tried to remove the integration through HACS to make this stop, but it says it is still configured.

image

However home assistant's /config/integrations shows it as disabled:

image

Next step is I'll delete the integration. and see if I can

Debug log

Using netstat to see leaked UDP sockets:

# netstat -aun | grep -P '0.0.0.0:\d\d\d\d\d'
udp        0      0 0.0.0.0:33680           0.0.0.0:*
udp        0      0 0.0.0.0:45492           0.0.0.0:*
udp        0      0 0.0.0.0:43451           0.0.0.0:*
udp        0      0 0.0.0.0:34748           0.0.0.0:*
udp        0      0 0.0.0.0:53701           0.0.0.0:*
udp        0      0 0.0.0.0:59853           0.0.0.0:*
udp        0      0 0.0.0.0:54249           0.0.0.0:*
udp        0      0 0.0.0.0:37867           0.0.0.0:*
udp        0      0 0.0.0.0:45548           0.0.0.0:*
udp        0      0 0.0.0.0:56303           0.0.0.0:*
udp        0      0 0.0.0.0:48638           0.0.0.0:*
udp        0      0 0.0.0.0:43014           0.0.0.0:*
udp        0      0 0.0.0.0:40458           0.0.0.0:*
udp        0      0 0.0.0.0:55313           0.0.0.0:*
udp        0      0 0.0.0.0:32789           0.0.0.0:*
udp        0      0 0.0.0.0:33309           0.0.0.0:*
udp        0      0 0.0.0.0:36441           0.0.0.0:*
udp        0      0 0.0.0.0:45663           0.0.0.0:*
udp        0      0 0.0.0.0:59510           0.0.0.0:*
udp        0      0 0.0.0.0:42628           0.0.0.0:*
udp        0      0 0.0.0.0:53913           0.0.0.0:*
udp        0      0 0.0.0.0:53411           0.0.0.0:*
udp        0      0 0.0.0.0:44733           0.0.0.0:*
udp        0      0 0.0.0.0:53450           0.0.0.0:*
udp        0      0 0.0.0.0:46795           0.0.0.0:*
udp        0      0 0.0.0.0:44255           0.0.0.0:*
udp        0      0 0.0.0.0:39653           0.0.0.0:*
udp        0      0 0.0.0.0:10001           0.0.0.0:*
udp        0      0 0.0.0.0:50977           0.0.0.0:*
udp        0      0 0.0.0.0:39713           0.0.0.0:*
udp        0      0 0.0.0.0:39752           0.0.0.0:*
udp        0      0 0.0.0.0:35157           0.0.0.0:*
udp        0      0 0.0.0.0:50014           0.0.0.0:*
udp        0      0 0.0.0.0:42335           0.0.0.0:*
udp        0      0 0.0.0.0:58213           0.0.0.0:*
udp        0      0 0.0.0.0:49009           0.0.0.0:*
udp        0      0 0.0.0.0:50546           0.0.0.0:*
udp        0      0 0.0.0.0:52610           0.0.0.0:*
udp        0      0 0.0.0.0:45965           0.0.0.0:*
udp        0      0 0.0.0.0:45455           0.0.0.0:*
# egrep gecko home-assistant.log
2023-03-02 22:03:42.135 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integra
tion gecko which has not been tested by Home Assistant. This component might cause stability pr
oblems, be sure to disable it if you experience issues with Home Assistant
2023-03-02 22:05:02.734 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:05:16.667 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry
 count exceeded
2023-03-02 22:06:02.796 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:06:20.815 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry
 count exceeded
2023-03-02 22:07:02.858 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:07:24.964 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry
 count exceeded
2023-03-02 22:08:02.916 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:08:29.112 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry
 count exceeded
2023-03-02 22:09:02.999 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:09:33.267 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry
 count exceeded
2023-03-02 22:10:03.078 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:10:37.682 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry
 count exceeded
2023-03-02 22:11:03.139 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:11:41.833 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:12:03.219 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:12:45.989 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:13:03.291 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations
to complete setup: gecko
2023-03-02 22:13:50.155 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:15:00.581 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:16:04.761 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:17:08.926 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:18:13.078 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:19:17.229 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:20:21.386 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:21:25.543 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:22:29.702 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:23:33.855 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:24:38.016 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:25:42.175 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:26:46.327 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:27:50.500 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:28:54.660 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:29:59.069 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:31:03.221 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:32:07.390 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:33:11.560 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:34:15.715 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:35:19.864 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded
2023-03-02 22:36:24.018 ERROR (MainThread) [geckolib.async_spa] Cannot get file, protocol retry count exceeded

One more piece of evidence: I use Munin for some lightweight host monitoring. Here's a 1 year view of the count of the open file descriptors in the kernel's file table.

image

The number of entries was pretty stable throughout the year until I shut the spa down in late December. Where it drops back down is when I've restarted Home Assistant for updates or config changes. When things failed I hadn't restarted Home Assistant since I upgraded to 2023.2.3.

rct commented 1 year ago

Update: Deleting the Gecko integration through the Home Assistant integration UI did solve the problem by preventing the gecko from trying to connect as it should. The only mention of gecko in the log is the custom integration warning. That's pretty much what I would expect.

Still I'm curious why with the Gecko integration disabled via the integration GUI why it was still trying to connect to the spa. Is that expected or is that a bug?