AirVisual device integration leaking file handles on failed SMB connection

ratsputin commented 3 years ago

The problem

The AirVisual device integration apparently creates a temporary file in /tmp when attempting to make an SMB connection to an AirVisual device. If the SMB connection fails due to an incorrect password, the temporary file and its file handle is not cleaned up.

As this happens once per minute, HA will eventually hang due to file handle exhaustion.

What is version of Home Assistant Core has the issue?

core-2021.4.5

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

AirVisual

Link to integration documentation on our website

https://www.home-assistant.io/integrations/airvisual/

Example YAML snippet

N/A

Anything in the logs that might be useful for us?

2021-04-22 15:17:09 INFO (MainThread) [SMB.SMBConnection] Authentication with remote machine "AIRVISUAL" for user "airvisual" will be using NTLM v2 authentication (with extended security)
2021-04-22 15:17:09 INFO (SyncWorker_3) [SMB.SMBConnection] Now switching over to SMB2 protocol communication
2021-04-22 15:17:09 INFO (SyncWorker_3) [SMB.SMBConnection] SMB2 dialect negotiation successful
2021-04-22 15:17:09 INFO (SyncWorker_3) [SMB.SMBConnection] Performing NTLMv2 authentication (on SMB2) with server challenge "b'08c4265f63560434'"
2021-04-22 15:17:09 INFO (SyncWorker_3) [SMB.SMBConnection] Performing NTLMv2 authentication (on SMB2) with server challenge "b'08c4265f63560434'"
2021-04-22 15:17:09 INFO (SyncWorker_3) [SMB.SMBConnection] Server supports SMB signing
2021-04-22 15:17:09 INFO (SyncWorker_3) [SMB.SMBConnection] SMB signing deactivated. SMB messages will NOT be signed.
2021-04-22 15:17:10 INFO (SyncWorker_3) [SMB.SMBConnection] Authentication (on SMB2) failed. Please check username and password.

Additional information

Each time I would see the above log messages, I would see another file created in /tmp. After about 36 hours, this is a snippet of the tail end of an lsof.

hass    6590 homeassistant 1011u      REG                8,2         0 10622808 /tmp/tmpnw6p0dpy
hass    6590 homeassistant 1012u      REG                8,2         0 10622809 /tmp/tmp_bxp9g7x
hass    6590 homeassistant 1013u      REG                8,2         0 10622810 /tmp/tmpu1w7a0ao
hass    6590 homeassistant 1014u      REG                8,2         0 10622811 /tmp/tmpubf3y9tx
hass    6590 homeassistant 1015u      REG                8,2         0 10622812 /tmp/tmp8klnn0s7
hass    6590 homeassistant 1016u      REG                8,2         0 10622813 /tmp/tmpuj65nz5a
hass    6590 homeassistant 1017u      REG                8,2         0 10622814 /tmp/tmpc55_v5b0
hass    6590 homeassistant 1018u      REG                8,2         0 10622815 /tmp/tmpn5vr6spl
hass    6590 homeassistant 1019u      REG                8,2         0 10622816 /tmp/tmp36s91a0c
hass    6590 homeassistant 1020u      REG                8,2         0 10622817 /tmp/tmplx36to0x
hass    6590 homeassistant 1021u      REG                8,2         0 10622818 /tmp/tmpw5851yhc
hass    6590 homeassistant 1022u      REG                8,2         0 10622820 /tmp/tmpv2g0vrfd
hass    6590 homeassistant 1023u      REG                8,2         0 10622821 /tmp/tmp2c3b9vfx

Note there are 1023 file handles open. Each of these handles had a corresponding file still existent in /tmp.

Fixing the password addressed the issue.

probot-home-assistant[bot] commented 3 years ago

airvisual documentation airvisual source _{^{(message by IssueLinks)}}

probot-home-assistant[bot] commented 3 years ago

Hey there @bachya, mind taking a look at this issue as its been labeled with an integration (airvisual) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

bachya commented 3 years ago

I've spun up a fresh VM/HASS instance to test and can't reproduce this.

First, I replicated the same pyairvisual code that the HASS integration uses:

https://github.com/home-assistant/core/blob/32dfaccf1fab5ce6d7ab73a3dee7c9aa4c93351d/homeassistant/components/airvisual/__init__.py#L235-L241

...fed it a bad password, and ran it numerous times. At no point did I didn't see a /tmp file created.

I can see that the UI incorrectly allows a bad password and need to fix that, but I'm puzzled as to why I don't see these spurious /tmp files...

ratsputin commented 3 years ago

Hmm... Let me provide a bit more information and see if it might provide a clue.

I did just update from core-2021.1.? and had never seen the issue before upgrading to this version
I'm pretty sure I added the integration on core-2021.1.?
Even after upgrading, I still had airvisual: along with the password entry for the API in my configuration.yaml and only removed it after seeing the message that it was no longer supported. Removing that and restarting HA did not fix the tmp file issue. Note, I did remove that entry and restart before I fixed the SMB password.
I'm running in a python venv with version 3.8.5 on Ubuntu 20.04
This isn't the only HA component that is creating files with that name format in /tmp. I see dozens of files like that as well as .py files with the same name format there. Case-in-point, there are 84 files there right now after 6 hours of uptime.

bachya commented 3 years ago

This isn't the only HA component that is creating files with that name format in /tmp. I see dozens of files like that as well as .py files with the same name format there. Case-in-point, there are 84 files there right now after 6 hours of uptime.

How are you able to tell which integration (including AirVisual) is creating these files?

ratsputin commented 3 years ago

I watched syslog at the same time I was watching /tmp. Each time the authentication failure message came across, a new file was created in /tmp at the same time. This continued to happen during each of my troubleshooting steps (disabling integrations, enabling debug, etc.), including restarting HA and rebooting. This behavior was consistent over several reboots for about 1.5 weeks.

Once I deleted the integration and entered the correct password, the issue stopped. Nothing else was changed.

bachya commented 3 years ago

To make sure I understand: are you talking about your Home Assisstant machine's syslog? Failed authentication attempts to an AirVisual Node were showing up there?

ratsputin commented 3 years ago

That is correct. After fixing the password, files were no longer created in /tmp when the success message would come up.

For your reference--from the archived syslog:

Unsuccessful (just prior to fixing the password), which would immediately create a file:

Apr 22 15:16:21 ha hass[69924]: 2021-04-22 15:16:21 INFO (MainThread) [SMB.SMBConnection] Authentication with remote machine "AIRVISUAL" for user "airvisual" will be using NTLM v2 authentication (with extended security)
Apr 22 15:16:22 ha hass[69924]: 2021-04-22 15:16:22 INFO (SyncWorker_4) [SMB.SMBConnection] Now switching over to SMB2 protocol communication
Apr 22 15:16:22 ha hass[69924]: 2021-04-22 15:16:22 INFO (SyncWorker_4) [SMB.SMBConnection] SMB2 dialect negotiation successful
Apr 22 15:16:22 ha hass[69924]: 2021-04-22 15:16:22 INFO (SyncWorker_4) [SMB.SMBConnection] Performing NTLMv2 authentication (on SMB2) with server challenge "b'53936aa79c165459'"
Apr 22 15:16:22 ha hass[69924]: 2021-04-22 15:16:22 INFO (SyncWorker_4) [SMB.SMBConnection] Performing NTLMv2 authentication (on SMB2) with server challenge "b'53936aa79c165459'"
Apr 22 15:16:22 ha hass[69924]: 2021-04-22 15:16:22 INFO (SyncWorker_4) [SMB.SMBConnection] Server supports SMB signing
Apr 22 15:16:22 ha hass[69924]: 2021-04-22 15:16:22 INFO (SyncWorker_4) [SMB.SMBConnection] SMB signing deactivated. SMB messages will NOT be signed.
Apr 22 15:16:22 ha hass[69924]: 2021-04-22 15:16:22 INFO (SyncWorker_4) [SMB.SMBConnection] Authentication (on SMB2) failed. Please check username and password.

Successful, no file created:

Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (MainThread) [SMB.SMBConnection] Authentication with remote machine "AIRVISUAL" for user "airvisual" will be using NTLM v2 authentication (with extended security)
Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (SyncWorker_19) [SMB.SMBConnection] Now switching over to SMB2 protocol communication
Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (SyncWorker_19) [SMB.SMBConnection] SMB2 dialect negotiation successful
Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (SyncWorker_19) [SMB.SMBConnection] Performing NTLMv2 authentication (on SMB2) with server challenge "b'155820a7a8e4ac6b'"
Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (SyncWorker_19) [SMB.SMBConnection] Performing NTLMv2 authentication (on SMB2) with server challenge "b'155820a7a8e4ac6b'"
Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (SyncWorker_19) [SMB.SMBConnection] Server supports SMB signing
Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (SyncWorker_19) [SMB.SMBConnection] SMB signing deactivated. SMB messages will NOT be signed.
Apr 22 15:24:35 ha hass[69924]: 2021-04-22 15:24:35 INFO (SyncWorker_19) [SMB.SMBConnection] Authentication (on SMB2) successful!

bachya commented 3 years ago

@ratsputin Thanks! Still can't directly reproduce, but I have some updates that may help. Will get those submitted soon.

home-assistant / core