home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
73.74k stars 30.85k forks source link

continues login crashes AsusWRT #103230

Closed danielskowronski closed 6 months ago

danielskowronski commented 1 year ago

The problem

This is reopen of https://github.com/home-assistant/core/issues/75056

Out-of-the-box official integration AsusWRT, slowly but steadily crashes AsusWRT Merlin running router by opening and failing to close SSH connections, ultimately filling up the entire RAM.

On my router (which has 1GB of RAM), each such connections leaves 3704K of garbage, and last time it crashed, there were over 4500 instances of dropbear.

It appears that if there's an issue during connection, then all channels will be closed, but not the connection itself. It's also the reason why Dropbear is unable to treat those connections as inactive and auto-close them.


In this state, this official integration is effectively a router-killer. If there's no chance for quick solutions, I think there should be at least a warning added.

What version of Home Assistant Core has the issue?

core-2023.11.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

AsusWRT

Link to integration documentation on our website

https://www.home-assistant.io/integrations/asuswrt/

Diagnostics information

diag.json

Example YAML snippet

No response

Anything in the logs that might be useful for us?

AsusWRT merlin version seems to be irrelevant; however, I can confirm it's affecting 3004.388.4 on RT-AX88U Pro. Other reports on the official FW forum exist (e.g. this one)

ha.log contains logs from HA with debug logs enabled for AsusWRT

netstat.log is dump of established connections from HomeAssistant to AsusWRT, seconds after logs were obtained

router.log contains syslog from AsusWRT

From there, it's easy to track ssh connection states. For example:

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @kennedyshead, @ollo69, mind taking a look at this issue as it has been labeled with an integration (asuswrt) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `asuswrt` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign asuswrt` Removes the current integration label and assignees on the issue, add the integration domain after the command.

(message by CodeOwnersMention)


asuswrt documentation asuswrt source (message by IssueLinks)

ollo69 commented 1 year ago

Working on PR #95720 that could fix this problems implementing HTTP protocol, unfortunately this is taking a lot of time.

danielskowronski commented 1 year ago

So the SSH connector is abandoned? At first glance, it seems like it's some missing connection close in asyncssh - couldn't find anything on Dropbear side.

ollo69 commented 1 year ago

I do not see any maintenance from times in ssh lbrary, but you can try to open a issue in aioasuswrt and see if someone will take it in charge.

danielskowronski commented 1 year ago

It looks like issue is partially there on all levels.

First, AsusWRT Merlin has a misleading option "Idle timeout" under SSH server config. It actually sets nvram shell_timeout, which is only used to construct forced env variable TMOUT. This one is read by interactive shells, however in the situation we have here, there's no interactive session - only SSH connection with no channels. It can be verified by checking dropbear parameters - it's only -p ADDR:22 -a with -I <idle_timeout> omitted, so it defaults to 0 or never. However, it looks like this behaviour is inherited from upstream vanilla Asus code (#define DEFAULT_IDLE_TIMEOUT 0 in Dropbear source code and nothing else setting -I).

Second, there's connection leak in aioasuswrt at least here - https://github.com/kennedyshead/aioasuswrt/blob/master/aioasuswrt/connection.py#L45 and it seems like that async_connect should be wrapped in asyncio.Lock(). Just like https://github.com/kennedyshead/aioasuswrt/pull/68

Finally, the layout of this module is not helping - explicit disconnect is available, but implemented only for Telnet mode - https://github.com/home-assistant/core/blob/dev/homeassistant/components/asuswrt/bridge.py#L164

I've been testing my theory with manual edits to aioasuswrt and it looks like it's the root problem we should tackle. I'll try to make the necessary PR there, and it shouldn't require any change in this module :)

issue-triage-workflows[bot] commented 9 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

danielskowronski commented 9 months ago

/no further activity occurs as all affected user have their routers crashed

ollo69 commented 9 months ago

@danielskowronski,

did you try to reconfigure integration selecting https protocol or this solution doesn't work for you?

issue-triage-workflows[bot] commented 6 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.