search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.82k stars 30.08k forks source link

ASUSWRT integration: multiple SSH connections crash my modem #73978

Closed maicol07 closed 11 months ago

maicol07 commented 2 years ago

The problem

Hi, I have an ASUS modem and I installed the ASUSWRT integration. However, the integration creates a lot of SSH connections and leads to router/modem crash. Further details: https://www.snbforums.com/threads/error-no-space-left-on-device.79449/#post-770999

A possible fix would be creating a connection at startup and use the same for retrieving data without creating a new one.

What version of Home Assistant Core has the issue?

Core 2022.6.6

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Core

Integration causing the issue

AsusWRT

Link to integration documentation on our website

https://www.home-assistant.io/integrations/asuswrt/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

probot-home-assistant[bot] commented 2 years ago

asuswrt documentation asuswrt source (message by IssueLinks)

probot-home-assistant[bot] commented 2 years ago

Hey there @kennedyshead, @ollo69, mind taking a look at this issue as it has been labeled with an integration (asuswrt) you are listed as a code owner for? Thanks! (message by CodeOwnersMention)

sfortis commented 2 years ago

same problem here, ASUS AC86U / merlin firmware [386.7]

router log file indicates a new login every few minutes

Jul 2 12:32:44 dropbear[14231]: Password auth succeeded for 'sfortis' from 192.168.1.41:58402 Jul 2 12:35:44 dropbear[14785]: Password auth succeeded for 'sfortis' from 192.168.1.41:58404 Jul 2 12:37:45 dropbear[15163]: Password auth succeeded for 'sfortis' from 192.168.1.41:58406 Jul 2 12:42:45 dropbear[16110]: Password auth succeeded for 'sfortis' from 192.168.1.41:58408 Jul 2 12:43:15 dropbear[16204]: Password auth succeeded for 'sfortis' from 192.168.1.41:58410 Jul 2 12:57:45 dropbear[18926]: Password auth succeeded for 'sfortis' from 192.168.1.41:58412 Jul 2 12:58:45 dropbear[19121]: Password auth succeeded for 'sfortis' from 192.168.1.41:58414

image

shmick commented 2 years ago

Adding my +1 to this as I'm experiencing the same thing.

Home Assistant Core 2022.6.7 ASUS RT-AC68U Merlin Current Version : 386.7

When I ssh into the router manually and logout, I see this in the logs:

Jul  6 21:40:52 RT-AC68U-9950 dropbear[18758]: Exit (admin) from <192.168.118.22:57490>: Exited normally
Jul  6 21:43:30 RT-AC68U-9950 dropbear[19516]: Exit (admin) from <192.168.118.22:57905>: Disconnect received

I don't see these Exit entries from the Home Assistant ssh sessions, which leads me to think the sessions is getting stuck open and then new sessions are created until there's hundreds of dropbear sessions and the router runs out of memory.

I only see these from the Home Assistant host

Jul  6 21:49:48 RT-AC68U-9950 dropbear[20497]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56494
Jul  6 21:49:48 RT-AC68U-9950 dropbear[20498]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56496
Jul  6 21:51:48 RT-AC68U-9950 dropbear[20801]: Child connection from 192.168.118.10:56498
Jul  6 21:51:48 RT-AC68U-9950 dropbear[20801]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56498
Jul  6 21:52:18 RT-AC68U-9950 dropbear[20882]: Child connection from 192.168.118.10:56500
Jul  6 21:52:18 RT-AC68U-9950 dropbear[20882]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56500
Jul  6 21:53:18 RT-AC68U-9950 dropbear[21035]: Child connection from 192.168.118.10:56502
Jul  6 21:53:18 RT-AC68U-9950 dropbear[21035]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56502
Jul  6 21:54:18 RT-AC68U-9950 dropbear[21189]: Child connection from 192.168.118.10:56504
Jul  6 21:54:18 RT-AC68U-9950 dropbear[21189]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56504
Jul  6 21:54:48 RT-AC68U-9950 dropbear[21257]: Child connection from 192.168.118.10:56506
Jul  6 21:54:48 RT-AC68U-9950 dropbear[21257]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56506
Jul  6 21:54:48 RT-AC68U-9950 dropbear[21283]: Child connection from 192.168.118.10:56508  
Jul  6 21:54:48 RT-AC68U-9950 dropbear[21283]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56508
Jul  6 21:56:48 RT-AC68U-9950 dropbear[21563]: Child connection from 192.168.118.10:56510
Jul  6 21:56:48 RT-AC68U-9950 dropbear[21565]: Child connection from 192.168.118.10:56512
Jul  6 21:56:48 RT-AC68U-9950 dropbear[21563]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56510
Jul  6 21:56:48 RT-AC68U-9950 dropbear[21565]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56512
Jul  6 21:56:48 RT-AC68U-9950 dropbear[21591]: Child connection from 192.168.118.10:56514
Jul  6 21:56:49 RT-AC68U-9950 dropbear[21591]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56514
Jul  6 21:58:48 RT-AC68U-9950 dropbear[21879]: Child connection from 192.168.118.10:56516
Jul  6 21:58:48 RT-AC68U-9950 dropbear[21879]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56516
Jul  6 21:59:18 RT-AC68U-9950 dropbear[21964]: Child connection from 192.168.118.10:56518
Jul  6 21:59:18 RT-AC68U-9950 dropbear[21965]: Child connection from 192.168.118.10:56520
Jul  6 21:59:18 RT-AC68U-9950 dropbear[21964]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56518
Jul  6 21:59:18 RT-AC68U-9950 dropbear[21965]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56520
Jul  6 21:59:18 RT-AC68U-9950 dropbear[21983]: Child connection from 192.168.118.10:56522
Jul  6 21:59:18 RT-AC68U-9950 dropbear[21983]: Pubkey auth succeeded for 'admin' with ssh-ed25519 key from 192.168.118.10:56522

When restarting HA, we can see all the sessions get closed.

Jul  6 22:57:21 RT-AC68U-9950 dropbear[30237]: Exit (admin) from <192.168.118.10:56618>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29844]: Exit (admin) from <192.168.118.10:56612>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[30691]: Exit (admin) from <192.168.118.10:56626>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29613]: Exit (admin) from <192.168.118.10:56608>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31142]: Exit (admin) from <192.168.118.10:56630>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[30305]: Exit (admin) from <192.168.118.10:56620>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31767]: Exit (admin) from <192.168.118.10:56644>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31839]: Exit (admin) from <192.168.118.10:56646>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31997]: Exit (admin) from <192.168.118.10:56648>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[28876]: Exit (admin) from <192.168.118.10:56590>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31453]: Exit (admin) from <192.168.118.10:56640>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[28935]: Exit (admin) from <192.168.118.10:56592>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31373]: Exit (admin) from <192.168.118.10:56636>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[28999]: Exit (admin) from <192.168.118.10:56594>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31301]: Exit (admin) from <192.168.118.10:56634>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29073]: Exit (admin) from <192.168.118.10:56596>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31614]: Exit (admin) from <192.168.118.10:56642>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29380]: Exit (admin) from <192.168.118.10:56602>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[31220]: Exit (admin) from <192.168.118.10:56632>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[30996]: Exit (admin) from <192.168.118.10:56628>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29462]: Exit (admin) from <192.168.118.10:56604>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[30541]: Exit (admin) from <192.168.118.10:56624>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29464]: Exit (admin) from <192.168.118.10:56606>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[30388]: Exit (admin) from <192.168.118.10:56622>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29227]: Exit (admin) from <192.168.118.10:56598>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29928]: Exit (admin) from <192.168.118.10:56614>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29377]: Exit (admin) from <192.168.118.10:56600>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[29772]: Exit (admin) from <192.168.118.10:56610>: Exited normally
Jul  6 22:57:21 RT-AC68U-9950 dropbear[30147]: Exit (admin) from <192.168.118.10:56616>: Exited normally
sfortis commented 2 years ago

As a workaround, i've made a simple script in my /jffs/scripts to clean-up the sessions and free some ram.

netstat -p | grep homeassistant | grep ESTABLISHED | awk '{print $7}' | awk -F'/' '{print "kill -9", $1}' | sh

I've put this on crontab every midnight.

shmick commented 2 years ago

@sfortis nice job on the kill script. That was on my to-do list. Seems to be working well so far. I've got it running every 20 minutes on my AC68U since it's only got 256MB of memory and that was getting exhausted every 6 or so hours.

Jul  7 10:21:19 RT-AC68U-9950 dropbear[4276]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56898
Jul  7 10:21:19 RT-AC68U-9950 dropbear[4278]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56902
Jul  7 10:21:19 RT-AC68U-9950 dropbear[4277]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56900
Jul  7 10:21:49 RT-AC68U-9950 dropbear[4345]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56906
Jul  7 10:22:19 RT-AC68U-9950 dropbear[4440]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56908
Jul  7 10:22:49 RT-AC68U-9950 dropbear[4528]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56910
Jul  7 10:23:19 RT-AC68U-9950 dropbear[4624]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56912
Jul  7 10:24:19 RT-AC68U-9950 dropbear[4799]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56914
Jul  7 10:25:19 RT-AC68U-9950 dropbear[4954]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56916
Jul  7 10:26:19 RT-AC68U-9950 dropbear[5154]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56918
Jul  7 10:28:19 RT-AC68U-9950 dropbear[5511]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56920
Jul  7 10:28:49 RT-AC68U-9950 dropbear[5592]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56922
Jul  7 10:29:19 RT-AC68U-9950 dropbear[5676]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56924
Jul  7 10:31:49 RT-AC68U-9950 dropbear[6056]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56926
Jul  7 10:31:49 RT-AC68U-9950 dropbear[6058]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56928
Jul  7 10:32:19 RT-AC68U-9950 dropbear[6145]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56930
Jul  7 10:32:19 RT-AC68U-9950 dropbear[6146]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56932
Jul  7 10:33:19 RT-AC68U-9950 dropbear[6300]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56936
Jul  7 10:33:19 RT-AC68U-9950 dropbear[6301]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56938
Jul  7 10:35:19 RT-AC68U-9950 dropbear[6618]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56940
Jul  7 10:36:19 RT-AC68U-9950 dropbear[6771]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56942
Jul  7 10:37:19 RT-AC68U-9950 dropbear[6925]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56946
Jul  7 10:38:19 RT-AC68U-9950 dropbear[7083]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56948
Jul  7 10:39:49 RT-AC68U-9950 dropbear[7308]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56950
Jul  7 10:39:49 RT-AC68U-9950 dropbear[7310]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56952
Jul  7 10:40:19 RT-AC68U-9950 dropbear[7397]: Pubkey auth succeeded for 'admin' with ssh-ed25519 from 192.168.118.10:56954
Jul  7 10:41:01 RT-AC68U-9950 dropbear[5511]: Exit (admin) from <192.168.118.10:56920>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4624]: Exit (admin) from <192.168.118.10:56912>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4954]: Exit (admin) from <192.168.118.10:56916>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4440]: Exit (admin) from <192.168.118.10:56908>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4276]: Exit (admin) from <192.168.118.10:56898>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4345]: Exit (admin) from <192.168.118.10:56906>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4278]: Exit (admin) from <192.168.118.10:56902>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[5676]: Exit (admin) from <192.168.118.10:56924>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6058]: Exit (admin) from <192.168.118.10:56928>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6146]: Exit (admin) from <192.168.118.10:56932>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[7397]: Exit (admin) from <192.168.118.10:56954>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6771]: Exit (admin) from <192.168.118.10:56942>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6925]: Exit (admin) from <192.168.118.10:56946>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6145]: Exit (admin) from <192.168.118.10:56930>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[7308]: Exit (admin) from <192.168.118.10:56950>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6300]: Exit (admin) from <192.168.118.10:56936>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4799]: Exit (admin) from <192.168.118.10:56914>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4277]: Exit (admin) from <192.168.118.10:56900>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[5154]: Exit (admin) from <192.168.118.10:56918>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[7083]: Exit (admin) from <192.168.118.10:56948>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6301]: Exit (admin) from <192.168.118.10:56938>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[7310]: Exit (admin) from <192.168.118.10:56952>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6618]: Exit (admin) from <192.168.118.10:56940>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[6056]: Exit (admin) from <192.168.118.10:56926>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[4528]: Exit (admin) from <192.168.118.10:56910>: Terminated by signal
Jul  7 10:41:01 RT-AC68U-9950 dropbear[5592]: Exit (admin) from <192.168.118.10:56922>: Terminated by signal

Hopefully @kennedyshead or @ollo69 can track down this bug and squash it!

sfortis commented 2 years ago

@sfortis nice job on the kill script. That was on my to-do list. Seems to be working well so far. I've got it running every 20 minutes on my AC68U since it's only got 256MB of memory and that was getting exhausted every 6 or so hours.

Hopefully @kennedyshead or @ollo69 can track down this bug and squash it!

Im glad it worked for you! Don't forget to put the cru command into the /jffs/scripts/services-start which is executed when the router reboots.

ollo69 commented 2 years ago

I have created some times ago PR #71899 that is implementing a new protocol for Asus router, with the objective to fix some of the issue caused by ssh / telnet protocol. The PR was still not take in charge, but I also duplicated the code in this custom integration. It would be good if you could install and test custom integration in place of the native one, waiting for core team take in charge the PR.

shmick commented 2 years ago

@ollo69 I'm trying to configure your custom integration, but I'm getting this error This integration does not support configuration via the UI. If you followed this link from the Home Assistant website make sure you run the latest version of Home Assistant.

A second restart of Home Assistant was needed and now it's working. Will begin testing it now. Thanks.

github-actions[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

shmick commented 1 year ago

Waiting for #71899 to be merged to fix this

iliketoprogram14 commented 1 year ago

Anything we can do to get this reviewed more quickly, such as anyone we can ping on a discord or something like that? This is kind of a serious bug since it can crash the router, or at least make it go OOM, so it should be much higher priority, right?

ollo69 commented 1 year ago

At this moment I created a second PR #84152 hoping that a smaller PR will be keep in consideration. The old ssh/telnet library have big problem with new router / new firmware version, but I have not idea on how to accelerate this process. Ping on PR have normally opposite effect, maybe someone could try with discord...

sfortis commented 1 year ago

Anything we can do to get this reviewed more quickly, such as anyone we can ping on a discord or something like that? This is kind of a serious bug since it can crash the router, or at least make it go OOM, so it should be much higher priority, right?

Yes, user Asus Router integration which is http(s) based.

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

sfatula commented 1 year ago

Still happening for me on 2023.4.4. I had over 400 dropbear processes and it crashed the router.

sfortis commented 1 year ago

Still happening for me on 2023.4.4. I had over 400 dropbear processes and it crashed the router.

Try ha-asusrouter integration, works flawlessly!

https://github.com/Vaskivskyi/ha-asusrouter

issue-triage-workflows[bot] commented 1 year ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

iliketoprogram14 commented 1 year ago

@ollo69 this is still a work in progress, right?

ollo69 commented 1 year ago

Yes, now PR is under review. Maybe new version will be available in next major release.

issue-triage-workflows[bot] commented 11 months ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.