Roomba no longer detected

MinimalMule commented 1 year ago

The problem

My iRobot s9+ is no longer adding to home assistant. It had been working well previously but stopped (presumably with a firmware update), and I can no longer add it to home assistant.

It SOMETIMES auto detects, but when I try to add it goes into manual mode, and then if I enter the IP address I receive a fail to connect error.

I have factory reset the Roomba twice and deleted the home assistant integration, but I don’t get any meaningful error messages. I have also tried changing the IP address and moving the Roomba to a different subnet, all giving the same result. Thanks!

What version of Home Assistant Core has the issue?

2022.11.4

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

roomba

Link to integration documentation on our website

https://github.com/home-assistant/core/tree/dev/homeassistant/components/roomba

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 year ago

Hey there @pschmitt, @cyr-ius, @shenxn, mind taking a look at this issue as it has been labeled with an integration (roomba) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `roomba` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Change the title of the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign roomba` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

roomba documentation roomba source _{^{(message by IssueLinks)}}

andreabenini commented 1 year ago

Subscribing here too for updates. I've the same problem after a pip install --upgrade, I'm now running homeassistant 2022.11.4, (linux AMD64, real machine not containerized, python 3.10.8). I can see these possible problems:

error: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled (_ssl.c:997) and error: [Errno 111] Connection refused (From logs)
binary_sensor.VACUUMNAME_bin_full no longer exists (probably because of the SSL Cert Error)

Vacuum has been already restarted and password retrieval (with Dorita) already done, no changes on the client side, major difference here is homeassistant update (on my side). Feel free to ask to create a separate issue for me if this, according to your opinion, is not stricly related

MinimalMule commented 1 year ago

Any maintainers still working on this integration? Happy to provide any additional information that might help!

redespace commented 1 year ago

Same here. It used to work before and now I am getting Connection refused in the logs.

andreabenini commented 1 year ago

I have an update for all folks here watching this thread and an advice for package mantainers. I'll start with the advice because I've found where the real problem is, my idea is just a workaround and final solution should be engineered properly. The problem and the solution is exactly in the logs reported above: [SSL: UNSAFE_LEGACY_RENEGOTIATION_DISABLED] unsafe legacy renegotiation disabled. I've faced it after upgrading python and homeassistant (in virtualenv). In my case I'm using Python 3.10.9, please note that from 3.10 and later python explicitly forbids unsafe (or self assigned) certs and if you need to get a rid or explicitly ignore them or this kind of error always occours. Previous python versions are less strict on these kind of things but you now totally need to deal with it. If you've an "older" (3/4y or more) Roomba like my 986 you'll surely have a self assigned cert inside it and when you connect to it with python debug flags you can see all these kind of errors.

To package mantainers: when you connect to the roomba with explicit request() functions avoid/ignore self assigned certs from the roomba or you won't be able to connect to the device when python is set in enforcing mode or when you've a 3.10 (or later) versions

As a workaround I've solved by adding SSL variables but this is a dirty hack and proper solution should be held in place. In my case I've simply added two lines in main hass script:

# these are inside 'hass' script as a workaround
import os
os.environ['OPENSSL_CONF'] = os.path.dirname(__file__) + '/openssl.cnf'

Or you can obviously use something like export OPENSSL_CONF=...path...openssl.cnf to link to your ssl configuration, in my case I'm dropping certs with something like:

openssl_conf = openssl_init

[openssl_init]
ssl_conf = ssl_sect

[ssl_sect]
system_default = system_default_sect

[system_default_sect]
Options = UnsafeLegacyRenegotiation

But it's better to patch the module, this is just a temp workaround. Feel free to comment or add information to this topic

foobarth commented 1 year ago

Thanks @andreabenini, that worked for me!

andreabenini commented 1 year ago

Please bear in mind this is just a workaround until someone will fix python https request to Roombas with self signed certs. It might not be a problem for newer Ix (i5, i7, ...) machines but if you combine new python versions with older 8xx,9xx roombas and their self signed certs you'll surely spot this problem. Using OPENSSL_CONF variable with a custom conf is simply awkward and proper http request (with ignore cert) should be held in place in the vacuum plugin. I never wrote patches for that module but feel free to ask if you need a MergeRequest for it.

foobarth commented 1 year ago

I'm aware of the nastiness of the workaround, but after all, it's just that and working fine for my i6 for now until a proper patch is available. Thanks again and happy new year!

MinimalMule commented 1 year ago

@andreabenini - I'm all for a dodgy workaround, just struggling to understand what you're referring to by the "main hass script"?

andreabenini commented 1 year ago

@MinimalMule You've several ways if you want to achieve it.

use your home assistant shell environment and add something like export OPENSSL_CONF=...path...openssl.cnf to let python and virtualenv know you are using a well documented OPENSSL variable. You must point to openssl.cnf file I've provided before
inject those two lines (also, see before) in your hass python script. It's usually located in $PYTHON_VENV/bin/hass if you've installed HA with pip (but you can even search it in your installation). This is nasty because you're adding that var inside the whole os.environ python space. Unfortunately I don't have a lot of time to debug the module and instead of patching the vacuum module I've just brutally added something to the whole Env. Wife Acceptance Factor (WAF) usually predicts a 5 nines uptime (99.999) and it's easy to exceed your SLA on these tickets :)

I'll probably take a look at it after all these holidays but maintainers might probably preceed me on it.

MinimalMule commented 1 year ago

@andreabenini Thanks for responding so quickly! I think the issue here is that I'm running HASSOS, and if I'm right that means that I can't access that script?

Matttman commented 1 year ago

I'm having the same issue and running Home Assistant OS so I'm unable to modify the scripts unless I'm missing something. Is there any other workaround for this?

andreabenini commented 1 year ago

You can search a script named hass which barely starts HA or you can edit the shell environment where it runs.
My suggestion was related to SSL environment and its behavior by setting OPENSSL_CONF (export OPENSSL_CONF='....') to get a rid of autoassigned/expired certificates, but it's just a mere trick for getting a rid of older roomba certs. I cannot see maintainers replies around so I'll probably fix the module on my own later on but I have no time and an official Merge Request might take a lot if it depends on me, I'm a bare user too. It might be helpful to have some help from author here.

ferdyvi commented 1 year ago

Hello, I have the same problem with a 960 robot, and there is another issue open here: https://github.com/home-assistant/core/issues/69594 As the root cause seems to be related to the SSL, I think the right place to write is here. @andreabenini thanks for sharing your workaround, but it will downgrade the SSL protocol for all homeassistant connections, right?

@pschmitt @Cyr-ius @shenxn could you kindly have a look on the problem discovered by @andreabenini ? Thank you so much!

andreabenini commented 1 year ago

Even worse, I'm altering the python SSL environment to basically ignore SSL certificates validation, everything will be trusted and it doesn't matter what you present in front of it. They (iRobot) are basically ignoring possible expired certs and always trust HTTPS/SSL connections, that's the wrong approach but even the reason why everything still works if you're using their app (which I dropped ~3/4y ago, I only use HA). You've probably updated your python environment in the last few months but starting from version 3.10 (I guess) when you issue an HTTPS request() without further parameters (like ssl IGNORE params) you won't trust anymore self generated or expired ssl certificates, every roomba (like my 9xx, ~4y old) have these. My two days hack during Christmas was just focused on mitigating the WAF (Wife Acceptance Factor) without altering or touching the codebase. I work as SRE for a company and I use python for a living but I'm not (yet) part of the HA community and I've never touched an HA module. If I was the author of it I'd probably alter only the single http request() to the Roomba to fully trust the connection because they (iRobot) simply don't care to update certs there. I won't crush the SSL environment for just this. If nothing moves I'll probably submit some patches on mine to the repository but I still need to find some time to do it. BTW this is still a really solid HA module, one of my favorites and I was planning to extend it a little bit in order to integrate live data maps, this sounds achievable even if iRobot doesn't do this for my model.

austinmroczek commented 1 year ago

Had the same problem with Total Connect. See how we fixed it in the underlying package at https://github.com/craigjmidwinter/total-connect-client/pull/196

Dilski commented 1 year ago

This is an issue that would need fixing in the roombapy library. I've created an issue there about implementing a fix: https://github.com/pschmitt/roombapy/issues/175

Dilski commented 1 year ago

SSL changes are in the latest release of roombapy, and now working on getting that merged into home assistant. Hopefully we can be automatically vacuuming together soon!

MinimalMule commented 1 year ago

Had high hopes for the fix, but it still doesn't seem to have resolved the problem with my Roomba s9+!

Dilski commented 1 year ago

@MinimalMule what is the error you're getting? This fix fixed the legacy renegotiation error on my 600 series

MinimalMule commented 1 year ago

@MinimalMule what is the error you're getting? This fix fixed the legacy renegotiation error on my 600 series

I know these log records look ridiculous, but all I'm getting is:

2023-03-10 23:02:56.482 DEBUG (SyncWorker_9) [roombapy.discovery] Socket server started, port 5678 2023-03-10 23:02:56.483 DEBUG (SyncWorker_9) [roombapy.discovery] Message sent 2023-03-10 23:03:01.488 INFO (SyncWorker_9) [roombapy.discovery] Socket timeout 2023-03-10 23:03:07.492 DEBUG (SyncWorker_7) [roombapy.discovery] Socket server started, port 5678 2023-03-10 23:03:07.492 DEBUG (SyncWorker_7) [roombapy.discovery] Message sent 2023-03-10 23:03:12.498 INFO (SyncWorker_7) [roombapy.discovery] Socket timeout

After some more digging I've managed to get it to spit out:

2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Socket server started, port 5678
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 0
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 1
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 2
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 3
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 4
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 5
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 6
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 7
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 8
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 9
2023-03-11 00:25:05.015 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 10
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 11
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 12
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 13
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 14
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 15
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 16
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 17
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 18
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 19
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 20
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 21
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 22
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 23
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Broadcast message sent: 24
2023-03-11 00:25:05.016 DEBUG (SyncWorker_8) [roombapy.discovery] Received response: b'irobotmcs', address: ('XXX.XXX.XXX.XXX', 5678)
2023-03-11 00:25:05.017 DEBUG (SyncWorker_8) [roombapy.discovery] Received response: b'irobotmcs', address: ('XXX.XXX.XXX.XXX', 5678)
2023-03-11 00:25:05.031 DEBUG (SyncWorker_8) [roombapy.discovery] Received response: b'irobotmcs', address: ('XXX.XXX.XXX.XXX', 5678)
2023-03-11 00:25:05.039 DEBUG (SyncWorker_8) [roombapy.discovery] Received response: b'irobotmcs', address: ('XXX.XXX.XXX.XXX', 5678)
2023-03-11 00:25:05.040 DEBUG (SyncWorker_8) [roombapy.discovery] Received response: b'irobotmcs', address: ('XXX.XXX.XXX.XXX', 5678)

Still not happily connecting, I'm not sure why but the address it is quoting back at me there (XXX'd out) is the address of my homeassistant instance. I have two subnets, homeassistant has addresses in both of them. @Dilski I'm guessing my problem is something in my network and the issue was just coincidental!

Dilski commented 1 year ago

To debug, you can try and clone this project and run the scripts to discover roombas (from a machine in the same subnet as roomba) to rule out networking vs roomba device issues

https://github.com/pschmitt/roombapy

ferdyvi commented 1 year ago

I confirm the updated library, with python 3.11, does not fix the problem Is it possible to re-open the issue?

bobloadmire commented 1 year ago

would this be related to why im trying to setup a roomba s9 brand new today and getting "error cannot connect"? I successfully retrieved my blid and password, but for some reason the integration will not connect.

MinimalMule commented 1 year ago

@bobloadmire - certainly sounds like we're in the same boat!

Dilski commented 1 year ago

Would agree to re-open. @MinimalMule @bobloadmire @ferdyvi Without any more detailed log information, it's hard to nail down what's causing your problem. If I were to provide you with a python script to run at the command line to generate some debug out, is that something you'd be able to run?

MinimalMule commented 1 year ago

@Dilski absolutely - will give running roombapy a go shortly, I'm running on HASSOS so a few of the debugging items are a bit trickier in that platform, but anything I can do in python world I'm happy to run from another machine?

bobloadmire commented 1 year ago

Would agree to re-open. @MinimalMule @bobloadmire @ferdyvi Without any more detailed log information, it's hard to nail down what's causing your problem. If I were to provide you with a python script to run at the command line to generate some debug out, is that something you'd be able to run?

I can run whatever you want as long as it works on my rpi4

ferdyvi commented 1 year ago

Would agree to re-open. @MinimalMule @bobloadmire @ferdyvi Without any more detailed log information, it's hard to nail down what's causing your problem. If I were to provide you with a python script to run at the command line to generate some debug out, is that something you'd be able to run?

We can try!

MinimalMule commented 1 year ago

@Dilski I've tried to run roombapy on another machine - doesn't look happy:

Roomba have to be on Home Base powered on.
Press and hold HOME button until you hear series of tones.
Release button, Wi-Fi LED should be flashing
Press Enter to continue...
Traceback (most recent call last):
  File "/usr/local/bin/roomba-password", line 8, in <module>
    sys.exit(password())
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 28, in password
    _validate_roomba_info(roomba_info)
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 69, in _validate_roomba_info
    raise Exception("cannot find roomba")
Exception: cannot find roomba

$ roomba-discovery
$ roomba-discovery 192.168.XXX.XXX
None

Roomba is responding to a ping at that address, and roomba app control works fine

Also just tried...

Roomba980-Python (no success there either)
changing Roomba to another wifi network and IP address
playing with a few network settings

Nothing made a dent. Just in case there's commonality here - @bobloadmire and @ferdyvi - are either of you running ubiquiti networks? (just wondering if it could be some obscure ubiquiti thing - wouldn't be the first time)

Dilski commented 1 year ago

@MinimalMule I've got a branch here that enables debug logging for those scripts: https://github.com/Dilski/roombapy/tree/debug-

Alternatively copy-paste this into what you've got checked out already and give it another go https://github.com/Dilski/roombapy/commit/0b22e9ab49a7cf817981e476bb02421108e25aa0

MinimalMule commented 1 year ago

@Dilski

roomba-password 192.168.XXX.XXX
Roomba have to be on Home Base powered on.
Press and hold HOME button until you hear series of tones.
Release button, Wi-Fi LED should be flashing
Press Enter to continue...
DEBUG:roombapy.discovery:Socket server started, port 5678
DEBUG:roombapy.discovery:Message sent
INFO:roombapy.discovery:Socket timeout
Traceback (most recent call last):
  File "/usr/local/bin/roomba-password", line 8, in <module>
    sys.exit(password())
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 31, in password
    _validate_roomba_info(roomba_info)
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 72, in _validate_roomba_info
    raise Exception("cannot find roomba")
Exception: cannot find roomba

/usr/local/lib/python3.9/dist-packages/roombapy$ roomba-discovery 192.168.XXX.XXX
DEBUG:roombapy.discovery:Socket server started, port 5678
DEBUG:roombapy.discovery:Message sent
INFO:roombapy.discovery:Socket timeout
None

/usr/local/lib/python3.9/dist-packages/roombapy$ roomba-connect 192.168.XXX.XXX
Traceback (most recent call last):
  File "/usr/local/bin/roomba-connect", line 8, in <module>
    sys.exit(connect())
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 44, in connect
    _validate_password(roomba_password)
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 67, in _validate_password
    raise Exception("password cannot be null")
Exception: password cannot be null

/usr/local/lib/python3.9/dist-packages/roombapy$ roomba-connect 192.168.XXX.XXX ""
DEBUG:roombapy.discovery:Socket server started, port 5678
DEBUG:roombapy.discovery:Message sent
INFO:roombapy.discovery:Socket timeout
Traceback (most recent call last):
  File "/usr/local/bin/roomba-connect", line 8, in <module>
    sys.exit(connect())
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 48, in connect
    _validate_roomba_info(roomba_info)
  File "/usr/local/lib/python3.9/dist-packages/roombapy/entry_points.py", line 72, in _validate_roomba_info
    raise Exception("cannot find roomba")
Exception: cannot find roomba

And just to check I'm not totally crazy:

PING 192.168.XXX.XXX (192.168.XXX.XXX) 56(84) bytes of data.
64 bytes from 192.168.XXX.XXX: icmp_seq=1 ttl=64 time=118 ms
64 bytes from 192.168.XXX.XXX: icmp_seq=2 ttl=64 time=36.2 ms
64 bytes from 192.168.XXX.XXX: icmp_seq=3 ttl=64 time=70.3 ms
64 bytes from 192.168.XXX.XXX: icmp_seq=4 ttl=64 time=76.3 ms
64 bytes from 192.168.XXX.XXX: icmp_seq=5 ttl=64 time=99.1 ms

MinimalMule commented 1 year ago

@Dilski OK... So I'm left even more confused now!

I just installed roombapy on a windows machine with python, and lo and behold, it connected first try!

So I'm now left trying to chase down the differences between these machines, and the biggest one I can think of is that both my HASSOS install and the linux machine I was testing with are installed on a single Hyper-V hypervisor as Linux VMs, and maybe something is going wrong in the translation process there...

Alternatively, it could be something that the linux firewall or some other part of the linux system is blocking. Will revert back if I work this junk out...

MinimalMule commented 1 year ago

Slightly more testing:

works on Windows 11 machine running Python 3.8
works on Windows 11 machine running Python 3.9.2
works on Windows 11 machine running Python 3.11
doesn't work on Debian machine running Python 3.9.2
doesn't work on homeassistant

The Windows 11 machine, Debian machine, and homeassistant are all running in the same VLAN. I don't have a dedicated linux box but I'll see what I can scrounge together to test whether it's the Hyper-V platform that is giving me a problem. Any ideas of things I can check on the linux box I have to see why it isn't getting through would be great!

Update:

So, I ran Wireshark on the Linux machine, I can see UDP packets heading out from the machine, going to the correct address, and I can't see anything coming back... Going to try and delve deeper into the network and see if I can find the packets anywhere.

Likely final update!

So, after lots of digging - it seems like the UDP packets just can't escape the Hyper-V host. I have no idea why this problem is new, considering I was using my Roomba on this rig quite happily before. Also have no idea why it seems to only be impacting the Roomba extension. Seems like the quick and dirty thing is to just find another piece of hardware, put HA on that, and see if it wakes up happily in a non-virtualised state.

@Dilski - thank you for all of your help, and sorry for the wild goose chase!

bobloadmire commented 1 year ago

So that hyper v issue shouldn't affect us ha users on rpi though right?

Oh and no I'm not using ubiquity, have crappy consumer Linksys mesh system on wired backhaul

MinimalMule commented 1 year ago

@bobloadmire I can't see how it would... I would suggest that as per my insane rants above you do some similar testing - starting with the step of using a different machine with Python and attempting to connect. Certainly there were a bunch of different places where this could go wrong, and maybe you will have more luck that I did in the debugging stages, for me it was just total loss of communication (data not reaching destination).

You might find that trying to monitor with Wireshark is a good bet as that is where I was able to confirm that no data was coming back. Unfortunately this one seems extremely sneaky!

bobloadmire commented 1 year ago

Ok I'll see what I can do, I should be able to run it on a WSL on a windows rig right? I saw this issue marked as closed so that's why I was asking. Doesn't seem solved.

Dilski commented 1 year ago

I wonder if you have some firewall/port blocking going on. From my understanding roombapy starts up a port and instructs roomba to connect to it. If roomba can't connect that may be a problem that's hard to debug (and yeah you may need to wireshark)

bobloadmire commented 1 year ago

I can put my rpi on a DMZ in the router settings and see if that clears it up

MinimalMule commented 1 year ago

@bobloadmire you can run it direct on windows - just download Python and use pip to install roombapy and away you go

MinimalMule commented 1 year ago

well - I'm happy to call this one closed, but for future people looking - I have confirmed that Homeassistant running as a VM on Hyper-V does not happily communicate over UDP, and this prevents the Roomba from working. I ended up installing HASSOS on bare metal and the Roomba plugin worked out of the box without a problem.

Thanks @Dilsk and @andreabenini for persevering with this one, it's certainly the oddest issue that I've had...

bobloadmire commented 1 year ago

Well now I'm scratching my head on why can't my rpi haos install connect to my S9? DMZing the rpi didn't help. "Error cannot connect" but HA discovers the s9snip automatically and everything looks fine

home-assistant / core