Mikrotik Integration fails with Unicode error

chrisgilldc commented 3 years ago

The problem

The Mikrotik integration is choking on a Unicode response from the router. If the integration is added with the UI, this prevents it from completing setup. If added directly to configuration.yaml, it appears to prevent updates after the initial response.

This is because, for some reason, my GE Range insists on defining its hostname as "û"

Environment

Home Assistant Core release with the issue: 0.118.5
Last working Home Assistant Core release (if known): 0.118.0
Operating environment (OS/Container/Supervised/Core): Hassio 2020.12.6
Integration causing this issue: Mikrotik
Link to integration documentation on our website: https://www.home-assistant.io/integrations/mikrotik

Problem-relevant `configuration.yaml`

mikrotik:
  - name: <HOSTNAME>
    host: <IP>
    username: homeassistant
    password: <PASSWORD>
    verify_ssl: false
    arp_ping: false
    force_dhcp: true
    detection_time: 300

Traceback/Error logs

This is the traceback placed in home_assistant.log when trying to set up the MikroTik integration from the interface.

2020-12-10 14:22:22 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Mikrotik for mikrotik
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/config_entries.py", line 236, in async_setup
    result = await component.async_setup_entry(hass, self)  # type: ignore
  File "/usr/src/homeassistant/homeassistant/components/mikrotik/__init__.py", line 68, in async_setup_entry
    if not await hub.async_setup():
  File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 382, in async_setup
    await self.hass.async_add_executor_job(self._mk_data.update)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 278, in update
    self.update_devices()
  File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 172, in update_devices
    self.all_devices = self.get_list_from_interface(DHCP)
  File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 159, in get_list_from_interface
    result = self.command(MIKROTIK_SERVICES[interface])
  File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 253, in command
    response = list(self.api(cmd=cmd))
  File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 28, in __call__
    yield from self.readResponse()
  File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 60, in readResponse
    reply_word, words = self.readSentence()
  File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 46, in readSentence
    reply_word, words = self.protocol.readSentence()
  File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in readSentence
    sentence = tuple(word for word in iter(self.readWord, ''))
  File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in <genexpr>
    sentence = tuple(word for word in iter(self.readWord, ''))
  File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 206, in readWord
    return word.decode(encoding=self.encoding, errors='strict')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfb in position 12: invalid start byte

Additional information

probot-home-assistant[bot] commented 3 years ago

mikrotik documentation mikrotik source _{^{(message by IssueLinks)}}

probot-home-assistant[bot] commented 3 years ago

Hey there @engrbm87, mind taking a look at this issue as its been labeled with an integration (mikrotik) you are listed as a codeowner for? Thanks! _{^{(message by CodeOwnersMention)}}

ChevySSinSD commented 3 years ago

I have the same issue, and also have two GE appliances with hostname û•. I am seeing the same behaviors as shown above.

valki2 commented 3 years ago

Same here - using latest Mikrotik 6.48

Logger: homeassistant.config_entries Source: components/mikrotik/hub.py:253 First occurred: 9:23:40 (1 occurrences) Last logged: 9:23:40

Error setting up entry Mikrotik for mikrotik Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 236, in async_setup result = await component.async_setup_entry(hass, self) # type: ignore File "/usr/src/homeassistant/homeassistant/components/mikrotik/init.py", line 68, in async_setup_entry if not await hub.async_setup(): File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 382, in async_setup await self.hass.async_add_executor_job(self._mk_data.update) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 278, in update self.update_devices() File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 172, in update_devices self.all_devices = self.get_list_from_interface(DHCP) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 159, in get_list_from_interface result = self.command(MIKROTIK_SERVICES[interface]) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 253, in command response = list(self.api(cmd=cmd)) File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 28, in call yield from self.readResponse() File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 60, in readResponse reply_word, words = self.readSentence() File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 46, in readSentence reply_word, words = self.protocol.readSentence() File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in readSentence sentence = tuple(word for word in iter(self.readWord, '')) File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in sentence = tuple(word for word in iter(self.readWord, '')) File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 206, in readWord return word.decode(encoding=self.encoding, errors='strict') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 15: invalid start byte

ChevySSinSD commented 3 years ago

Here are my details: Home Assistant version: HA 2020.12.7 Mikrotik Hardware: RBD52G-5HacD2HnD RouterOS version: v6.48 (also occurred using v6.44.1, I updated router during troubleshooting, but no change)

ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Mikrotik for mikrotik Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 236, in async_setup result = await component.async_setup_entry(hass, self) # type: ignore File "/usr/src/homeassistant/homeassistant/components/mikrotik/init.py", line 68, in async_setup_entry if not await hub.async_setup(): File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 382, in async_setup await self.hass.async_add_executor_job(self._mk_data.update) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 278, in update self.update_devices() File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 172, in update_devices self.all_devices = self.get_list_from_interface(DHCP) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 159, in get_list_from_interface result = self.command(MIKROTIK_SERVICES[interface]) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 253, in command response = list(self.api(cmd=cmd)) File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 28, in call yield from self.readResponse() File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 60, in readResponse reply_word, words = self.readSentence() File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 46, in readSentence reply_word, words = self.protocol.readSentence() File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in readSentence sentence = tuple(word for word in iter(self.readWord, '')) File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in sentence = tuple(word for word in iter(self.readWord, '')) File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 206, in readWord return word.decode(encoding=self.encoding, errors='strict') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfb in position 12: invalid start byte

ChevySSinSD commented 3 years ago

@luqasz do you have any insight to add from the librouteros code?

luqasz commented 3 years ago

MikroTik is inconsistent with encoding across all platform. Sometimes it's ASCII, sometimes it's UTF-8, sometimes some other. Where does the hostname come from ? DHCP leases ?

ChevySSinSD commented 3 years ago

MikroTik is inconsistent with encoding across all platform. Sometimes it's ASCII, sometimes it's UTF-8, sometimes some other. Where does the hostname come from ? DHCP leases ?

Yes, that is correct. Here is a screen capture from the DHCP Leases in Winbox, the odd hostnames are from appliances made by GE:

luqasz commented 3 years ago

Do they display correctly in CLI ? GE ? û is from what language ? Maybe it is actually some ISO encoding. You can try librouteros as standalone from e.g. ipython and pass different encoding. But still this does not guarantee that other fields will be decoded correctly.

ChevySSinSD commented 3 years ago

The CLI shows \11\FB\95 as the hostname for the offending leases:

GE is short for General Electric. All four of these DHCP leases are for connections to "smart" appliances made by GE. Specifically, I have the following appliances connected to the internet made by GE: https://appliances.monogram.com/us/specs/ZISS420DNSS https://appliances.monogram.com/us/specs/ZTD90DSSNSS https://appliances.monogram.com/us/specs/ZSB9131NSS https://appliances.monogram.com/us/specs/ZDT985SSNSS

As far as I can tell, there is no way to change the hostname configured on the appliance.

luqasz commented 3 years ago

In [26]: b                                                                                                                                                             
Out[26]: b'\x11\xfb\x95'

In [27]: b.decode(encoding='ISO-8859-1')                                                                                                                               
Out[27]: '\x11û\x95'

Seems to decode something. I don't know what encoding is this exactly.

mhpetiwala commented 3 years ago

The CLI shows \11\FB\95 as the hostname for the offending leases:

GE is short for General Electric. All four of these DHCP leases are for connections to "smart" appliances made by GE. Specifically, I have the following appliances connected to the internet made by GE: https://appliances.monogram.com/us/specs/ZISS420DNSS https://appliances.monogram.com/us/specs/ZTD90DSSNSS https://appliances.monogram.com/us/specs/ZSB9131NSS https://appliances.monogram.com/us/specs/ZDT985SSNSS

As far as I can tell, there is no way to change the hostname configured on the appliance.

Same issue GE smart washer and dryer. I'm trying to see if I can write a script on the Mikrotik side to update the hostname (overwrite the crap they're setting it to). First device out of at least 100+ to have this problem. GE/Haier seems to be the culprit plus no way to set it on their side.

ChevySSinSD commented 3 years ago

Thanks for the comment, I wasn't able to figure out how to have the router override the hostname provided by the GE appliances, so if you are successful in finding a way, please post it here!

I submitted a support request from the GE Smart Appliances group and they said this is a "known issue with all of their WiFi modules."

mhpetiwala commented 3 years ago

Thanks for the comment, I wasn't able to figure out how to have the router override the hostname provided by the GE appliances, so if you are successful in finding a way, please post it here!

I submitted a support request from the GE Smart Appliances group and they said this is a "known issue with all of their WiFi modules."

Hi - that didn't work the client GE appliances don't honor the option 12 from the router and still go ahead and set it to the weird characters (seems Chinese but not sure). Anyways after tinkering and going thru the error in the logs I modified the encoding from UTF-8 to latin-1 and after restarting HA (I'm using the base running inside Python venv not the docker one) - it was able to come up fine. I am not sure if we can configure the encoding for mikrotik/device tracker using HA configuration.yaml or the integration - I couldn't find it so just modified the init.py in librouteros package. Here's the change I did: vi /srv/homeassistant/lib/python3.8/site-packages/librouteros/init.py

from librouteros.api import Api

DEFAULTS = { 'timeout': 10, 'port': 8728, 'saddr': '', 'subclass': Api, 'encoding': 'latin-1', 'ssl_wrapper': lambda sock: sock, 'login_method': plain, }

I know this will impact other components using this lib but for now that worked and i can at least use my setup without issues... better way would be to set this encoding when init the conn - as a param - that's supported (i'll go thru the mikrotik component code in HA and try to change it there and update you once it works). let me know if this works for you. I thought there was a PR closed that supported config the encoding using config for mikrotik in HA but I couldn't find it. That would be the best option here. Fortunately, latin-1 works and takes care of these Chinese characters that GE appliances are passing. On another note my next step is to integrate the GE appliances using the gekitchen component (if you or anyone has insight on it please help).

github-actions[bot] commented 3 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

chrisgilldc commented 3 years ago

Just attempted to set up the Mikrotik integration again. Home Assistant Core 2021.4.6, Supervisor 2021.04.0, Home Assistant OS 5.13. In the interface the integration claims it succeds but never gets any entities. In the log, I'm still getting utf-8 errors:

2021-05-02 10:48:50 ERROR (MainThread) [homeassistant.config_entries] Error setting up entry Mikrotik for mikrotik Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/config_entries.py", line 249, in async_setup result = await component.async_setup_entry(hass, self) # type: ignore File "/usr/src/homeassistant/homeassistant/components/mikrotik/__init__.py", line 68, in async_setup_entry if not await hub.async_setup(): File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 386, in async_setup await self.hass.async_add_executor_job(self._mk_data.update) File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 282, in update self.update_devices() File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 177, in update_devices self.all_devices = self.get_list_from_interface(DHCP) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 164, in get_list_from_interface result = self.command(MIKROTIK_SERVICES[interface]) File "/usr/src/homeassistant/homeassistant/components/mikrotik/hub.py", line 258, in command response = list(self.api(cmd=cmd)) File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 28, in __call__ yield from self.readResponse() File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 60, in readResponse reply_word, words = self.readSentence() File "/usr/local/lib/python3.8/site-packages/librouteros/api.py", line 46, in readSentence reply_word, words = self.protocol.readSentence() File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in readSentence sentence = tuple(word for word in iter(self.readWord, '')) File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 189, in <genexpr> sentence = tuple(word for word in iter(self.readWord, '')) File "/usr/local/lib/python3.8/site-packages/librouteros/protocol.py", line 206, in readWord return word.decode(encoding=self.encoding, errors='strict') UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfb in position 12: invalid start byte If homeassistant itself doesn't support valid latin1 character, then the Mikrotik integration should accept them - since it's valid in the API - and strip or translate them appropriately.

github-actions[bot] commented 3 years ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core