Encoding issue - Githubissues

Hi folks, I've been getting this issue recently and I can't really figure out if my environment is faulty or if some other piece is. Long story short here's the problem and my env: ERROR:

>>> import salt.client
>>> local = salt.client.LocalClient()
>>> local.cmd('*', 'test.ping')
Traceback (most recent call last):
  File "msgpack/_unpacker.pyx", line 461, in msgpack._unpacker.Unpacker._unpack
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 753, in cmd
    **kwargs):
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 1621, in get_cli_event_returns
    **kwargs
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 1196, in get_iter_returns
    for raw in ret_iter:
  File "/usr/lib/python3.6/site-packages/salt/client/__init__.py", line 1121, in get_returns_no_block
    no_block=True, auto_reconnect=self.auto_reconnect)
  File "/usr/lib/python3.6/site-packages/salt/utils/event.py", line 649, in get_event
    ret = self._get_event(wait, tag, match_func, no_block)
  File "/usr/lib/python3.6/site-packages/salt/utils/event.py", line 548, in _get_event
    raw = self.subscriber.read_sync(timeout=wait)
  File "/usr/lib/python3.6/site-packages/salt/transport/ipc.py", line 707, in read_sync
    return ret_future.result()
  File "/usr/lib64/python3.6/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib64/python3.6/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python3.6/site-packages/salt/transport/ipc.py", line 683, in _read_sync
    raise exc_to_raise  # pylint: disable=E0702
  File "/usr/lib/python3.6/site-packages/salt/transport/ipc.py", line 654, in _read_sync
    for framed_msg in self.unpacker:
  File "msgpack/_unpacker.pyx", line 536, in msgpack._unpacker.Unpacker.__next__
  File "msgpack/_unpacker.pyx", line 481, in msgpack._unpacker.Unpacker._unpack
msgpack.exceptions.UnpackValueError: 'utf-8' codec can't decode byte 0x82 in position 22: invalid start byte

MY ENV

[root@localhost dragos]# python3.6
Python 3.6.3 (default, Jan  4 2018, 16:40:53) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux

[root@localhost /]# salt --version
salt 2018.3.2 (Oxygen)

CentOS Linux release 7.5.1804 (Core) 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"

I've tweaked the source code a little bit and I got it to work but my main concern is why it doesn't work with the official code. Here's what I did: In /lib/python3.6/site-packages/salt/transport/ipc.py I replaced this piece of code which pops up twice I think:

        if six.PY2:
            encoding = None
        else:
            encoding = 'utf-8'

with

        if six.PY2:
            encoding = None
        else:
            encoding = None

and then I replaced every instance I found of framed_msg['body'] with framed_msg['body'.encode()] and surely enough it worked (csr2 was not powered on so there's no output collected):

>>> local.cmd('*', 'net.arp')
{'csr1': {'comment': '', 'result': True, 'out': [{'interface': 'GigabitEthernet1', 'ip': '192.168.56.51', 'mac': '08:00:27:30:B0:62', 'age': 0.0}, {'interface': 'GigabitEthernet1', 'ip': '192.168.56.101', 'mac': '08:00:27:00:93:D4', 'age': 3.0}, {'interface': 'GigabitEthernet2', 'ip': '192.168.100.11', 'mac': '08:00:27:81:FA:04', 'age': 0.0}]}, 'csr2': False}

>>> local.cmd('*', 'net.interfaces')
{'csr1': {'comment': '', 'result': True, 'out': {'GigabitEthernet1': {'is_enabled': True, 'description': 'WAN IFACE ', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '08:00:27:30:B0:62', 'speed': 1000}, 'GigabitEthernet2': {'is_enabled': True, 'description': '', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '08:00:27:81:FA:04', 'speed': 1000}, 'Loopback104': {'is_enabled': True, 'description': 'LOOP-104', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '', 'speed': 8000}, 'Loopback0': {'is_enabled': True, 'description': '', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '', 'speed': 8000}, 'Loopback102': {'is_enabled': True, 'description': 'LOOP-102', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '', 'speed': 8000}, 'Loopback101': {'is_enabled': True, 'description': 'LOOP-101', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '', 'speed': 8000}, 'Loopback100': {'is_enabled': True, 'description': 'LOOP-100', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '', 'speed': 8000}, 'Loopback103': {'is_enabled': True, 'description': 'LOOP-103', 'last_flapped': -1.0, 'is_up': True, 'mac_address': '', 'speed': 8000}}}, 'csr2': False}

I find this a bit strange since I replaced the hard-coded encoding 'utf-8' with the default one provided by the encode() method which as far as I know (and I might be wrong here) is 'utf-8'. I guess my question at this point is if there is some sort of issue with the way I've set up python or salt in my environment. Where could I start looking please? Any help/suggestion is highly appreciated. Thanks!

napalm-automation / napalm-salt

Encoding issue #50