search

tarantool / tarantool-python

Python client library for Tarantool
https://www.tarantool.io
BSD 2-Clause "Simplified" License
100 stars 48 forks source link

New python-msgpack version brokes connector #155

Closed RepentantGopher closed 4 years ago

RepentantGopher commented 4 years ago

New version of python-msgpack library was released which enforce only string keys in map by default. This change brokes whole connector because version of python msgpack library is not enforced in requirements.

Commit https://github.com/msgpack/msgpack-python/commit/d8e3cf0563989a660398318a7c788645124e1d8b

Error on connection to tarantool:

test_tags.py:6: in <module>
    from .common import (
common.py:22: in <module>
    operational_storage = tarantool.connect("localhost", 3303, "tarantool", "tarantool")
venv/lib/python3.7/site-packages/tarantool/__init__.py:51: in connect
    encoding=encoding)
venv/lib/python3.7/site-packages/tarantool/connection.py:130: in init
    self.connect()
venv/lib/python3.7/site-packages/tarantool/connection.py:208: in connect
    raise NetworkError(e)
E   tarantool.error.NetworkError: int is not allowed for map key
pcdinh commented 4 years ago

I got the same issue in Ubuntu 18.04, Python 3.8.2, Tarantool 0.6.6

Totktonada commented 4 years ago

Should be fixed with PR #156.

artembo commented 4 years ago

In the new release of msgpack also encoding parameter was removed.

Remove encoding option from the Packer and Unpacker.

@Totktonada suggested

If the encoding option is given to the connector, then you can programmatically go through recursively and do .decode (encoding = 'utf-8') for all bytes (Python 3) or str (Python 2) values. It seems like this should work, and fully transparently support everything that was previously supported:

Python 2:

>>> import msgpack
>>> msgpack.unpackb(b'\xd9\x08\xd0\xb0\xd0\xb1\xd0\xb2\xd0\xb3', raw=True).decode(encoding='utf-8')
u'\u0430\u0431\u0432\u0433'
>>> msgpack.unpackb(b'\xd9\x08\xd0\xb0\xd0\xb1\xd0\xb2\xd0\xb3', encoding='utf-8')
u'\u0430\u0431\u0432\u0433'

Python 3:

>>> import msgpack
>>> msgpack.unpackb(b'\xd9\x08\xd0\xb0\xd0\xb1\xd0\xb2\xd0\xb3', raw=True).decode(encoding='utf-8')
'абвг'
>>> msgpack.unpackb(b'\xd9\x08\xd0\xb0\xd0\xb1\xd0\xb2\xd0\xb3', encoding='utf-8')
__main__:1: DeprecationWarning: encoding is deprecated, Use raw=False instead.
'абвг'
Totktonada commented 4 years ago

I dived a bit further and it reminds me changes we did for msgpack-0.5.2.

We already use raw=False when encoding is 'utf-8' since msgpack-0.5.2 (2097884cc531a419cdf30e1e153094994faf274b), because otherwise it emits the deprecation warning regarding encoding option usage. So we should just continue passing raw=False on msgpack-1.0.0.