search

exxeleron / qPython

interprocess communication between Python and kdb+
http://www.devnet.de
Apache License 2.0
152 stars 89 forks source link

Sending Unicode strings as QSTRING type #38

Closed majorsimon closed 8 years ago

majorsimon commented 8 years ago

On Python 2.7: I'm trying to send non-ASCII (utf-8) data to a q process. It works fine when sending the data as a QSYMBOL (using str.encode('utf-8')), but always fails when trying to send the data as a QSTRING.

If I send the data as a unicode string I get: QWriterException: Unable to serialize type: <type 'unicode'> in line 119 of qwriter.py.

If I convert the unicode string to a str type (using str.encode('utf-8')) I get: UnicodeDecodeError: 'ascii' codec can't decode byte [...]: ordinal not in range(128). in line 167 of qwriter.

Setting encoding to 'utf-8' when initialising the connection doesn't help either. Do I need to write a custom _write_string override to get this to work properly?

slawomirpanic commented 8 years ago

Q internal-process communication doesn't support Unicode, only strings coded in ASCII. Using QSYMBOL as a solution is not supported workaround, which can fail and provide partial information.

You may try to encode string into an UTF-8 byte array in Python and send it to a q process. Then decode the string on a q side. In order to send byte list use type QBYTE_LIST available in qPython.

majorsimon commented 8 years ago

Okay, that fixed my problem. Didn't realise IPC doesn't support unicode! Cheers