exxeleron / qPython

interprocess communication between Python and kdb+
http://www.devnet.de
Apache License 2.0
151 stars 90 forks source link

pandas parsing failed with pandas >1.3.5 #82

Open robinsedman opened 2 years ago

robinsedman commented 2 years ago

Running the following works with pandas <=1.3.5 but fails with >1.3.5

from qpython import qconnection
with qconnection.QConnection(server, username, port, raw=False) as q:
    query = f"(0b, 1b, 1b, 0b, 1b)"
    data = q(query, pandas=True)

Python 3.9 qpython==2.0.0

wrkyle commented 2 years ago

I can confirm this. I was trying to get a table with an integer column and I got this error arising from the pandas conversion:

File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\qpython-2.0.0-py3.9.egg\qpython\qconnection.py", line 380, in receive
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\qpython-2.0.0-py3.9.egg\qpython\qreader.py", line 139, in read
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\qpython-2.0.0-py3.9.egg\qpython\qreader.py", line 338, in _read_general_list
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\qpython-2.0.0-py3.9.egg\qpython\qreader.py", line 338, in <listcomp>
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\qpython-2.0.0-py3.9.egg\qpython\qreader.py", line 227, in _read_object
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\qpython-2.0.0-py3.9.egg\qpython\_pandas.py", line 116, in _read_list
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\pandas-1.4.2-py3.9-win-amd64.egg\pandas\core\series.py", line 4960, in replace
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\pandas-1.4.2-py3.9-win-amd64.egg\pandas\core\generic.py", line 6745, in replace
    new_data = self._mgr.replace(
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\pandas-1.4.2-py3.9-win-amd64.egg\pandas\core\internals\managers.py", line 441, in replace
    return self.apply(
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\pandas-1.4.2-py3.9-win-amd64.egg\pandas\core\internals\managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\pandas-1.4.2-py3.9-win-amd64.egg\pandas\core\internals\blocks.py", line 683, in replace
    mask = missing.mask_missing(values, to_replace)
  File "C:\Users\kyle_w\AppData\Roaming\Python\Python39\site-packages\pandas-1.4.2-py3.9-win-amd64.egg\pandas\core\missing.py", line 98, in mask_missing
    new_mask = new_mask.to_numpy(dtype=bool, na_value=False)
AttributeError: 'bool' object has no attribute 'to_numpy'

The error disappeared when I removed the integer column from the table I was getting.

EDIT: I rolled back to pandas 1.3.5 using pip install -I pandas==1.3.5 and the error is gone.

wrkyle commented 2 years ago

To future readers, this project is not being actively maintained. The Fintech Open Source Foundation has forked it and is actively maintaining: https://github.com/finos/qPython.

Install with pip install qpython3. Imports are the same so no code changes required.