null values in enumerated columns causes crash

[alexzhang2010]

Below is my test case. Please advise.

Thanks! Ray

from pyq import q
import numpy as np
import pandas as pd

q('suits: `hearts`clubs`spades`diamonds')
q.good = q('([] c1: 1 2 3 4; c2:`suits$(`hearts`clubs`clubs`diamonds))')
na = np.array(q.good)
# column c2 now has a null value
q.bad = q('([] c1: 1 2 3 4; c2:`suits$(`hearts`clubs``diamonds))')
q.bad_column = q('bad[`c2]')
# q.bad and q.bad_column both  type <class 'pyq.K'>
# each of the following 3 lines will crash due to null values in enumerated column
t = list(q.bad_column)
na = np.array(q.bad)
df = pd.DataFrame(na)

[abalkin]

@alexzhang2010,

Can you post the output of the following command?

pyq --versions

This will help us identify the versions of software that you are using.

I was not able to reproduce the crash. On my system,

$ pyq --versions
PyQ 4.1.1
NumPy 1.12.1
KDB+ 3.5 (2017.05.02) m32
Python 3.6.1 (default, Jun 13 2017, 14:45:26)
[GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]

I get

>>> list(q.bad_column)
['hearts', 'clubs', 'hearts', 'diamonds']
>>> np.array(q.bad)
array([(1, 'hearts'), (2, 'clubs'), (3, 'hearts'), (4, 'diamonds')],
      dtype=[('c1', '<i8'), ('c2', 'O')])
>>> pd.DataFrame(np.array(q.bad))
   c1        c2
0   1    hearts
1   2     clubs
2   3    hearts
3   4  diamonds

I do see that the null value is not converted correctly (the value of c2 in the third row should be an empty string, not "hearts"), but there is no crash.

Can you also post the output that you see when pyq crashes?

[abalkin]

@alexzhang2010,

Thank you for your report. We identified a bug introduced in PyQ 4.0. Please try applying the attached patch. This will be included in the next bug-fix release.

0001-BUG-Fixes-955-Check-for-negative-values-including-0N.patch.txt

[alexzhang2010]

Here is my package versions. An earlier version of kdb?

Thanks.

$ pyq --versions PyQ 4.1.1 NumPy 1.13.1 KDB+ 3.2 (2015.05.27) l64 Python 3.6.2 |Anaconda custom (64-bit)| (default, Jul 20 2017, 13:51:32) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

Here is the output while it crashes

$pyq nullbug.py

Sorry, this application or an associated library has encountered a fatal error and will exit. If known, please email the steps to reproduce this error to tech@kx.com with a copy of the kdb+ startup banner. Thank you. /apps/shared/anaconda3/bin/pyq() [0x469d19] /lib64/libpthread.so.0() [0x334780f7e0] /apps/shared/anaconda3/lib/python3.6/site-packages/pyq/_k.cpython-36m-x86_64-linux-gnu.so(+0x78d4) [0x7fe04136a8d4] /apps/shared/anaconda3/lib/python3.6/site-packages/pyq/_k.cpython-36m-x86_64-linux-gnu.so(+0x7bad) [0x7fe04136abad] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(+0xa520e) [0x7fe04f94f20e] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(+0xa5668) [0x7fe04f94f668] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(+0xda2ac) [0x7fe04f9842ac] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(_PyObject_FastCallDict+0x9e) [0x7fe04f915c1e] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(+0x14895b) [0x7fe04f9f295b] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(_PyEval_EvalFrameDefault+0x2c40) [0x7fe04f9f5d40] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(+0x147100) [0x7fe04f9f1100] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCodeEx+0x63) [0x7fe04f9f1583] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(PyEval_EvalCode+0x3b) [0x7fe04f9f15cb] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(PyRun_FileExFlags+0x130) [0x7fe04fa23ee0] /apps/shared/anaconda3/bin/../lib/libpython3.6m.so.1.0(PyRun_SimpleFileExFlags+0x173) [0x7fe04fa254a3] Segmentation fault

[abalkin]

@alexzhang2010, our posts have crossed. Please see the patch I posted a few minutes ago. Thanks for your report.

[alexzhang2010]

awesome, let me try this out

[alexzhang2010]

I confirm the bug has been addressed. thank you!