search

exxeleron / qPython

interprocess communication between Python and kdb+
http://www.devnet.de
Apache License 2.0
151 stars 90 forks source link

added qtable to dataframe conversion utility #9

Closed buckie closed 9 years ago

buckie commented 10 years ago

Pertains to https://github.com/exxeleron/qPython/issues/6

Decided it made more sense to go the pandas route over numpy because to get a true "native" representation from q (which is what I was after) you need to have nulls in any datatype. Unfortunately, I couldn't get numpy's masked arrays to play nicely with the datetime/timedelta columns (there are a couple outstanding bugs in it's implementation that were found in numpy 1.7) during reconstruction of component columns into the table itself. Also, the type promotion's interplay with q's default values for nulls causes headaches.

Instead panda's DataFrames seemed to be the winner, as they mirror q's tables much more closely in terms of representation and usage (but not memory layout or types). I figure that high performance work will always use numpy and have the special cases worked in but that for more exploratory work, a DataFrames was the likely choice.

Some of the QTemporalList to numpy MaskedArray functions are included as well, in case someone else want's to pick up where I left off.

Signed-off-by: buckie wjmartino@gmail.com

maciejlach commented 9 years ago

Pandas support has been incorporated in qPython (SHA: 8aa5cb1691448c00990579b3e38f1e14b9054dab).