search

exxeleron / qPython

interprocess communication between Python and kdb+
http://www.devnet.de
Apache License 2.0
152 stars 89 forks source link

slow uncompression #1

Closed maciejlach closed 10 years ago

maciejlach commented 10 years ago

The uncompress on larger results is very slow.

The test query result is the a 20,000 row table with bid/ask columns. From a q IDE this takes <200ms. qPython receives the data from the server quickly (<100ms), but the uncompress itself takes 4-5 seconds.

Uncompression is handled in qreader.py, the _uncompress function called by read_data.

Same test in Java and the uncompress time is negligible.

Details:

maciejlach commented 10 years ago

Current implementation of the uncompression algorithm is on average 10 times faster than the original one.

maciejlach commented 10 years ago

Uncompression algorithm can now be compiled as C-extension via Cython.