Error condition being triggered unexpectedly

[richardbutler]

Hi Michael

I have a "good" query, which returns a table back from KDB via the q command line. Most of the time, it returns fine through node-q, but when I give it a certain date range (the query is a time series call), the condition if (type === -128) resolves to true, and the entire binary response gets spat back at me as an error.

The query is...

.ceq.getQuotes[`BP.L;2015.09.08;10:52:42.861t;2015.09.08;10:57:36.911t;(`source`tz)!(`casheq;`UTC)]

...which I'm submitting as a raw string, and works for most dates. I assume the response packet must contain a -128 for another purpose, possibly one of the column values, but my binary/hex isn't great so I can't tell.

Any ideas what might be causing this? It doesn't seem to be isolated to a particular row in the response - a binary search of the date range doesn't uncover anything obvious.

[michaelwittig]

Just a few short questions to get a better idea of whats going on:

Can you tell me the output of meta and count function on the resulting data.

does the same function invocation in terms of parameters always fail?

What version of node-q are you using?

Am 09.09.2015 um 13:21 schrieb Richard Butler notifications@github.com:

Hi Michael

I have a "good" query, which returns a table back from KDB via the q command line. Most of the time, it returns fine through node-q, but when I give it a certain date range (the query is a time series call), the condition if (type === -128) resolves to true, and the entire binary response gets spat back at me as an error.

The query is...

.ceq.getQuotes[BP.L;2015.09.08;10:52:42.861t;2015.09.08;10:57:36.911t;(sourcetz)!(casheq;`UTC)] ...which I'm submitting as a raw string, and works for most dates. I assume the response packet must contain a -128 for another purpose, possibly one of the column values, but my binary/hex isn't great so I can't tell.

Any ideas what might be causing this? It doesn't seem to be isolated to a particular row in the response - a binary search of the date range doesn't uncover anything obvious.

— Reply to this email directly or view it on GitHub.

[richardbutler]

Can you tell me the output of meta and count function on the resulting data.

q) meta quotes
c           | t f a
------------| -----
date        | d
sym         | s
time        | n
bid         | f
bsize       | f
ask         | f
asize       | f
exchangeTime| n

q) count quotes
1882

does the same function invocation in terms of parameters always fail?

Always fails with these exact dates and times - move them a few milliseconds either side and it doesn't fail.

What version of node-q are you using?

1.0.0 on Node 0.10.33.

[michaelwittig]

One thing I forgot to ask: What's the value of err.message?

[richardbutler]

A lot of binary as ascii - tried pasting, but barely any of it is preserved - lots of �.

[michaelwittig]

Maybe this is related to compression (are you connection to a server not localhost?).

Can you make a change to node_modules/node-q/index.js on line 93. Change

b.writeUInt8(0x3, n);

to

b.writeUInt8(0x0, n);

and let's see if it works. This basically tells that your client connection is not able to hanlde compression, so the q process will not compress the data when sending it via network. (http://code.kx.com/wiki/Reference/ipcprotocol#Handshake)

[richardbutler]

It gets weirder - that hangs all calls, socket data handler never gets called. No, not localhost.

[michaelwittig]

is the data confidential or can you send me the table saved via

save `quotes

to post@michaelwittig.info ?

[richardbutler]

I'm afraid it is - even though it's only quote data, it's more than my life is worth to send it outside these walls, which is frustrating.

Sorry, I'm not giving you much to go on here. I guess I figured there might be a case where the error check was getting confused with another header byte (length, columns?), but hard to tell without stepping through the code against the query.

[michaelwittig]

No problem. I can fully understand your point. I will need ti find some free time to investigate this bug.

[michaelwittig]

hi @richardbutler I was playing around a little with compression and added a few tests.

Can you try one thing:

count -8!yourtable

and

count -18!yourtable

If the numbers are the same than there is no compression going on :)

[michaelwittig]

And maybe you can try if version 2.0.0 of this lib still has the issue.