Trying to load tallbook from a large file results in `domain error

[connorgoggins]

I'm currently trying to decode data from CME (xcme_md_es_opt_20190322-r-00013) using this repo to generate the tallbook. However, when I try to run the relevant command, the script runs for a while and then returns the following error:

2019.04.09D18:27:50.787671000|algo-trading-big|cmedecoder|cmedecoder|INF|logfile|Finished processing file: :sample/
xcme_md_es_opt_20190322-r-00013_chunks/xcme_md_es_opt_20190322-r-00013_chunk0
'domain
  [11] /home/cme/deploy/code/cmedecoder/tall_book.q:11: bk0:{[x;y;z;bk;d] a:.[bk;(z;::;1_ml);:;-1_'bk[z;;ml
:x+til d-x]];.[a;(z;::;x);:;y]}

Any suggestions? Also, it might be worth noting that the file is rather large (~3gb compressed, ~20gb uncompressed).

[frenzy21]

Hi,

Someone will respond tomorrow.

James

[connorgoggins]

Hi - just wanted to follow up on this!

[JRobbins91]

Hi,

I've been looking into your query and it appears that either your rawquote or rawdefinitions tables in the hdb directory have not been populated properly. To verify this, could you please load your database (type q hdb in your deploy directory) and then post the results of the following queries:

select MarketDepth by sym from rawdefinitions

and

select MDPriceLevel by sym from rawquote

The results from this should give us a clearer idea of where the issue lies and how to go about resolving it.

Kind regards, Josh

[frenzy21]

Hi Connor,

Was this resolved?

James