search

tfgm / sbedecoder

Python based Simple Binary Encoding (SBE) decoder
MIT License
76 stars 44 forks source link

errors: invalid tcpdump header #8

Closed rumcode closed 8 years ago

rumcode commented 8 years ago

Hi, I've been trying to get your program to work, but I get a message -packages\dpkt\pcap.py", line 122, in init raise ValueError('invalid tcpdump header') ValueError: invalid tcpdump header

My xml file is from the CME (followed your link) and so is the data which is e-mini futures posted at http://www.cmegroup.com/market-data/datamine-historical-data.html

I was running 2.7 python (anaconda) and ran setup.py before running mdp_decoder.py

I ran your tests and they were fine, but it seems that you don't use real data in your test.

Any suggestions?

Thanks in advance.

tfgm-bud commented 8 years ago

@rumcode Thanks for trying the SBE decoder. I took a look at those sample files at http://www.cmegroup.com/market-data/datamine-historical-data.html. While the files there are MDP 3.0 they aren't SBE encoded -- they are plan ASCII FIX files:

$ gunzip XCME_MD_ES_FUT_20160315.gz
$ cat XCME_MD_ES_FUT_20160315 | tr '\001' '\|' | head
1128=9|9=131|35=f|49=CME|34=5915937|52=20160314214500009131503|60=20160314214500001946065|75=20160315|5799=10000000|1151=ES|326=21|327=0|1174=4|10=075|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915938|52=20160314214500437290133|60=20160314214500437207105|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434488|270=201875.0|271=1|346=1|1023=1|10=153|
1128=9|9=182|35=X|49=CME|75=20160315|34=5915939|52=20160314214500964855160|60=20160314214500964728850|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434489|270=202425.0|271=1|346=1|1023=10|10=228|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915940|52=20160314214501236422156|60=20160314214501236279685|5799=10000100|268=1|279=1|269=1|48=6505|55=ESM6|83=4428027|270=201000.0|271=14|346=5|1023=2|10=146|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915941|52=20160314214501260912493|60=20160314214501260788281|5799=10000100|268=1|279=1|269=1|48=49705|55=ESH6|83=2434490|270=201875.0|271=4|346=2|1023=1|10=164|
1128=9|9=183|35=X|49=CME|75=20160315|34=5915942|52=20160314214501469025445|60=20160314214501468880637|5799=10000100|268=1|279=1|269=0|48=49705|55=ESH6|83=2434491|270=201575.0|271=12|346=4|1023=10|10=016|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915943|52=20160314214502284489462|60=20160314214502284402077|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434492|270=201900.0|271=3|346=1|1023=2|10=160|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915944|52=20160314214502900288541|60=20160314214502900214739|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434493|270=201925.0|271=3|346=1|1023=3|10=161|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915945|52=20160314214503380287759|60=20160314214503380201283|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434494|270=201950.0|271=3|346=1|1023=4|10=168|
1128=9|9=181|35=X|49=CME|75=20160315|34=5915946|52=20160314214503973789379|60=20160314214503973709166|5799=10000100|268=1|279=0|269=1|48=49705|55=ESH6|83=2434495|270=202000.0|271=3|346=1|1023=6|10=193|

The upside for you is that you don't need to decode those files to use them. The downside is that you can't take advantage of sbe decoder's ability to print the tag=value pairs in a more human readable format or the sbe decoder's ability to assemble market data books. :-(

Since it is just FIX, it is pretty easy to work with though. A quick and dirty approach to get something more readable is to just use "split" to parse the lines into a list of key value pairs and then use a dictionary lookup to translate the keys (and values as necessary) to a textual form.

If you need book building, you probably could adapt our 'orderbook' pretty easily too:

https://github.com/tfgm/sbedecoder/tree/master/orderbook