Parsing 552 <NoSides> incorrectly

[kmalyar-tic]

Hi @daino3 ,

So we are still chugging away. We have encountered an issue. It looks like for tag 552 it is pulling a bunch of inappropriate tags into a nested group when it shouldn't

eg:

http://www.quickfixengine.org/FIX44.xml

But in the actual parsing of a message:

the actual fix message that caused this: "8=FIX.4.4\u00019=458\u000135=AE\u000134=1613\u000149=blablabla-blablabla\u000156=blablabla-blablabla\u000152=20170831-18:56:45.294\u0001571=LSZUFPPS49391601999999795\u0001568=tcr-2017-08-30-17-05-23\u0001150=F\u000139=2\u000117=blablabla\u0001570=N\u000155=USD/MXN\u0001167=FOR\u000138=6000000.000000\u00016054=106931160.000000\u000144=17.821860\u000132=blablabla.000000\u000131=99.99\u0001194=99.99\u0001195=0.0000\u000175=20170831\u00016215=SP\u000160=20170831-18:56:45.294\u0001552=1\u000154=1\u000137=blablabla\u0001453=1\u0001448=blablabla\u0001447=D\u0001452=11\u00011=blablabla blablabla blablabla-blablabla NA\u000115=USD\u000140=D\u0001487=0\u000164=20170905\u000110=002\u0001"

Any chance you can help us out?

So far its an awesome plugin!

Regards, Kelvin

[daino3]

thank you for the bug report, @kmalyar-tic. I will try to get to it this week.

[cbusbey]

@kmalyar-tic can you try grabbing the latest quickfix-jruby gem and trying again? This version includes quickfixj 1.6.4, which might correct the issue you are seeing.

[kmalyar-tic]

@cbusbey, I came in and tried to do as you suggested. Unfortunately, i'm getting bitten by not really understanding the ruby development environment. I tried installing the plugin again (hoping it would trigger an update of the gem) no luck.

I tried this:

 ~/tmp/logstash [] $ gem list

*** LOCAL GEMS ***

bundler-unload (1.0.2)
did_you_mean (default: 1.0.1)
executable-hooks (1.3.2)
gem-wrappers (1.3.0)
jar-dependencies (default: 0.3.9)
jruby-openssl (0.9.20 java, 0.9.18 java)
jruby-readline (1.1.1 java)
json (1.8.3 java)
minitest (default: 5.4.1)
net-telnet (default: 0.1.1)
power_assert (default: 0.2.3)
psych (2.2.2 java)
racc (1.4.14 java)
rake (12.0.0, default: 10.4.2)
rdoc (default: 4.2.0)
rubygems-bundler (1.4.4)
rvm (1.11.3.9)
test-unit (default: 3.1.1)
 ~/tmp/logstash [] $ ls
bin           config        data     Gemfile.jruby-1.9.lock  LICENSE  logstash-core             NOTICE.TXT  sample.conf
CHANGELOG.md  CONTRIBUTORS  Gemfile  lib                     logs     logstash-core-plugin-api  path2       vendor

No joy, can you give me the dummy's guide to how I would grab the latest version (I try not to ask the same question twice though if its any consolation).

EDIT:

Looks like I have it installed here:

logstash/vendor/bundle/jruby/1.9/gems: quickfix-jruby-1.6.5-java:

I also have 1.6.0 so i'm not sure of the overlap...

If i need to actually change any other configuration file (to make sure i'm using the right version) can you let me know?

EDIT2:

Back to the original error:

[2017-09-11T09:45:39,239][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method get_field_type_enum' for #<LogStash::Filters::DataDictionary:0x19a0e733>>, :backtrace=>["/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:35:infield_type'", "org/jruby/RubyArray.java:1613:in each'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:34:infield_type'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:104:in is_group?'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:66:infield_map_to_hash'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:24:in to_hash'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_protocol.rb:52:infilter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:164:inmulti_filter'", "org/jruby/RubyArray.java:1613:in each'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:161:inmulti_filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filter_delegator.rb:43:in multi_filter'", "(eval):418:ininitialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):415:ininitialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):458:ininitialize'", "org/jruby/RubyArray.java:1613:in each'", "(eval):454:ininitialize'", "org/jruby/RubyProc.java:281:in call'", "(eval):278:infilter_func'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:370:in filter_batch'", "org/jruby/RubyProc.java:281:incall'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224:in each'", "org/jruby/RubyHash.java:1342:ineach'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223:in each'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:369:infilter_batch'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:350:in worker_loop'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:317:instart_workers'"]}

Thanks!

[cbusbey]

Taking a closer look at the message, I believe the plugin is working as expected. NoSides is a repeating group in message type AE (and for all messages that use it in the FIX44 spec for that matter). The FIX field value of NoSides may be 1 or 2, but a value of 1 indicates there is one group, 2 indicates there are two. The logstash plugin appropriately converts this to an array of group objects. If you want the value of NoSides, this is the length of the NoSides array that gets produced by the plugin.

[daino3]

thanks for taking care of this @cbusbey! 🥇

[kmalyar-tic]

Wait my pipeline is completely broken trying to follow advice, can someone help me out with thel atest? although this doesn't make any sense to me.

Thanks Kelvin

[daino3]

@kmalyar-tic:

The error you're seeing is due to correctly installing quickfix-jruby 1.6.5. To have your gem revert to 1.6.0, simply uninstall 1.6.5.

gem uninstall quickfix-jruby --version 1.6.5

You can confirm you're using 1.6.0 with:

gem list quickfix-jruby

I have opened a ticket to have the plugin work with 1.6.5

[kmalyar-tic]

hi @daino3 ,

 ~ [] $ gem uninstall quickfix-jruby --version 1.6.5
 ~ [] $ gem list quickfix-jruby

*** LOCAL GEMS ***

 ~ [] $

 ~/tmp/logstash [] $ gem install quickfix-jruby --version 1.6.0
Ignoring executable-hooks-1.3.2 because its extensions are not built.  Try: gem pristine executable-hooks --version 1.3.2
Ignoring gem-wrappers-1.3.0 because its extensions are not built.  Try: gem pristine gem-wrappers --version 1.3.0
Error loading RubyGems plugin "/home/kmalyar/.rvm/gems/jruby-9.1.7.0@global/gems/executable-hooks-1.3.2/lib/rubygems_plugin.rb": no such file to load -- executable-hooks/wrapper (LoadError)
Error loading RubyGems plugin "/home/kmalyar/.rvm/gems/jruby-9.1.7.0@global/gems/gem-wrappers-1.3.0/lib/rubygems_plugin.rb": no such file to load -- gem-wrappers (LoadError)
Fetching: quickfix-jruby-1.6.0-java.gem (100%)
Successfully installed quickfix-jruby-1.6.0-java
1 gem installed
 ~/tmp/logstash [] $ ./bin/logstash -f ~/git/fix/pipelines/dev/fix.conf
Sending Logstash's logs to /home/kmalyar/tmp/logstash/logs which is now configured via log4j2.properties
[2017-09-14T09:27:41,051][INFO ][logstash.filters.fixprotocol] Using version 0.1.x filter plugin 'fix_protocol'. This plugin isn't well supported by the community and likely has no maintainer.
[2017-09-14T09:27:42,283][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://appl166-stc:9200/]}}
[2017-09-14T09:27:42,286][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://appl166-stc:9200/, :path=>"/"}
[2017-09-14T09:27:42,415][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0xfdf4349 URL:http://appl166-stc:9200/>}
[2017-09-14T09:27:42,417][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x54ca0044 URL://appl166-stc>]}
[2017-09-14T09:27:42,500][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-09-14T09:27:42,729][INFO ][logstash.pipeline        ] Pipeline main started
[2017-09-14T09:27:42,801][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2017-09-14T09:27:42,954][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined method `get_field_type_enum' for #<LogStash::Filters::DataDictionary:0x37cd8e34>", "backtrace"=>["/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:35:in `field_type'", "org/jruby/RubyArray.java:1613:in `each'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:34:in `field_type'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:104:in `is_group?'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:66:in `field_map_to_hash'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_message.rb:24:in `to_hash'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.2/lib/logstash/filters/fix_protocol.rb:52:in `filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filter_delegator.rb:43:in `multi_filter'", "(eval):418:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):415:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):458:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):454:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):278:in `filter_func'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:370:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223:in `each'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:369:in `filter_batch'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:350:in `worker_loop'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:317:in `start_workers'"]}

[daino3]

gem install quickfix-jruby --version 1.6.0

[kmalyar-tic]

@daino3

If you check in the comment, I did that before running it (and it breaking)