search

connamara / logstash-filter-fix_protocol

FIX Protocol Logstash Filter
Other
42 stars 27 forks source link

Fatal Exception on Nested Groups/SubGroups #77

Closed kmalyar-tic closed 7 years ago

kmalyar-tic commented 7 years ago

Hi @daino3 ,

We have a message with (FIXv4.4) and it uses the group and then subgroup tags.

eg: field number='453' name='NoPartyIDs' type='NUMINGROUP' and field number='802' name='NoPartySubIDs' type='NUMINGROUP'

It crashes when it finds another group (beyond the first level it seems like)

[2017-06-21T11:06:35,571][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>"undefined methodto_hash' for #", "backtrace"=>["/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.1/lib/logstash/filters/fix_message.rb:87:in field_map_to_hash'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.1/lib/logstash/filters/fix_message.rb:72:infield_map_to_hash'", "org/jruby/RubyRange.java:479:in each'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/activesupport-4.1.16/lib/active_support/core_ext/range/each.rb:7:ineach_with_time_with_zone'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.1/lib/logstash/filters/fix_message.rb:69:in field_map_to_hash'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.1/lib/logstash/filters/fix_message.rb:25:into_hash'", "/home/kmalyar/tmp/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-fix_protocol-0.3.1/lib/logstash/filters/fix_protocol.rb:52:in filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:145:indo_filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:164:in multi_filter'", "org/jruby/RubyArray.java:1613:ineach'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/filter_delegator.rb:43:inmulti_filter'", "(eval):211:in initialize'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):203:in initialize'", "org/jruby/RubyProc.java:281:incall'", "(eval):142:in filter_func'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:370:infilter_batch'", "org/jruby/RubyProc.java:281:in call'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:224:ineach'", "org/jruby/RubyHash.java:1342:in each'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:223:ineach'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:369:in filter_batch'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:350:inworker_loop'", "/home/kmalyar/tmp/logstash/logstash-core/lib/logstash/pipeline.rb:317:in start_workers'"]}

Have you run into this before?

Thanks! Kelvin

daino3 commented 7 years ago

H Kelvin,

I have not. But all I need from you is the FIX message that fails so I can attempt to reproduce the issue and write a test / fix for it. Can you send please & thank you?

Is there anything special about your DD?

kmalyar-tic commented 7 years ago

@daino3 ,

The only change to the DD that I made was per this SO question which was breaking quickfix:

https://stackoverflow.com/questions/22941666/quickfix-configuration-failed-message-contains-no-fields

So removed that one line, otherwise, it is stock.

I'm working on getting you a sanitized fix message

kmalyar-tic commented 7 years ago

@daino3 (just to ping) FIX Message:

8=FIX.4.4^A9=0996^A35=8^A49=FOOO^A56=BAR^A34=1505^A128=BAZ^A115=BLAH^A52=20170620-19:22:00^A60=20170620-19:22:00^A150=F^A31=9999.11111^A151=0^A541=20171109^A32=150000000^A662=1.179999948^A423=4^A663=4^A64=20170621^A5975 =FOOBAR:20170620:1073:6^A6=9999.11111^A37=FOOBAR:20170620:1072:6^A38=150000000^A218=14.2591147256^A548=FOOBAR:20170620:1073:6^A39=2^A159=0^A669=9999.11111^A699=912796MB2^A460=6^A761=1^A223=0^A14=925000000^A15=USD^A75=20170620^A106=TREASURY BILL ^A17=FOOBAR:20170620:1072:6:12^A167=TBILL^A48=912796KX6^A198=7771:20170620:1072:6^A470=US^A381=123456789^A22=1^A54=2^A7014=21^A55=[N/A]^A118=123456789^A453=4^A448=FOOBAR^A447=D^A452=1^A448=JDR4:282205^A447=D^A452=11^A802=2^A523=27^A803=4^A523=25906^A803=4000^A448=ACME CORPORATION^A447=D^A452=13^A448=BBUNNY:13785105^A447=D^A452=36^A10=001^A

kmalyar-tic commented 7 years ago

Hiya @daino3,

Just wanted to ping you. I didn't hear anything back.
Hope all is well!

Kelvin

daino3 commented 7 years ago

Hi @kmalyar-tic.

Yes - been busy. I'll try to get around to this sometime this week. Thanks for the ping.

daino3 commented 7 years ago

@cbusbey - please label this as a bug. A PR is forthcoming.