crash reports on ... with 0 neighbours crashed with reason: no match of right hand value

[emmd123]

Hi guys!

I'm getting lots of crash reports on the production vernemq. I looked it up and saw an existing closed issue from 2016. But I'm currently using 1.5 which shouldnt have the issue. Just want to check if I'm . missing something.

Thanks!

Existing closed issued based on log: https://github.com/vernemq/vernemq/issues/244 (2016)

VerneMQ Version: 1.5 OS: Ubuntu 16.04.5 LTS Erlang: Erlang/OTP 20 Conf: mostly default except for ip and cookies for cluster Cluster size/standalone: 6 nodes in cluster

Expected behavior

No errors in log.

Actual behaviour

CRASH REPORT Process with 0 neighbours crashed with reason: no match of right hand value <<0,22 Ranch listener {{0,0,0,0},1883} terminated with reason: {badmatch,

[larshesel]

Hi, could you please show us the exact logs you see as well as describe what is going on in the system at the time? Is there a way for us to reproduce this? Can you reproduce it on VerneMQ 1.6.2?

[emmd123]

Hi!

This is from the error.log. I'm not sure how to reproduce it but crash is happening every hour now and only in production. I'm not sure if we can easily upgrade to 1.6.2 at this time. =(

Thanks for the help!

2019-01-24 15:09:30.812 [error] <0.25676.47> CRASH REPORT Process <0.25676.47> with 0 neighbours crashed with reason: no match of right hand value <<0,22,117,115,101,114,115,47,49,48,53,50,56,48,48,47,108,105,115,116,105,110,103,115,1,0,123,34,98,117,121,34,58,123,34,105,100,34,58,50,57,53,50,49,49,52,44,34,105,109,103,34,58,34,50,57,53,50,49,49,52,47,112,47,49,53,51,52,56,54,48,50,57,53,46,50,55,51,57,50,46,106,112,103,34,44,34,117,115,101,114,110,97,109,101,34,58,34,66,106,102,114,105,99,107,55,49,34,125,44,34,99,34,58,49,52,57,57,57,44,34,99,111,109,34,58,49,48,44,34,104,34,58,102,97,108,115,101,44,34,105,100,34,58,49,50,51,53,49,48,...>> in vmq_parser:parse/4 line 68 2019-01-24 15:09:30.812 [error] <0.25676.47> Ranch listener {{0,0,0,0},1883} terminated with reason: {badmatch,<<0,22,117,115,101,114,115,47,49,48,53,50,56,48,48,47,108,105,115,116,105,110,103,115,1,0,123,34,98,117,121,34,58,123,34,105,100,34,58,50,57,53,50,49,49,52,44,34,105,109,103,34,58,34,50,57,53,50,49,49,52,47,112,47,49,53,51,52,56,54,48,50,57,53,46,50,55,51,57,50,46,106,112,103,34,44,34,117,115,101,114,110,97,109,101,34,58,34,66,106,102,114,105,99,107,55,49,34,125,44,34,99,34,58,49,52,57,57,57,44,34,99,111,109,34,58,49,48,44,34,104,34,58,102,97,108,115,101,44,34,105,100,34,58,49,50,...>>}

[larshesel]

Are you perhaps using a custom plugin using an old version of the parser as in this issue: https://github.com/vernemq/vernemq/issues/895#issuecomment-430229330

Also note the discussion leading to this conclusion in the issue here: https://github.com/vernemq/vernemq/issues/895#issuecomment-430130301

AFAICT VerneMQ 1.5.0 should not be able to produce a crash as you mention due to the guards mentioned in the comment above.

[emmd123]

[{vmq_parser,parse,4,[{file,".../_build/default/lib/vmq_commons/src/vmq_parser.erl"},{line,68}]},{vmq_mqtt_fsm,data_in,3,[{file,"/opt/vernemq/distdir/1.5.0/_build/default/lib/vmq_server/src/vmq_mqtt_fsm.erl"},{line,138}]},{vmq_ranch,handle_message,2,[{file,"/opt/vernemq/distdir/1.5.0/_build/default/lib/vmq_server/src/vmq_ranch.erl"},{line,174}]},{vmqranch,loop,1,[{file,"/opt/vernemq/distdir/1.5.0/_build/default/lib/vmq_server/src/vmq_ranch.erl"},{line,117}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}

I am using a plugin. I checked the rebar.config and its using the one from master. {vmq_commons, ".*", {git, "git://github.com/erlio/vmq_commons.git", {branch, "master"}}},

also github.com/marianoguerra/jwt-erl (2016) but the error doesnt seem to indicate its from that repo.

[larshesel]

ok, so you have the same problem: you're using a plugin which pulls in vmq_commons which is obsolete and contains a buggy parser which overrides the one which is shipped with VerneMQ - you'd have to remove vmq_commons as a dependency of (all the plugins) and it should work - depending on the plugins you may have to do a bit of migration work.

Hope this helps

[emmd123]

thanks so much! will try to remove the dependency. =) Cheers!