Errors in rabbitmq+jasmin when sending sms via http

[cawabang]

Hello people. I have an instance of jasminsms OS Ubuntu 20.04.2 Jasmin is installed from source Rabbit version 3.8.2 Erlang 22.2.7 Recently it started to hung. We are using smpp and http messages. i believe it hanging bcz of http. Got error only in rabbitmq.log, nothing in messages.log.

2022-08-18 11:30:17.017 [error] <0.23885.68> Generic server <0.23885.68> terminating Last message in was {'$gen_cast',{method,{'basic.publish',0,<<"messaging">>,<<"dlr.deliver_sm">>,false,false},{content,60,none,<<32,128,0,0,0,216,4,116,121,112,101,83,0,0,0,10,100,101,108,105,118,101,114,95,115,109,3,99,105,100,83,0,0,0,8,98,101,101,108,105,110,101,50,9,100,108,114,95,100,108,118,114,100,83,0,0,0,3,48,48,49,7,100,108,114,95,115,117,98,83,0,0,0,3,48,48,49,9,100,108,114,95,115,100,97,116,101,83,0,0,0,10,50,50,48,56,49,56,49,49,51,48,9,100,108,114,95,100,100,97,116,101,83,0,0,0,10,50,50,48,56,49,56,49,49,51,48,7,100,108,114,95,101,114,114,83,0,0,0,3,48,48,48,8,100,108,114,95,116,101,120,116,83,0,0,0,12,5,0,3,239,191,189,3,3,4,52,0,46,0,52,6,100,108,114,95,105,100,83,0,0,0,18,50,50,57,54,56,52,55,54,51,52,56,50,49,51,57,53,52,54,8,100,108,114,95,115,116,97,116,83,0,0,0,7,82,69,74,69,67,84,68,18,50,50,57,54,56,52,55,54,51,52,56,50,49,51,57,53,52,54>>,rabbit_framing_amqp_0_9_1,[<<"REJECTD">>]},flow}} When Server state == {ch,{conf,running,rabbit_framing_amqp_0_9_1,1,<0.23879.68>,<0.23883.68>,<0.23879.68>,<<"127.0.0.1:38812 -> 127.0.0.1:5672">>,undefined,{user,<<"guest">>,[administrator],[{rabbit_auth_backend_internal,none}]},<<"/">>,<<"submit.sm.beeline2">>,<0.23880.68>,[],none,0,134217728,undefined,#{}},{lstate,<0.23884.68>,false},none,847193,{7,{[{847192,<<"SMPPClientFactory-megafon">>,1660811416887,{<0.879.0>,238716}},{847191,<<"SMPPClientFactory-megafon">>,1660811416756,{<0.879.0>,238715}},{847190,<<"SMPPClientFactory-megafon">>,1660811416501,{<0.879.0>,238713}},{847189,<<"SMPPClientFactory-megafon">>,1660811416501,{<0.879.0>,238714}},{847188,<<"SMPPClientFactory-megafon">>,1660811416501,{<0.879.0>,238710}}],[{847186,<<"SMPPClientFactory-megafon">>,1660811416501,{<0.879.0>,238712}},{847187,<<"SMPPClientFactory-megafon">>,1660811416501,{<0.879.0>,238711}}]}},#{<0.838.0> => {resource,<<"/">>,queue,<<"DLRLookup-dlrlookupd-01">>},<0.853.0> => {resource,<<"/">>,queue,<<"RouterPB_deliver_sm_all">>},<0.859.0> => {resource,<<"/">>,queue,<<"RouterPB_bill_request_submit_sm_resp_all">>},<0.864.0> => {resource,<<"/">>,queue,<<"deliver_sm_thrower">>},<0.874.0> => {resource,<<"/">>,queue,<<"submit.sm.mts">>},<0.879.0> => {resource,<<"/">>,queue,<<"submit.sm.megafon">>},<0.884.0> => {resource,<<"/">>,queue,<<"submit.sm.beeline2">>},<0.24421.2> => {resource,<<"/">>,queue,<<"mes2">>}},{state,#{<0.838.0> => #Ref<0.1793802155.2786066433.82084>,<0.853.0> => #Ref<0.1793802155.2786066433.81762>,...},...},...} Reason for termination == ** {function_clause,[{rabbit_binary_parser,parse_table,[<<0,52,6,100,108,114,95,105,100,83,0,0,0,18,50,50,57,54,56,52,55,54,51,52,56,50,49,51,57,53,52,54,8,100,108,114,95,115,116,97,116,83,0,0,0,7,82,69,74,69,67,84,68>>],[{file,"src/rabbit_binary_parser.erl"},{line,50}]},{rabbit_binary_parser,parse_table,1,[{file,"src/rabbit_binary_parser.erl"},{line,52}]},{rabbit_binary_parser,parse_table,1,[{file,"src/rabbit_binary_parser.erl"},{line,52}]},{rabbit_framing_amqp_0_9_1,decode_properties,2,[{file,"src/rabbit_framing_amqp_0_9_1.erl"},{line,943}]},{rabbit_binary_parser,ensure_content_decoded,1,[{file,"src/rabbit_binary_parser.erl"},{line,155}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1324}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,632}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]}]} 2022-08-18 11:30:17.017 [error] <0.23879.68> Error on AMQP connection <0.23879.68> (127.0.0.1:38812 -> 127.0.0.1:5672, vhost: '/', user: 'guest', state: running), channel 1: {function_clause, [{rabbit_binary_parser,parse_table, [<<0,52,6,100,108,114,95,105,100,83,0,0,0,18,50,50,57,54,56,52,55,54, 51,52,56,50,49,51,57,53,52,54,8,100,108,114,95,115,116,97,116,83, 0,0,0,7,82,69,74,69,67,84,68>>], [{file,"src/rabbit_binary_parser.erl"},{line,50}]}, {rabbit_binary_parser,parse_table,1, [{file,"src/rabbit_binary_parser.erl"},{line,52}]}, {rabbit_binary_parser,parse_table,1, [{file,"src/rabbit_binary_parser.erl"},{line,52}]}, {rabbit_framing_amqp_0_9_1,decode_properties,2, [{file,"src/rabbit_framing_amqp_0_9_1.erl"},{line,943}]}, {rabbit_binary_parser,ensure_content_decoded,1, [{file,"src/rabbit_binary_parser.erl"},{line,155}]}, {rabbit_channel,handle_method,3, [{file,"src/rabbit_channel.erl"},{line,1324}]}, {rabbit_channel,handle_cast,2, [{file,"src/rabbit_channel.erl"},{line,632}]}, {gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]}]}

Can you tell me at least where to look?

[farirat]

Is this related to a specific sms content/params ? or is it general to any message you are trying to deliver ? mt or mo ? are you using specific http args ?

[cawabang]

Is this related to a specific sms content/params ? or is it general to any message you are trying to deliver ? mt or mo ? are you using specific http args ?

Its MT messages and it is bulk(50 of them) sms from another program through jasmin via http. Text is the same in all of them, but for some reason they are hanging jasmin. All of the msg are stored in rabbit, and after i restart jasmin e.g. "systemctl restart jasmind" it continues to send sms from rabbit. SMS test is in hex content, since its cyrillic Example from http-access.log: POST /send?hex-content=042304320430043604300435043C044B043900200432043B043004340435043B043504460020043E0440044304360438044F00210020041E0412041E0020043F043E0020041F043504420440043E04330440043004340441043A043E043C044300200440002D043D0443002004400435043A043E043C0435043D043404430435044200200443044104420430043D043E043204380442044C0020044104380433043D0430043B04380437043004460438044E002004320020041204300448043504390020043A0432043004400442043804400435002E002004220435043B002E0032003400360034003200370037002E041C043E043D044704350433043E04400441043A0430044F002C00200434002E0034&coding=8&to=79500265338&password=PWD&username=USER

[zielkam]

Hello cawabang, maybe its the same issue as that described in ticket "Coding crash amqp #1066". In my case i`ve gathered network communication between jasmin and broker and it confirmed that issue is in DLR text content. Wireshark shaw that some publish amqp message were broken and amq server close connection

[cawabang]

So, change the SMS text or add &dlr=no? But i thought that dlr is by default = no, as it stated in jasmin documenation.

[zielkam]

cawabang, youre right,default value of dlr is no, i think dlr is not yours problem - in my case it was. Maybe network trace from communication between gateway and broker will help. In my case tcpdump showed which message (in which queue) was disrupted causing broker crash. Ive found that, gathering data with tcpdump command on broker port (default 5672) and analyze it in wireshark looking fom amqp connection close message. Just before connection close there was Publish message with binary content which was causing crash.

[cawabang]

Changed sms text and it worked fine for 2 times in a row. But now ive got the same error in rabbit, but a new one in messages.log in jasmin: (<class 'txamqp.client.ConnectionClosed'>) while handling submit_sm_resp pdu for msgid:4517dac4-2128-4460-9567-30e72f6a0778: Method(name=close, id=50) (541, 'INTERNAL_ERROR', 0, 0) content = None

In rabbitmq error is the same, numbers are different: exception exit: {{function_clause,[{rabbit_binary_parser,parse_table,[<<4,56,6,100,108,114,95,105,100,83,0,0,0,18,51,50,56,55,54,51,57,53,53,50,56,53,49,55,55,57,50,52,8,100,108,114,95,115,116,97,116,83,0,0,0,7,82,69,74,69,67,84,68>>],[{file,"src/rabbit_binary_parser.erl"},{line,50}]},{rabbit_binary_parser,parse_table,1,[{file,"src/rabbit_binary_parser.erl"},{line,52}]},{rabbit_binary_parser,parse_table,1,[{file,"src/rabbit_binary_parser.erl"},{line,52}]},{rabbit_framing_amqp_0_9_1,decode_properties,2,[{file,"src/rabbit_framing_amqp_0_9_1.erl"},{line,943}]},{rabbit_binary_parser,ensure_content_decoded,1,[{file,"src/rabbit_binary_parser.erl"},{line,155}]},{rabbit_channel,handle_method,3,[{file,"src/rabbit_channel.erl"},{line,1324}]},{rabbit_channel,handle_cast,2,[{file,"src/rabbit_channel.erl"},{line,632}]},{gen_server2,handle_msg,2,[{file,"src/gen_server2.erl"},{line,1067}]}]},[{gen_server2,terminate,3,[{file,"src/gen_server2.erl"},{line,1183}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,249}]}]} ancestors: [<0.16029.93>,<0.16028.93>,<0.16025.93>,<0.16024.93>,<0.448.0>,<0.447.0>,<0.446.0>,rabbit_sup,<0.306.0>]