Not a valid type: “x” with byte array element in Headers

[ghost]

I am trying to fetch messages from queue using ruby amqp gems and getting error.

I have one byte array in header (i.e. actual record) which is must from development prospective and type cannot be changed. I have tried

qParser.pop do | payload| and consumer2.consume.on_delivery do |metadata, payload|

but same error in both issues. I found amqp protocol -> table raise this exception. The byte array type (header) should be supported as rabittMQ supports it.

Here is the error

ArgumentError: Not a valid type: "x"
Data: "\x00\x00\x064\fChannel-TypeS\x00\x00\x00\x03PHN\aApi-KeyS\x00\x00\x00\x03RTM\x0EContent-Lengthl\x00\x00\x00\x00\x00\x00\x01l\rActual-Recordx\x00\x00\x04{{\"custRoleTypeCd\":\"\",\"msgDlvryProcTs\":\"11/06/2015 05:16:22\",\"chnlTypeCd\":\"TMSG\",\"dlvryId\":\"0003713221\",\"mktgInd\":\"N\",\"vendrId\":\"11\",\"cntctPntAdrTxt\":\"3022000023\",\"sorId\":\"7\",\"elctrMsgFrmtDesc\":\"SMS\",\"msgCustId\":\"1000000165146\",\"msgLangCd\":\"\",\"quietPerdTznCd\":\"\",\"prodDesc\":\"MASTERCARD PLATINUM\",\"rplyToAdr\":\"\",\"cycleDt\":\"20151014\",\"last4DgtsTxt\":\"3599\",\"thldValTxt\":\"\",\"substitutionKeyValueData\":{\"AlertCode\":\"PAYPST\",\"CustomerFirstName\":\"Crysta\",\"VARBL_5_FCL_TXT\":\"298,35\",\"PaymentPostedDate\":\"OCT 14, 2015\",\"PaymentAmount\":\"298.35\",\"VARBL_9_ENG_TXT\":\"lloA7T%2FXblrp1Bc7aoIK0A%3D%3D\",\"productdescription\":\"MASTERCARD PLATINUM\",\"AccountEndingIn\":\"3599\",\"VARBL_10_FCL_TXT\":\"14 octobre 2015\",\"VARBL_10_ENG_TXT\":\"XqLuxBTABbIDvpw56ba34hF0imQ5McLi4fDdnotsiaY%3D\"},\"alrtTypeCd\":\"PAYPST\",\"quietPerdEndTm\":\"\",\"srcSysUnqId\":\"\",\"acctId\":\"10000371367\",\"evtRsltTypeDesc\":\"The system is preparing a message for delivery.\",\"tmplId\":\"10001\",\"loadTs\":\"11/06/2015 05:16:22\",\"fromAdr\":\"\",\"msgLobCd\":\"CARD\",\"wirlsPlanId\":\"\",\"quietPerdStrtTm\":\"\",\"elctrAdrTypeCd\":\"PHN\",\"subjLine\":\"\",\"oldDlvryId\":\"\",\"evtRsltReasId\":\"6020\",\"recProcId\":\"N\",\"sorCustId\":\"850784\"}\vcontentTypeS\x00\x00\x00\ntext/plain\nSet-CookieS\x00\x00\x00bJSESSIONID=91EED106D2925239197C35CF201727F4; Path=/int-dms-content-assembly-web/; Secure; HttpOnly\x06ServerS\x00\x00\x00\nCapitalOne\rCache-ControlS\x00\x00\x00.no-cache, no-store, max-age=0, must-revalidate\x06PragmaS\x00\x00\x00\bno-cache\x04Datel\x00\x00\x01Q\xA1\xF5\xB80\x13originalContentTypeS\x00\x00\x00\x10application/json\x0Fhttp_statusCodeS\x00\x00\x00\x03200\fContent-TypeS\x00\x00\x00\x10application/json"
Unprocessed data: "\x00\x00\x04{{\"custRoleTypeCd\":\"\",\"msgDlvryProcTs\":\"11/06/2015 05:16:22\",\"chnlTypeCd\":\"TMSG\",\"dlvryId\":\"400000000003713221\",\"mktgInd\":\"N\",\"vendrId\":\"11\",\"cntctPntAdrTxt\":\"3022000023\",\"sorId\":\"7\",\"elctrMsgFrmtDesc\":\"SMS\",\"msgCustId\":\"1000000165146\",\"msgLangCd\":\"\",\"quietPerdTznCd\":\"\",\"prodDesc\":\"MASTERCARD PLATINUM\",\"rplyToAdr\":\"\",\"cycleDt\":\"20151014\",\"last4DgtsTxt\":\"3599\",\"thldValTxt\":\"\",\"substitutionKeyValueData\":{\"AlertCode\":\"PAYPST\",\"CustomerFirstName\":\"Crysta\",\"VARBL_5_FCL_TXT\":\"298,35\",\"PaymentPostedDate\":\"OCT 14, 2015\",\"PaymentAmount\":\"298.35\",\"VARBL_9_ENG_TXT\":\"lloA7T%2FXblrp1Bc7aoIK0A%3D%3D\",\"productdescription\":\"MASTERCARD PLATINUM\",\"AccountEndingIn\":\"3599\",\"VARBL_10_FCL_TXT\":\"14 octobre 2015\",\"VARBL_10_ENG_TXT\":\"XqLuxBTABbIDvpw56ba34hF0imQ5McLi4fDdnotsiaY%3D\"},\"alrtTypeCd\":\"PAYPST\",\"quietPerdEndTm\":\"\",\"srcSysUnqId\":\"\",\"acctId\":\"10000371367\",\"evtRsltTypeDesc\":\"The system is preparing a message for delivery.\",\"tmplId\":\"10001\",\"loadTs\":\"11/06/2015 05:16:22\",\"fromAdr\":\"\",\"msgLobCd\":\"CARD\",\"wirlsPlanId\":\"\",\"quietPerdStrtTm\":\"\",\"elctrAdrTypeCd\":\"PHN\",\"subjLine\":\"\",\"oldDlvryId\":\"\",\"evtRsltReasId\":\"6020\",\"recProcId\":\"N\",\"sorCustId\":\"850784\"}\vcontentTypeS\x00\x00\x00\ntext/plain\nSet-CookieS\x00\x00\x00bJSESSIONID=91EED106D2925239197C35CF201727F4; Path=/int-dms-content-assembly-web/; Secure; HttpOnly\x06ServerS\x00\x00\x00\ne\rCache-ControlS\x00\x00\x00.no-cache, no-store, max-age=0, must-revalidate\x06PragmaS\x00\x00\x00\bno-cache\x04Datel\x00\x00\x01Q\xA1\xF5\xB80\x13originalContentTypeS\x00\x00\x00\x10application/json\x0Fhttp_statusCodeS\x00\x00\x00\x03200\fContent-TypeS\x00\x00\x00\x10application/json"
Offset: 80
Total size: 1588
Processed data: {"Channel-Type"=>"PHN", "Api-Key"=>"RTM", "Content-Length"=>364}

[michaelklishin]

First of all, there are no byte arrays in Ruby. Rather, Ruby values are strings that are passed around as byte arrays. So that part is supported in the most straightforward way possible.

You are not providing any code examples but the issue can be value-specific.

Please post a code sample we can run to reproduce.

[michaelklishin]

Oh, and knowing what exact version of both gems you use would be useful, too.

[michaelklishin]

A quick glance at value encoder, value decoder, and table type constants suggests that the payload comes from a different client. amq-protocol indeed does not use TYPE_BYTE_ARRAY for serialisation.

So what client is that?

[ghost]

Here is the code:

MoreInfo.xlsx

Attached file has more info..

Versions:

amq-protocol (2.0.0, 1.9.2) amqp (1.5.1)

I have attached a sample message from my queue...

[ghost]

The above code snippet worked fine with simple queue which did not have Actual-Record element. However with actual record header is processed till channel type and then raise exception...

i.e.

Api-Key:    RTM
Channel-Type:   EML
Actual-Record:  {"msgDlvryProcTs":"11/06/2015 05:16:22","custRoleTypeCd":"","chnlTypeCd":"","dlvryId":"03713247","mktgInd":"N","vendrId":"1","sorId":"7","cntctPntAdrTxt":"keejaykumar@gmail.com","elctrMsgFrmtDesc":"TEXT","msgCustId":"1000000165146","msgLangCd":"","quietPerdTznCd":"","prodDesc":"MASTERCARD PLATINUM","cycleDt":"20151014","rplyToAdr":"","last4DgtsTxt":"3599","thldValTxt":"","substitutionKeyValueData":{"AlertCode":"PAYPST","CustomerFirstName":"Crysta","VARBL_5_FCL_TXT":"298,35","PaymentPostedDate":"OCT 14, 2015","PaymentAmount":"298.35","VARBL_9_ENG_TXT":"lloA7T%2FXblrp1Bc7aoIK0A%3D%3D","productdescription":"MASTERCARD PLATINUM","AccountEndingIn":"3599","VARBL_10_FCL_TXT":"14 octobre 2015","VARBL_10_ENG_TXT":"XqLuxBTABbIDvpw56ba34hF0imQ5McLi4fDdnotsiaY%3D"},"alrtTypeCd":"PAYPST","quietPerdEndTm":"","srcSysUnqId":"","acctId":"10000371367","evtRsltTypeDesc":"The system is preparing a message for delivery.","tmplId":"10001","loadTs":"11/30/2015 05:16:22","fromAdr":"","wirlsPlanId":"","msgLobCd":"CARD","quietPerdStrtTm":"","elctrAdrTypeCd":"EML","subjLine":"","oldDlvryId":"","evtRsltReasId":"6020","recProcId":"N","sorCustId":"850784"}
contentType:    text/plain
Content-Type:   application/json

[michaelklishin]

@mohitjagga this is the code you consume with. What code publishes messages?

[ghost]

@michaelklishin that code is springXD code. Developed by the Java developers and I am consuming the streams created by their modules by creating taps on streams.

[michaelklishin]

@mohitjagga alright. Last question: what do you expect to come out on the Ruby end where Java publishers use byte arrays? Ruby doesn't have byte arrays.

[ghost]

@michaelklishin Correct me if I am wrong... You are saying there is no concept of byte array in Ruby and Ruby will not handle it.

However I am using queue.pop(payload)... I want some way I can fetch payload without even bother about headers. What is happening => it inspect headers and throws exception. Is it possible to fetch payload in this scenario where we have byte array in header... as of now all amqp methods fails with same error and I am stuck with as even though I do not need headers...

[michaelklishin]

There is no data structure for byte and byte array the same way they are in Java. Byte arrays can be turned into strings in Ruby land. The question is what most Ruby users would expect.

You can't consume a message without consuming (and parsing) its headers.

[ghost]

Thanks a lot for your help... I will see if I can use Java for this

[michaelklishin]

Hold on. There is a missing deserialisation feature in this library: most people never hit it because many popular programming languages don't operate byte buffers, and thus don't pass them around, or at least not in headers.

I gather the best way to treat byte buffers in Ruby would be to turn them into strings in binary encoding.

[ghost]

I am sure Ruby users will need some other way to handle byte array. Converting into string will increase the size of header. We are using byte array as size of actual-record header is big and string cannot handle that.

[ghost]

@michaelklishin that would be awesome :-)

[michaelklishin]

Most Ruby users wouldn't even say with confidence what a byte array is. I'm going to add byte array deserialisation that produces strings (much like message bodies are strings, even though they really are byte streams, and byte[]s in Java, which is quite close).

[garyrussell]

I guess the only issue is if the ruby app publishes a reply, echoing the headers, the Java app would see the echoed header as a String instead of a byte[].

But I see there's not really much that can be done about that.

[michaelklishin]

@mohitjagga please try with amq-protocol master. While you are at it, give Bunny a try, since amqp gem use is highly discouraged for new projects these days.

[ghost]

@michaelklishin It worked like a charm. We are doing the same in out app.. byte array is just for transfer data and then we convert it to string.. byte array helps in transporting heavy data which string cant :-).. Thanks a lot for your help!

PS: Any idea when this fix will be available in rubygems.org?

[michaelklishin]

It already is.