search

Azure / azure-sdk-for-js

This repository is for active development of the Azure SDK for JavaScript (NodeJS & Browser). For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/javascript/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-js.
MIT License
2.03k stars 1.19k forks source link

Buffer from Service-Bus adds special characters if you convert it to a string #30474

Closed RobbinKok closed 1 month ago

RobbinKok commented 1 month ago

Describe the bug When the dataset is too big, the service bus sends a Buffer to the client. When translating that buffer to a string, a lot of special characters (like ›☺☻→☻o) are added to the message, even though I know for sure that they aren't there when we send te message to the queue. With smaller messages that are the simular, it just sends the message with converting it into a buffer first and in there the data doesnt contain the special characters. I tried to conversion from a buffer to a string using utf-8, utf8, win-1252 and all inserted different special characters.

To Reproduce Steps to reproduce the behavior:

  1. Send a large dataset through the Azure Service bus (A 20k lines json would work for this)
  2. Receive the buffer on your nodejs application
  3. Decode the buffer (Can be done thorugh Buffer.from() or using the iconv-lite package
  4. Print the buffer to see the results

Expected behavior The expected behavior would be that the data stays the same when you decode the buffer.

Screenshots If applicable, add screenshots to help explain your problem. image image image

Additional context Add any other context about the problem here.

github-actions[bot] commented 1 month ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @EldertGrootenboer.

jeremymeng commented 1 month ago

@RobbinKok thanks for reporting the issue. Our Service Bus package provides customers a convenience of attempt to decode the message body as JSON object. This may not be desired. I don't know for sure whether your issue is related yet, but one quick thing to try is use the skipParsingBodyAsJson: true, option when creating the receiver. Could you please give it a try to see if there's any difference?

RobbinKok commented 1 month ago

The option skipParsingBodyAsJson: true, didn't change anything in the message. This is part of the message with it set to true: image

This is the same part of the message with it turned to false/ turned off: image

For context, this is how the message (a string) looks like when we send it to the service bus: image

jeremymeng commented 1 month ago

@RobbinKok thank you for testing the option! We don't apply other transformations when receiving messages so this puzzled me. How was your large dataset sent to Azure Service Bus? via JS SDK client, SDK in another programming language, or some other tools?

When the dataset is too big, the service bus sends a Buffer to the client.

Does this mean that sometimes your code receives a Buffer, and sometimes it receives a non-Buffer when the dataset is not too big?

jeremymeng commented 1 month ago

I just tried receiving a JSON of 20000 lines (size > 256 KB) but didn't have any issue. @RobbinKok If you could share more details, if possible a randomly generated JSON that demonstrates the issue would be great!

RobbinKok commented 1 month ago

From the looks of it our sending mechanism (other system and a java application) uses some sort of compression mechanism. I've asked the team that works on it to clarify what exactly happens with a message before its send and what compression it uses. Looks like something goes wrong with the compression, but I'll update you when I know it for sure

RobbinKok commented 1 month ago

Got a response from the backend team and they changed the compression algorithm without notifying. Closing this issue now, since it has nothing to do with the service bus