Interoperability with Microsoft.WindowsAzure.Storage.Queue

[christophwille]

Is your feature request related to a problem? Please describe. We are migrating an existing application from WindowsAzure.Storage.Queue to Azure.Storage.Queues. We will have existing messages in our queues - and this is the problem. The old client writes base64-encoded messages, the new client doesn't. Thus we have two differently encoded messages and we cannot discern them (Azure Storage Explorer does show that one is base64 encoded, the other as not).

Describe the solution you'd like An ability to get the encoding of the QueueMessage we are retrieving.

Describe alternatives you've considered base64 encoding the messages we send with the new client, or detecting based on the string itself whether it is base64 or not.

Additional context Old code of sending the message:

string modelAsJson = JsonSerializer.Serialize(model, _options);
var queueMessage = new CloudQueueMessage(modelAsJson);
await _queue.AddMessageAsync(queueMessage).ConfigureAwait(false);

New code for sending:

string modelAsJson = JsonSerializer.Serialize(model, _options);
await _queue.SendMessageAsync(modelAsJson).ConfigureAwait(false);

The old code auto-encodes base64, the new one is all UTF8.

Information Checklist Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• [X] Description Added
• [X] Expected solution specified

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @xgithubtriage

[AlexGhiondea]

/cc @tg-msft

[tg-msft]

The encoding isn't a service level concept we can provide info about and the old libraries were often doing it unnecessarily. We decided to drop that overhead in the Azure.Storage.Queues client library, but this will definitely be a bit of pain in the short term if you're manipulating queues with both versions.

I think the easiest workaround will be to check if Convert.TryFromBase64String returns true (or Convert.FromBase64String doesn't throw) for QueueMessage.MessageText.

[christophwille]

The FromBase64String approach will throw every time once only new, non-encoded messages are in the queue. So one exception per message isn't exactly a good thing to have.

Our ad-hoc approach was to base64-encode before posting with the new client so all messages (old and new ones) are encoded. But as this is inefficient, we will look into TryFromBase64String.

[ghost]

Thanks for working with Microsoft on GitHub! Tell us how you feel about your experience using the reactions on this comment.

[zmarty]

@paulbatum We have this problem in Azure Functions where we send queue items using the new library but the receiving Azure Function uses the old library and assumes base64 encoding. We worked around it by encoding in base64 before sending with the new library.

If Azure Functions would implement workaround https://github.com/Azure/azure-sdk-for-net/issues/8949#issuecomment-561420123 then we would not need to do that. Or, even better, have a way to indicate to the specific Azure Function to assume queue item is string and not base64 string?

[wjchristenson2]

We also ran into this migration issue. Our architecture has queue calls running through a QueueHelper. We were able to "auto-detect" if the message was base64 encoded and decode if so. Our function returns an IEnumerable of QueueMessage which presented a new problem. The QueueMessage object does not have a setter for the MessageText property. As a result, we used reflection to set the .MessageText property to our UTF8 string. Hardly optimal but once queues flush out the old base64 messages, we'll remove the workaround.

public async Task<IEnumerable<QueueMessage>> GetMessages(QueueClient queueClient, int maxMessages, TimeSpan? visibilityTimeout = null)
{
    QueueMessage[] messages = await queueClient.ReceiveMessagesAsync(maxMessages, visibilityTimeout).ConfigureAwait(false);

    if (messages != null && messages.Length > 0)
    {
        for (int i = 0; i <= messages.Length - 1; i++)
        {
            string messageText = messages[i].MessageText;
            if (Regex.IsMatch(messageText, "^([A-Za-z0-9+/]{4})*([A-Za-z0-9+/]{3}=|[A-Za-z0-9+/]{2}==)?$"))
            {
                try
                {
                    byte[] messageData = System.Convert.FromBase64String(messageText);
                    messages[i].SetPrivatePropertyValue("MessageText", System.Text.UTF8Encoding.UTF8.GetString(messageData));
                }
                catch (Exception)
                {
                }
            }
        }
    }

    return (messages ?? new QueueMessage[] { });
}

[kasobol-msft]

This has been addressed in https://github.com/Azure/azure-sdk-for-net/pull/16689 along with base64 option and available in https://www.nuget.org/packages/Azure.Storage.Queues/12.5.0