search

Azure / durabletask

Durable Task Framework allows users to write long running persistent workflows in C# using the async/await capabilities.
Apache License 2.0
1.51k stars 292 forks source link

Parsing errors while DTFx processes the messages from queue in latest version of DurableTask.AzureStorage 2.0.0-rc.2. #1141

Open Iamjithender opened 3 months ago

Iamjithender commented 3 months ago

Hii, We started consuming the 2.0.0-rc.2 version of DurableTask.AzureStorage nuget as previous versions contain reference of the deprecated versions of "WindowsAzure.Storage".

After consuming the nuget we deployed in the ECY Canary region for testing. After deployment, we started encountering the Json parsing issues while dtfx processes the queue messages. "Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: e. Path '', line 0, position 0.".

Full exception message with stack trace: 

An error occurred while processing messages on springboot-workitems: Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: e. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.JsonReader.ReadForType(JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader)
   at DurableTask.AzureStorage.Utils.DeserializeFromJson[T](JsonSerializer serializer, String jsonString) in /_/src/DurableTask.AzureStorage/Utils.cs:line 202
   at DurableTask.AzureStorage.MessageManager.DeserializeQueueMessageAsync(QueueMessage queueMessage, String queueName, CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/MessageManager.cs:line 176
   at DurableTask.AzureStorage.Messaging.WorkItemQueue.GetMessageAsync(CancellationToken cancellationToken) in /_/src/DurableTask.AzureStorage/Messaging/WorkItemQueue.cs:line 48

We started encountering these issues after updating to the "2.0.0-rc.2", prior to that we didn't observe these issues. We observed that a periodic workflow which we run every 5 mins is not getting executed but there are few successful executions of other workflows as well.

Let me know if you need anything further to debug this issue.

saurav-microsoft commented 3 months ago

Can you also attach the message from the queue ?

davidmrdavid commented 3 months ago

There's a known parsing error in the -rc.2 package - in particular, it is unable to de-serialize messages generated by the 1.x package. Can you try upgrading to rc.3, that should be backwards-compatible.

Please note that the -rc packages are not production ready, so I cannot recommend using them in production. The GA package of the 2.x track is about to be released. I'd hold off

Iamjithender commented 3 months ago

Thanks @davidmrdavid for updating us with the details. Okay sure, we will wait for 2.x GA version package release.

@davidmrdavid Can you share the ETA for the release for 2.x version?

Iamjithender commented 3 months ago

Can you also attach the message from the queue ?

Hey Saurav, I couldn’t get the message from queue, I took jit access and tried accessing the queue but I couldn’t find any messages in queue.

saurav-microsoft commented 3 months ago

@Iamjithender I had a similar issue as well. Just wanted to take a look at your message and see if the issue is same, s If your queue message was hidden (controlled by visibility timeout config), it will become visible when the timeout happens

nytian commented 2 months ago

Hey, thanks for using the DTFx package. Just an update that the GA version of DTFx.AzureStorage v2 is published in nuget: https://www.nuget.org/packages/Microsoft.Azure.DurableTask.AzureStorage/2.0.0. Please let me know if you have any problem with the package.