RaiseEvent throws `Invalid length parameter passed to the LEFT or SUBSTRING function` when input is serialized json string

[jundayin]

In some of our business flow, we want to pass in an object through the external event. Since the interface only accept string, we serialize the input and deserialize it once it is picked up by the workflow.

However, there is some chance that the SQL Provider throws SqlException of Invalid length parameter passed to the LEFT or SUBSTRING function. during raising the event. I tried removing the heading and trailing space from the serialized string but the problem may randomly occurs

Workaround that I've been done:

1. Disable the format during serialization

DTF Sql Provider version: 1.0.0-rc DTFx Version: 2.8.0

[cgillum]

@jundayin can you provide the call stack trace for this error? I'd like to confirm where exactly the error is coming from.

[jundayin]

Yes, I will provide it later today

[jundayin]

Microsoft.Data.SqlClient.SqlException:
   at Microsoft.Data.SqlClient.SqlConnection.OnError (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.TdsParser.TryRun (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.InternalEndExecuteNonQuery (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryInternal (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at Microsoft.Data.SqlClient.SqlCommand.EndExecuteNonQueryAsync (Microsoft.Data.SqlClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.SqlServer.SqlUtils+<WithRetry>d__36`1.MoveNext (DurableTask.SqlServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=2ea3c3a96309d850: /_/src/DurableTask.SqlServer/SqlUtils.cs:500)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.SqlServer.SqlUtils+<WithRetry>d__36`1.MoveNext (DurableTask.SqlServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=2ea3c3a96309d850: /_/src/DurableTask.SqlServer/SqlUtils.cs:507)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.SqlServer.SqlUtils+<ExecuteSprocAndTraceAsync>d__32`1.MoveNext (DurableTask.SqlServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=2ea3c3a96309d850: /_/src/DurableTask.SqlServer/SqlUtils.cs:449)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.SqlServer.SqlOrchestrationService+<SendTaskOrchestrationMessageAsync>d__27.MoveNext (DurableTask.SqlServer, Version=1.0.0.0, Culture=neutral, PublicKeyToken=2ea3c3a96309d850: /_/src/DurableTask.SqlServer/SqlOrchestrationService.cs:440)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at DurableTask.Core.TaskHubClient+<RaiseEventAsync>d__30.MoveNext (DurableTask.Core, Version=2.8.0.0, Culture=neutral, PublicKeyToken=d53979610a6e89dd: /_/src/DurableTask.Core/TaskHubClient.cs:696)

I truncated the stack traces from our business code. Let me know if that is also needed. @cgillum

[cgillum]

Thanks @jundayin. I think I found the code responsible for this error:

INSERT INTO Instances (
    [TaskHub],
    [InstanceID],
    [ExecutionID],
    [Name],
    [Version],
    [RuntimeStatus])
SELECT DISTINCT
    @TaskHub,
    E.[InstanceID],
    NEWID(),
    SUBSTRING(E.[InstanceID], 2, CHARINDEX('@', E.[InstanceID], 2) - 2),
    '',
    'Pending'
FROM @NewOrchestrationEvents E
WHERE NOT EXISTS (
    SELECT 1
    FROM Instances I
    WHERE [TaskHub] = @TaskHub AND I.[InstanceID] = E.[InstanceID])
GROUP BY E.[InstanceID]
ORDER BY E.[InstanceID] ASC

I think this error can occur when you try to raise an event to an orchestration instance that doesn't exist. Could that explain the problem you're running into?

[jundayin]

We use TaskCompletionSource to make the orchestration waits for the signal of external event

In some case, we are sending input within the external event and that is when the exception is thrown

[cgillum]

@jundayin if you call TaskHubClient.RaiseEventAsync(...) and specify an instance ID for an orchestration that hasn't yet been created using TaskHubClient.CreateOrchestrationInstanceAsync(...), then I expect you'll run into this problem. The problem is not related to your orchestration code - it's only a problem with your usage of TaskHubClient.

[jundayin]

I hardly believe it is about non-existing orchestration instance because we get the instance ID through context.OrchestrationInstance.InstanceID inside one TaskActivity. My understanding is when we retrieve the instance ID from the orchestration instance, it indicates the orchestration is existing and running thus when RaiseEvent can find the particular orchestration instance from db. Correct me if I'm wrong.

Does the RaiseEvent even filter orchestration instance by RuntimeStatus?

[cgillum]

So you're saying that the orchestration raises an external event to itself from one of its activities?

If you examine the SQL code above, that SUBSTRING(...) function (which I believe is the one returning the error) only gets invoked when the specified instance ID doesn't exist in the Instances table. In this case E.[InstanceID] represents the instance ID of the external event message - i.e., the one specified in TaskHubClient.RaiseEventAsync(...).

[jundayin]

Let me try to create a minimal repo from my side.

[lucianaparaschivei]

even if we raise an event for an orchestration that is not created yet, or was deleted, it would be helpful to get a more suggestive error

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[cgillum]

That’s fair. We can definitely improve the error message.