DeadlineExceeded exception when trying to create persistent subscription to $all with ES Cloud 21.10.4

[cbgrasshopper]

Describe the bug Using .NET gRPC client v 23.0.0 and creating a persistent subscription to $all. When I call CreateToAllAsync when connecting to Event Store Cloud v 22.10.0.0, single node cluster, everything works as expected. When I run the exact same code connecting to Event Store Cloud v 21.10.4, three-node cluster, I always get a DeadlineExceeded error, even with an extremely long deadline. I've tried both with esdb+discover and with a direct connection to the leader node; no luck. Cloud provider is Microsoft Azure.

To Reproduce Steps to reproduce the behavior:

1. Create a single node Event Store Cloud cluster, version 21.10.4 and a three-node cluster, version 22.10.0.0
2. Run the following client code against each cluster:

    var settings = new PersistentSubscriptionSettings(startFrom:Position.Start, resolveLinkTos: true, readBatchSize:1000);
    await _subscriptionClient.CreateToAllAsync("subscription-group", settings);

3. Observe that the single-node cluster call succeeds and the one to the three-node cluster fails. (I've also tried removing the filter but had the same problem.)

Expected behavior The three-node cluster v 21.10.4 should properly create a persistent subscription as the 22.10 cluster does; no DeadlineExceeded error in the client.

Actual behavior The following exception occurs:

   at EventStore.Client.Interceptors.TypedExceptionInterceptor.<AsyncUnaryCall>b__5_0[TRequest,TResponse](Task`1 t)
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at EventStore.Client.EventStorePersistentSubscriptionsClient.CreateInternalAsync(String streamName, String groupName, IEventFilter eventFilter, PersistentSubscriptionSettings settings, Nullable`1 deadline, UserCredentials userCredentials, CancellationToken cancellationToken)
   at EventStore.Client.EventStorePersistentSubscriptionsClient.CreateToAllAsync(String groupName, IEventFilter eventFilter, PersistentSubscriptionSettings settings, Nullable`1 deadline, UserCredentials userCredentials, CancellationToken cancellationToken)

Config/Logs/Screenshots

EventStore details

• EventStore server version: 21.10.4 and 22.10.0.0

• Operating system: Managed by Event Store Cloud

• EventStore client version (if applicable): .NET gRPC 23.0.0 (also tried 22.0.0)

Additional context

[cbgrasshopper]

After email exchanges with Event Store support, it turns out that the issue was having a readBatchSize greater than the buffer size; configuring the buffer size to be greater than the readBatchSize when creating the persistent subscription works. So the true unexpected behavior is more about how the error is reported back to the client.