Transactional Orleans with Cosmos persistence leads to deserialization exception

[slawomirpiotrowski]

Should persistence in Azure Cosmos work with transactional Orleans?

I'm getting following exception when trying to write anything using such setup:

fail: Orleans.Persistence.Cosmos.CosmosGrainStorage[0] Failure writing state for Grain Type IdentityLookup with Id main__identitylookup_~4~4email~02457 Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type Orleans.Runtime.GrainReference. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'State.PendingStates[0].TransactionManager.Reference.GrainId', line 1, position 366. at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues(JsonObjectContract contract, JsonProperty containerProperty, JsonReader reader, Type objectType) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters(JsonReader reader, JsonObjectContract contract, JsonProperty containerProperty, ObjectConstructor1 creator, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList(IList list, JsonReader reader, JsonArrayContract contract, JsonProperty containerProperty, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, Object existingValue, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonSerializer.Deserialize[T](JsonReader reader) at Microsoft.Azure.Cosmos.CosmosJsonDotNetSerializer.FromStream[T](Stream stream) at Microsoft.Azure.Cosmos.CosmosJsonSerializerWrapper.FromStream[T](Stream stream) at Microsoft.Azure.Cosmos.CosmosSerializerCore.FromStream[T](Stream stream) at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.ToObjectpublic[T](ResponseMessage responseMessage) at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.b__80[T](ResponseMessage cosmosResponseMessage) at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.ProcessMessage[T](ResponseMessage responseMessage, Func2 createResponse) at Microsoft.Azure.Cosmos.CosmosResponseFactoryCore.CreateItemResponse[T](ResponseMessage responseMessage) at Microsoft.Azure.Cosmos.ContainerCore.CreateItemAsync[T](T item, ITrace trace, Nullable1 partitionKey, ItemRequestOptions requestOptions, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.ClientContextCore.RunWithDiagnosticsHelperAsync[TResult](String containerName, String databaseName, OperationType operationType, ITrace trace, Func2 task, Func2 openTelemetry, String operationName, RequestOptions requestOptions) at Microsoft.Azure.Cosmos.ClientContextCore.OperationHelperWithRootTraceAsync[TResult](String operationName, String containerName, String databaseName, OperationType operationType, RequestOptions requestOptions, Func2 task, Func2 openTelemetry, TraceComponent traceComponent, TraceLevel traceLevel) at Orleans.Persistence.Cosmos.DefaultCosmosOperationExecutor.ExecuteOperation[TArg,TResult](Func`2 func, TArg arg) in //src/Azure/Shared/Cosmos/CosmosOptions.cs:line 153 at Orleans.Persistence.Cosmos.CosmosGrainStorage.WriteStateAsync[T](String grainType, GrainId grainId, IGrainState1 grainState) in /_/src/Azure/Orleans.Persistence.Cosmos/CosmosGrainStorage.cs:line 140

Above exception is thrown when transaction is about to be commited.

If I'm not mistaken: Cosmos storage driver is using Azure Cosmos driver and this driver is using Newtonsoft Json seralizer. The structure that is used when using Orleans in transactional mode contains Orleans.Runtime.GrainReference inside it and Newtonsoft Json deserializer doesn't know how to deserialize Orleans.Runtime.GrainReference.

[slawomirpiotrowski]

Just an update if anybody wanted to try it:

The exception is thrown if Orleans application is using Cosmos database when it writes to any grain using ITransactionalState. No other conditions has to be met.

Steps to reproduce: 1) download example bank account application https://github.com/dotnet/samples/tree/main/orleans/BankAccount 2) open it in Visual Studio 2022 (probably other environments would work too, it's just what I did to test it) 3) upgrade Orleans nuget packages to current ones (checked with 7.2.4) - have to do that as 7.0.0 didn't support Cosmos database 4) add package Microsoft.Orleans.Persistence.Cosmos to BankServer application 5) open Program.cs in BankServer application and adjust it to use cosmos database:

        siloBuilder
            .UseLocalhostClustering()
            .AddCosmosGrainStorageAsDefault(options =>
            {
                options.ConfigureCosmosClient("<connection string from Azure>");
                options.DatabaseName = "<database name>";
            })
            .UseTransactions();

6) run BankServer application 7) execute BankClient application after server has started.

It will throw on first grain state commit.

fail: Orleans.Persistence.Cosmos.CosmosGrainStorage[0]
      Failure writing state for Grain Type balance with Id default__account_Xiao
      Newtonsoft.Json.JsonSerializationException: Unable to find a constructor to use for type Orleans.Runtime.GrainReference. A class should either have a default constructor, one constructor with arguments or a constructor marked with the JsonConstructor attribute. Path 'State.PendingStates[0].TransactionManager.Reference.GrainId', line 1, position 356.
         at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject(JsonReader reader, JsonObjectContract objectContract, JsonProperty containerMember, JsonProperty containerProperty, String id, Boolean& createdFromNonDefaultCreator)

[slawomirpiotrowski]

Ok. I think I know how to fix that. But there is one pitfall and it's very serious.

Here is what happens: Cosmos driver (Microsoft.Azure.Cosmos) is using default newtonsoft serializer by default. And Orleans is using it's own serializer by default. GrainReference has attributes attached that tell Orleans serializer how to serialize and deserialize that class. There is no attributes that tell Newtonsoft serializer how to serialize and deserialize GrainReference and unfortunatelly it fails in it's default configuration.

So if Orleans were consistent about using it's own serializer everywhere by default than it would have been great. Unfortunatelly Cosmos persistence implementation in Orleans doesn't set CosmosClientOptions.Serializer property to configure it to use Orleans serializer.

It looks trivial to fix. Just set that option and problem solved (maybe allow a setting to change it by the developer to something else). Unfortunatelly... not. And here is the pitfall: This change would change format of serialized grains in cosmos database so would break grain persistence compatibility with previous Orleans versions. But it's needed to let Orleans use Cosmos persistence with Orleans transactions.

So how to fix that problem?

[ReubenBond]

We need to fix this, probably by configuring the JSON serialized which the CosmosDB provider uses with the contract resolvers in OrleansJsonSerializerSettings.

[cbgrasshopper]

Is this limited to transactions, or does it apply in any case where the default Newtonsoft serializer settings will not work?