Closed vrecluse closed 2 years ago
This sounds like something related to JSON.NET. I would guess that your dictionary is being concurrently modified and so there's a data race somewhere, but I'm not sure.
Yes, I think @ReubenBond is correct here.
I run into this problem again, and found that maybe it's caused by Json.Net, which is not thread-safe when reuse the same serializer:
https://github.com/JamesNK/Newtonsoft.Json/issues/1452
In this driver's JsonGrainStateSerializer, it reuses JsonSerializer, and was registered as Singleton:
public class JsonGrainStateSerializer : IGrainStateSerializer
{
private readonly JsonSerializer serializer;
public JsonGrainStateSerializer(ITypeResolver typeResolver, IGrainFactory grainFactory, MongoDBGrainStorageOptions options)
{
var jsonSettings = OrleansJsonSerializer.GetDefaultSerializerSettings(typeResolver, grainFactory);
options?.ConfigureJsonSerializerSettings?.Invoke(jsonSettings);
this.serializer = JsonSerializer.Create(jsonSettings);
if (options?.ConfigureJsonSerializerSettings == null)
{
//// https://github.com/OrleansContrib/Orleans.Providers.MongoDB/issues/44
//// Always include the default value, so that the deserialization process can overwrite default
//// values that are not equal to the system defaults.
this.serializer.NullValueHandling = NullValueHandling.Include;
this.serializer.DefaultValueHandling = DefaultValueHandling.Populate;
}
}
public void Deserialize(IGrainState grainState, JObject entityData)
{
var jsonReader = new JTokenReader(entityData);
serializer.Populate(jsonReader, grainState.State);
}
public JObject Serialize(IGrainState grainState)
{
return JObject.FromObject(grainState.State, serializer);
}
}
But in OrleansJsonSerializer.cs, the implemetation of Serialize method, calls JsonConverter.SerializeObject, which creates a new JsonSerializer everytime, avoids thread safety issue:
public static string SerializeObject(object value, Type type, JsonSerializerSettings settings)
{
JsonSerializer jsonSerializer = JsonSerializer.CreateDefault(settings);
return JsonConvert.SerializeObjectInternal(value, type, jsonSerializer);
}
Can you provide a PR for that?
Please note that if you are using a version lower than Orleans 7.0, you should exercise caution when using the default JsonGrainStateSerializer
. Specifically, if your grain state contains a large list, you may encounter exceptions such as 'Size {} is larger than 16m'. This is due to the fact that the serializer uses the ObjectCreationHandling.Auto
setting, which can result in the list being reused and duplicate items being created during deserialize via the serializer.Populate(jsonReader, grainState.State)
method.
To fix this, just add the following config when call AddMongoDBGrainStorage
:
.AddMongoDBGrainStorageAsDefault(optionsBuilder =>
{
optionsBuilder.Configure(op =>
{
op.ConfigureJsonSerializerSettings = jsonSettings =>
{
jsonSettings.NullValueHandling = NullValueHandling.Include;
jsonSettings.DefaultValueHandling = DefaultValueHandling.Populate;
jsonSettings.ObjectCreationHandling = ObjectCreationHandling.Replace;
};
//...
});
})
I'm using Orleans.Providers.MongoDB as grain storage, but ran into this exception.
Is it caused by circular reference or duplicated reference? But my state definition is fairly simple: