Open DanamoCP opened 3 years ago
This issue might resolve itself it CamelCasePropertyNamesContractResolver
were modified to use a ThreadSafeStore<StructMultiKey<Type, Type>, JsonContract>
for contract caching instead of implementing its own thread-safe locking for a Dictionary<StructMultiKey<Type, Type>, JsonContract>? _contractCache;
.
The reason I suspect this is that ThreadSafeStore<TKey, TValue>.AddValue(TKey key)
has a memory barrier just before the static _store
is reset:
#if !HAVE_CONCURRENT_DICTIONARY
private TValue AddValue(TKey key)
{
TValue value = _creator(key);
lock (_lock)
{
if (_store == null)
{
_store = new Dictionary<TKey, TValue>();
_store[key] = value;
}
else
{
// double check locking
if (_store.TryGetValue(key, out TValue checkValue))
{
return checkValue;
}
Dictionary<TKey, TValue> newStore = new Dictionary<TKey, TValue>(_store);
newStore[key] = value;
#if HAVE_MEMORY_BARRIER
Thread.MemoryBarrier();
#endif
_store = newStore;
}
return value;
}
}
#endif
However, CamelCasePropertyNamesContractResolver
has no equivalent barrier:
public override JsonContract ResolveContract(Type type)
{
if (type == null)
{
throw new ArgumentNullException(nameof(type));
}
// for backwards compadibility the CamelCasePropertyNamesContractResolver shares contracts between instances
StructMultiKey<Type, Type> key = new StructMultiKey<Type, Type>(GetType(), type);
Dictionary<StructMultiKey<Type, Type>, JsonContract>? cache = _contractCache;
if (cache == null || !cache.TryGetValue(key, out JsonContract contract))
{
contract = CreateContract(type);
// avoid the possibility of modifying the cache dictionary while another thread is accessing it
lock (TypeContractCacheLock)
{
cache = _contractCache;
Dictionary<StructMultiKey<Type, Type>, JsonContract> updatedCache = (cache != null)
? new Dictionary<StructMultiKey<Type, Type>, JsonContract>(cache)
: new Dictionary<StructMultiKey<Type, Type>, JsonContract>();
updatedCache[key] = contract;
// No Thread.MemoryBarrier(); !!
_contractCache = updatedCache;
}
}
return contract;
}
I don't really understand why a memory barrier is needed when the code is already inside a lock
statement -- but the inconsistency is clear. Replacing the inline locking with a ThreadSafeStore<StructMultiKey<Type, Type>, JsonContract>
looks to be the simplest way to resolve the inconsistency. (This will also make use of ConcurrentDictionary<TKey, TValue>
if available, which might also improve performance.)
See: this answer to Newtonsoft Json serializer Getting NullReferenceException when using CamelCasePropertyNamesContractResolver.
After reviewing:
I believe there may be two (and possibly a third) threading bugs in CamelCasePropertyNamesContractResolver.ResolveContract()
:
private static readonly object TypeContractCacheLock = new object();
private static Dictionary<StructMultiKey<Type, Type>, JsonContract>? _contractCache;
public override JsonContract ResolveContract(Type type)
{
if (type == null)
{
throw new ArgumentNullException(nameof(type));
}
// for backwards compadibility the CamelCasePropertyNamesContractResolver shares contracts between instances
StructMultiKey<Type, Type> key = new StructMultiKey<Type, Type>(GetType(), type);
Dictionary<StructMultiKey<Type, Type>, JsonContract>? cache = _contractCache;
if (cache == null || !cache.TryGetValue(key, out JsonContract contract))
{
contract = CreateContract(type);
// avoid the possibility of modifying the cache dictionary while another thread is accessing it
lock (TypeContractCacheLock)
{
// Bug: No checking to see whether the cache contains the contract inside the lock, which is needed in case `_contractCache` was modified by another thread between the outer check and the lock, or the value of `_contractCache` was stale.
// Possible bug: no volatile read.
cache = _contractCache;
Dictionary<StructMultiKey<Type, Type>, JsonContract> updatedCache = (cache != null)
? new Dictionary<StructMultiKey<Type, Type>, JsonContract>(cache)
: new Dictionary<StructMultiKey<Type, Type>, JsonContract>();
updatedCache[key] = contract;
// Bug: no Thread.MemoryBarrier()
// Possible bug: no volatile write.
_contractCache = updatedCache;
}
}
return contract;
}
The possible bugs include:
There is no Thread.MemoryBarrier()
before _contractCache
is reset. This allows the compiler to reorder the code as follows:
_contractCache = updatedCache;
updatedCache[key] = contract;
Which would allow unlocked reader threads to access the updatedCache
as it is being modified, possibly causing an exception.
Newtonsoft's ThreadSafeStore<TKey, TValue>
does this correctly.
There is no double check locking to see whether the required contract was added to the cache inside the lock.
Once again ThreadSafeStore<TKey, TValue>
does this correctly.
Possibly _contractCache
needs to be volatile
.
I am not sure this is necessary since it is only ever mutated from inside the lock
statement. It may be sufficient to add double check locking as mentioned above.
However, ThreadSafeStore<TKey, TValue>
does not use volatile
for its internal cache which suggests this is not necessary.
Switching to a static DefaultContractResolver
should resolve these issues as it uses ThreadSafeStore<TKey, TValue>
internally.
(Comment updated to the current version of this answer to Newtonsoft Json serializer Getting NullReferenceException when using CamelCasePropertyNamesContractResolver.)
@dbc2 It seems like the caching strategy is not being replaced in IL:
{
IL_0047: ldloc.3
IL_0048: ldloca.s V_4
IL_004a: call void [netstandard]System.Threading.Monitor::Enter(object,
bool&)
IL_004f: ldsfld class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract> Newtonsoft.Json.Serialization.CamelCasePropertyNamesContractResolver::_contractCache
IL_0054: stloc.2
IL_0055: ldloc.2
IL_0056: brtrue.s IL_005f
IL_0058: newobj instance void class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract>::.ctor()
IL_005d: br.s IL_0065
IL_005f: ldloc.2
IL_0060: newobj instance void class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract>::.ctor(class [netstandard]System.Collections.Generic.IDictionary`2<!0,!1>)
IL_0065: dup
IL_0066: ldloc.1
IL_0067: ldloc.0
IL_0068: callvirt instance void class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract>::set_Item(!0,
!1)
IL_006d: stsfld class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract> Newtonsoft.Json.Serialization.CamelCasePropertyNamesContractResolver::_contractCache
IL_0072: leave.s IL_007f
} // end .try
finally
{
IL_0074: ldloc.s V_4
IL_0076: brfalse.s IL_007e
IL_0078: ldloc.3
IL_0079: call void [netstandard]System.Threading.Monitor::Exit(object)
IL_007e: endfinally
} // end handler
Is it possible that during runtime optimization is occurring and changing the order of those lines?
Also, does using a DefaultContractResolver with CamelCaseNamingStrategy should work the same as CamelCasePropertyNamesContractResolver?
@DanamoCP
looking at the C# compiler output is the wrong place to look for signs of possible instruction reordering. It is not the C# compiler doing a possible reordering that is the sole concern here, but it's either reordering done by the JIT compiler or the instruction pipeline of the CPU itself reordering instructions while it fetches and processes them...
@DanamoCP
Also, does using a DefaultContractResolver with CamelCaseNamingStrategy should work the same as CamelCasePropertyNamesContractResolver?
Other than global caching, CamelCasePropertyNamesContractResolver
works exactly the same as new CamelCaseNamingStrategy { ProcessDictionaryKeys = true, OverrideSpecifiedNames = true }
. You can verify this from the reference source.
Source/destination types
CamelCase Serializer
Expected behavior
object being serialized
Actual behavior
Newtonsoft is occasionally getting a NullReferenceException, when trying to resolve contract.
stack trace:
Steps to reproduce