JamesNK / Newtonsoft.Json

Json.NET is a popular high-performance JSON framework for .NET
https://www.newtonsoft.com/json
MIT License
10.77k stars 3.25k forks source link

Newtonsoft Json serializer Getting NullReferenceException when using CamelCasePropertyNamesContractResolver #2507

Open DanamoCP opened 3 years ago

DanamoCP commented 3 years ago

Source/destination types

CamelCase Serializer

Expected behavior

object being serialized

Actual behavior

Newtonsoft is occasionally getting a NullReferenceException, when trying to resolve contract.

stack trace:

ex=System.NullReferenceException: Object reference not set to an instance of an object.
 at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
 at System.Collections.Generic.Dictionary`2..ctor(IDictionary`2 dictionary, IEqualityComparer`1 comparer)
 at Newtonsoft.Json.Serialization.CamelCasePropertyNamesContractResolver.ResolveContract(Type type)
 at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.CalculatePropertyValues(JsonWriter writer, Object value, JsonContainerContract contract, JsonProperty member, JsonProperty property, JsonContract& memberContract, Object& memberValue)
 at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
 at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
 at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
 at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IEnumerable values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)
 at Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty)

Steps to reproduce

image

image

dbc2 commented 3 years ago

This issue might resolve itself it CamelCasePropertyNamesContractResolver were modified to use a ThreadSafeStore<StructMultiKey<Type, Type>, JsonContract> for contract caching instead of implementing its own thread-safe locking for a Dictionary<StructMultiKey<Type, Type>, JsonContract>? _contractCache;.

The reason I suspect this is that ThreadSafeStore<TKey, TValue>.AddValue(TKey key) has a memory barrier just before the static _store is reset:

#if !HAVE_CONCURRENT_DICTIONARY
        private TValue AddValue(TKey key)
        {
            TValue value = _creator(key);

            lock (_lock)
            {
                if (_store == null)
                {
                    _store = new Dictionary<TKey, TValue>();
                    _store[key] = value;
                }
                else
                {
                    // double check locking
                    if (_store.TryGetValue(key, out TValue checkValue))
                    {
                        return checkValue;
                    }

                    Dictionary<TKey, TValue> newStore = new Dictionary<TKey, TValue>(_store);
                    newStore[key] = value;

#if HAVE_MEMORY_BARRIER
                    Thread.MemoryBarrier();
#endif
                    _store = newStore;
                }

                return value;
            }
        }
#endif

However, CamelCasePropertyNamesContractResolver has no equivalent barrier:

        public override JsonContract ResolveContract(Type type)
        {
            if (type == null)
            {
                throw new ArgumentNullException(nameof(type));
            }

            // for backwards compadibility the CamelCasePropertyNamesContractResolver shares contracts between instances
            StructMultiKey<Type, Type> key = new StructMultiKey<Type, Type>(GetType(), type);
            Dictionary<StructMultiKey<Type, Type>, JsonContract>? cache = _contractCache;
            if (cache == null || !cache.TryGetValue(key, out JsonContract contract))
            {
                contract = CreateContract(type);

                // avoid the possibility of modifying the cache dictionary while another thread is accessing it
                lock (TypeContractCacheLock)
                {
                    cache = _contractCache;
                    Dictionary<StructMultiKey<Type, Type>, JsonContract> updatedCache = (cache != null)
                        ? new Dictionary<StructMultiKey<Type, Type>, JsonContract>(cache)
                        : new Dictionary<StructMultiKey<Type, Type>, JsonContract>();
                    updatedCache[key] = contract;

                    // No Thread.MemoryBarrier(); !!
                    _contractCache = updatedCache;
                }
            }

            return contract;
        }

I don't really understand why a memory barrier is needed when the code is already inside a lock statement -- but the inconsistency is clear. Replacing the inline locking with a ThreadSafeStore<StructMultiKey<Type, Type>, JsonContract> looks to be the simplest way to resolve the inconsistency. (This will also make use of ConcurrentDictionary<TKey, TValue> if available, which might also improve performance.)

See: this answer to Newtonsoft Json serializer Getting NullReferenceException when using CamelCasePropertyNamesContractResolver.

dbc2 commented 3 years ago

After reviewing:

I believe there may be two (and possibly a third) threading bugs in CamelCasePropertyNamesContractResolver.ResolveContract():

private static readonly object TypeContractCacheLock = new object();
private static Dictionary<StructMultiKey<Type, Type>, JsonContract>? _contractCache;

public override JsonContract ResolveContract(Type type)
{
    if (type == null)
    {
        throw new ArgumentNullException(nameof(type));
    }

    // for backwards compadibility the CamelCasePropertyNamesContractResolver shares contracts between instances
    StructMultiKey<Type, Type> key = new StructMultiKey<Type, Type>(GetType(), type);
    Dictionary<StructMultiKey<Type, Type>, JsonContract>? cache = _contractCache;
    if (cache == null || !cache.TryGetValue(key, out JsonContract contract))
    {
        contract = CreateContract(type);

        // avoid the possibility of modifying the cache dictionary while another thread is accessing it
        lock (TypeContractCacheLock)
        {
            // Bug: No checking to see whether the cache contains the contract inside the lock, which is needed in case `_contractCache` was modified by another thread between the outer check and the lock, or the value of `_contractCache` was stale.
            // Possible bug: no volatile read.
            cache = _contractCache;
            Dictionary<StructMultiKey<Type, Type>, JsonContract> updatedCache = (cache != null)
                ? new Dictionary<StructMultiKey<Type, Type>, JsonContract>(cache)
                : new Dictionary<StructMultiKey<Type, Type>, JsonContract>();
            updatedCache[key] = contract;

            // Bug: no Thread.MemoryBarrier()
            // Possible bug: no volatile write.
            _contractCache = updatedCache;
        }
    }

    return contract;
}

The possible bugs include:

Switching to a static DefaultContractResolver should resolve these issues as it uses ThreadSafeStore<TKey, TValue> internally.

(Comment updated to the current version of this answer to Newtonsoft Json serializer Getting NullReferenceException when using CamelCasePropertyNamesContractResolver.)

DanamoCP commented 3 years ago

@dbc2 It seems like the caching strategy is not being replaced in IL:


  {
    IL_0047:  ldloc.3
    IL_0048:  ldloca.s   V_4
    IL_004a:  call       void [netstandard]System.Threading.Monitor::Enter(object,
                                                                           bool&)
    IL_004f:  ldsfld     class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract> Newtonsoft.Json.Serialization.CamelCasePropertyNamesContractResolver::_contractCache
    IL_0054:  stloc.2
    IL_0055:  ldloc.2
    IL_0056:  brtrue.s   IL_005f
    IL_0058:  newobj     instance void class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract>::.ctor()
    IL_005d:  br.s       IL_0065
    IL_005f:  ldloc.2
    IL_0060:  newobj     instance void class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract>::.ctor(class [netstandard]System.Collections.Generic.IDictionary`2<!0,!1>)
    IL_0065:  dup
    IL_0066:  ldloc.1
    IL_0067:  ldloc.0
    IL_0068:  callvirt   instance void class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract>::set_Item(!0,
                                                                                                                                                                                                                          !1)
    IL_006d:  stsfld     class [netstandard]System.Collections.Generic.Dictionary`2<valuetype Newtonsoft.Json.Serialization.ResolverContractKey,class Newtonsoft.Json.Serialization.JsonContract> Newtonsoft.Json.Serialization.CamelCasePropertyNamesContractResolver::_contractCache
    IL_0072:  leave.s    IL_007f
  }  // end .try
  finally
  {
    IL_0074:  ldloc.s    V_4
    IL_0076:  brfalse.s  IL_007e
    IL_0078:  ldloc.3
    IL_0079:  call       void [netstandard]System.Threading.Monitor::Exit(object)
    IL_007e:  endfinally
  }  // end handler

Is it possible that during runtime optimization is occurring and changing the order of those lines?
DanamoCP commented 3 years ago

Also, does using a DefaultContractResolver with CamelCaseNamingStrategy should work the same as CamelCasePropertyNamesContractResolver?

elgonzo commented 3 years ago

@DanamoCP

looking at the C# compiler output is the wrong place to look for signs of possible instruction reordering. It is not the C# compiler doing a possible reordering that is the sole concern here, but it's either reordering done by the JIT compiler or the instruction pipeline of the CPU itself reordering instructions while it fetches and processes them...

dbc2 commented 3 years ago

@DanamoCP

Also, does using a DefaultContractResolver with CamelCaseNamingStrategy should work the same as CamelCasePropertyNamesContractResolver?

Other than global caching, CamelCasePropertyNamesContractResolver works exactly the same as new CamelCaseNamingStrategy { ProcessDictionaryKeys = true, OverrideSpecifiedNames = true }. You can verify this from the reference source.