Dictionary is incorrectly deserialized on .NET 4.7.1 due to invalid CultureAwareComparer state.

[maksim0305]

Hello. I have faced with fatal bug that breaks our application. There is the following class and appropriate object:

[Serializable]
public class SomeClass
{
private Dictionary<string, object> someDictionary = new Dictionary<string, object>(StringComparer.InvariantCultureIgnoreCase);
}

For instance, someDictionary field contains "A", "B", "C" keys. I serialize the object on machine with .NET 4.6.2. Then I deserialize it on machine with .NET 4.7.1. After the deserialization I can add "A", "B", "C" values again! As result, the dictionary contains 6 keys: "A", "B", "C", "A", "B", "C". Looks like it relates to the following issues: 1) https://github.com/Microsoft/dotnet/issues/621 2) https://github.com/Microsoft/dotnet/blob/master/releases/net471/KnownIssues/521954%20-%20BCL%20CultureAwareComparer%20with%20ignore%20casing%20on%20serialized%20on%20previous%20versions%20of%20.NET%20do%20not%20correctly%20deserialize%20on%20.NET%204.7.1.md

I guess the problem is that during the deserializing StringComparer is case sensitive, but after deserializing it is case insesitive. As result, the string comparer returns different hashcodes and allow to insert duplicate keys. Is it known issue? And if so, when should it be fixed?

[AlexGhiondea]

Hi @maksim0305.

This is a known issue that we are working on releasing a fix for it. This has been reported here as well: https://github.com/Microsoft/dotnet/issues/679

Could you tell me more about your scenario and the impact it has?

Thanks, Alex

[maksim0305]

Hi Alex. Briefly: 1) The application deserialize the dictionary with that "A", "B", "C" data. 2) Then it inserts the data into the dictionary again as it can't find them (due to the changed comparer state). At this step we have "A", "B", "C", "A", "B", "C" dictionary. 3) Then it serializes this new "A", "B", "C", "A", "B", "C" dictionary.

Result: we can't deserialize that data anymore, as it fails with "the same key exists" exception.

To stabilize the application we need to remove the serialized data.

[AlexGhiondea]

Thanks for the additional information @maksim0305.

We are looking at releasing an update to address this in an upcoming servicing release for .NET.

I will post a reply here once the patch is available.

[maksim0305]

Thanks Alex. Is the patch date known?

[maksim0305]

Hi @AlexGhiondea. Could you please provide approximate timescale of the servicing release? We need to know it as we are going to decide either to wait for the fix or to make some workarounds in the code. Best regards.

[AlexGhiondea]

Hi @maksim0305. We are currently planning to ship a patch for this in our next servicing releases. Our next window for servicing is in May and we are working towards that. I will update this issue with the KB numbers once they have shipped.

[MattD-Hubble]

I’ve had a go at reproducing this problem, and after adding a call to my dictionary's OnDeserialization method in my serialised class’s OnDeserialized handler I see the behaviour @maksim0305 describes when deserialising on 4.7.1. When I try on a machine with .NET Framework 4.7.2 installed however it seems to be behaving correctly. Both machines are running Windows 7 with all of the current updates (including the 2018-05 Security and Quality Rollup for .NET Framework - KB4099633).

[MattD-Hubble]

PS The fixed behaviour I described above seems to apply to Window 7 but not Windows 10. Both are fully up to date, but Windows 10 with the April 2018 update and Framework 4.7.2 still has the broken behaviour.

[goprime]

As a non-developer I'm having a bit of a hard time following the conversation but it appears that as of May 21, the issue related to the link below is still present in 4.7.1 and 4.7.2. We have an application that only works corrrectly on Windows 10 1709 with the fix provided in the link below. First question is, has this fix been integrated into any cumulative release on 1709 such that we don't have to manually apply the patch? Second question is, what are the options for resolution in Windows 10 1803 (as some users now have)?

https://github.com/Microsoft/dotnet/blob/master/releases/net471/KnownIssues/521954%20-%20BCL%20CultureAwareComparer%20with%20ignore%20casing%20on%20serialized%20on%20previous%20versions%20of%20.NET%20do%20not%20correctly%20deserialize%20on%20.NET%204.7.1.md