ghost
commented
2 years ago Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.
Issue Details
The `HttpHeaders` collections were never thread-safe. Accessing a header may force lazy parsing of its value, resulting in modifications to the underlying data structures. Before .NET 6.0, reading from the collection concurrently happened to be thread-safe _in most cases_. If the collection contained an invalid header value, that value would be removed during enumeration. In rare cases, this could cause problems from modifying the underlying Dictionary concurrently. Starting with 6.0, less locking is performed around header parsing as it was no longer needed internally (#54130). Due to this change, multiple examples of users accessing the headers concurrently surfaced. Currently known ones to the team are gRPC's .NET client (#55898, #55896) and NewRelic's HttpClient instrumentation (https://github.com/newrelic/newrelic-dotnet-agent/issues/803). Violating thread safety in 6.0 may result in the header values being duplicated/malformed or various exceptions being thrown during enumeration/header accesses. I am posting a few known call stacks below to help other users hitting this issue discover the root cause:
IndexOutOfRangeException
``` System.IndexOutOfRangeException: Index was outside the bounds of the array. at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringArray(HeaderDescriptor descriptor, HeaderStoreItemInfo info) at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext() ```NullReferenceException
``` System.NullReferenceException: Object reference not set to an instance of an object. at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringArray(HeaderDescriptor descriptor, HeaderStoreItemInfo info) at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext() ```InvalidCastException
``` InvalidCastException: Unable to cast object of type 'System.Collections.Generic.List`1[System.Object]' to type 'System.Net.Http.Headers.MediaTypeHeaderValue'. at System.Net.Http.Headers.HttpContentHeaders.get_ContentType() ```There is precedent in concurrent reads being thread-safe on some collections (namely Dictionary) and users may assume HttpHeaders behaves the same.
A related issue here is that forcing the validation of an invalid value will result in its removal from the collection. These Schrödinger's headers are difficult to reason about as their existence depends on whether/how/when they are observed.
We can look to make the behavior more intuitive. We should investigate whether we can (efficiently) implement the collection in a way that satisfies the following:
- concurrent reads are thread-safe and return the same result to all readers
- concurrent write/modification/remove operations are not supported and their behavior continues to be undefined
- a "validating read" of an invalid value does not result in the removal of the invalid value. The invalid value will still be visible to the
NonValidatedview of the collection
Should enumerating the collection (with or without validation) while a concurrent write operation is in progress have defined behavior?
- We can choose to document that concurrent reads are supported, but not while a write operation is in progress. This matches
Dictionary's thread-safety model - Concurrent reads are expected to occur over the response headers where write operations are uncommon
| Author: | MihaZupan |
|---|---|
| Assignees: | - |
| Labels: | `area-System.Net.Http` |
| Milestone: | 7.0.0 |
amin-lu-xero
MihaZupan
Strepto
karelz
toddca
a-jackson
The
HttpHeaderscollections were never thread-safe. Accessing a header may force lazy parsing of its value, resulting in modifications to the underlying data structures.Before .NET 6.0, reading from the collection concurrently happened to be thread-safe in most cases. If the collection contained an invalid header value, that value would be removed during enumeration. In rare cases, this could cause problems from modifying the underlying Dictionary concurrently.
Starting with 6.0, less locking is performed around header parsing as it was no longer needed internally (#54130). Due to this change, multiple examples of users accessing the headers concurrently surfaced. Currently known ones to the team are gRPC's .NET client (#55898, #55896) and NewRelic's HttpClient instrumentation (https://github.com/newrelic/newrelic-dotnet-agent/issues/803). Violating thread safety in 6.0 may result in the header values being duplicated/malformed or various exceptions being thrown during enumeration/header accesses.
I am posting a few known call stacks below to help other users hitting this issue discover the root cause:
IndexOutOfRangeException
``` System.IndexOutOfRangeException: Index was outside the bounds of the array. at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringArray(HeaderDescriptor descriptor, HeaderStoreItemInfo info) at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext() ```NullReferenceException
``` System.NullReferenceException: Object reference not set to an instance of an object. at System.Net.Http.Headers.HttpHeaders.ReadStoreValues[T](Span`1 values, Object storeValue, HttpHeaderParser parser, Int32& currentIndex) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringOrStringArray(HeaderDescriptor descriptor, Object sourceValues, String& singleValue, String[]& multiValue) at System.Net.Http.Headers.HttpHeaders.GetStoreValuesAsStringArray(HeaderDescriptor descriptor, HeaderStoreItemInfo info) at System.Net.Http.Headers.HttpHeaders.GetEnumeratorCore()+MoveNext() ```InvalidCastException
``` InvalidCastException: Unable to cast object of type 'System.Collections.Generic.List`1[System.Object]' to type 'System.Net.Http.Headers.MediaTypeHeaderValue'. at System.Net.Http.Headers.HttpContentHeaders.get_ContentType() ```There is precedent in concurrent reads being thread-safe on some collections (namely
Dictionary) and users may assumeHttpHeadersbehaves the same.A related issue here is that forcing the validation of an invalid value will result in its removal from the collection. These Schrödinger's headers are difficult to reason about as their existence depends on whether/how/when they are observed.
We can look to make the behavior more intuitive. We should investigate whether we can (efficiently) implement the collection in a way that satisfies the following:
NonValidatedview of the collectionShould enumerating the collection (with or without validation) while a concurrent write operation is in progress have defined behavior?
Dictionary's thread-safety model