ProtocolException with iOS .NET 6.0 Application: The number of bytes available is inconsistent with the HTTP Content-Length header.

[chamons]

From @DuncWatts on Mon, 05 Sep 2022 12:20:03 GMT

I'm porting some code over from an existing Xamarin.Forms application however the WCF client I've written fails when run as a .NET 6.0 iOS application.

As WCF connected services don't work within iOS due to the restrictions on emitted code, I use a reference generated by the Silverlight tooling as a workaround and add System.ServiceModel.Http to use it.

Steps to Reproduce

I've found a random soap service to demonstrate this (http://www.dneonline.com/calculator.asmx?WSDL) and created a .NET Standard 2.0 class library that contains the WCF client and the code that calls it. I've then created a simple .NET 6.0 iOS application and a Xamarin.iOS application that executes that code that calls the webservice. In the Xamarin.iOS application it can correctly call the webservice while the .NET 6.0 iOS application fails with the stack trace below.

Expected Behavior

Web service is called correctly

Actual Behavior

Exception thrown when calling web service:

[0:] System.ServiceModel.ProtocolException: The number of bytes available is inconsistent with the HTTP Content-Length header.  There may have been a network error or the client may be sending invalid requests.
   at System.ServiceModel.Channels.HttpResponseMessageHelper.ReadBufferedMessageAsync(Task`1 inputStreamTask, TimeoutHelper timeoutHelper)
   at System.ServiceModel.Channels.HttpResponseMessageHelper.ParseIncomingResponse(TimeoutHelper timeoutHelper)
   at System.ServiceModel.Channels.HttpChannelFactory`1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.<ReceiveReplyAsync>d__17[[System.ServiceModel.Channels.IRequestChannel, System.Private.ServiceModel, Version=4.10.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.ServiceModel.Channels.RequestChannel.RequestAsync(Message message, TimeSpan timeout)
   at System.ServiceModel.Channels.RequestChannel.RequestAsyncInternal(Message message, TimeSpan timeout)
   at System.Runtime.AsyncResult.End[SendAsyncResult](IAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
   at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
   at System.ServiceModel.ClientBase`1.ChannelBase`1[[WcfExample.CalculatorSoap, WcfExample, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null],[WcfExample.CalculatorSoap, WcfExample, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null]].EndInvoke(String methodName, Object[] args, IAsyncResult result)
   at WcfExample.CalculatorSoapClient.CalculatorSoapClientChannel.EndAdd(IAsyncResult result) in C:\Users\me\source\repos\WcfExample\WcfExample\CalculatorService.cs:line 594
   at WcfExample.CalculatorSoapClient.WcfExample.CalculatorSoap.EndAdd(IAsyncResult result) in C:\Users\me\source\repos\WcfExample\WcfExample\CalculatorService.cs:line 258
   at System.Threading.Tasks.TaskFactory`1[[System.Int32, System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].FromAsyncCoreLogic(IAsyncResult , Func`2 , Action`1 , Task`1 , Boolean )
--- End of stack trace from previous location ---
   at WcfExample.Example.AddAsync(Int32 a, Int32 b) in C:\Users\me\source\repos\WcfExample\WcfExample\Example.cs:line 14
   at WcfExample.iOS.Net6.AppDelegate.<>c.<<FinishedLaunching>b__4_0>d.MoveNext() in C:\Users\me\source\repos\WcfExample\WcfExample.iOS.Net6\AppDelegate.cs:line 29

Environment

Version information

``` Microsoft Visual Studio Enterprise 2022 Version 17.4.0 Preview 1.0 VisualStudio.17.Preview/17.4.0-pre.1.0+32804.182 Microsoft .NET Framework Version 4.8.04084 Installed Version: Enterprise Visual C++ 2022 00476-80000-00000-AA112 Microsoft Visual C++ 2022 ADL Tools Service Provider 1.0 This package contains services used by Data Lake tools ASA Service Provider 1.0 ASP.NET and Web Tools 17.4.86.38009 ASP.NET and Web Tools Azure App Service Tools v3.0.0 17.4.86.38009 Azure App Service Tools v3.0.0 Azure Data Lake Tools for Visual Studio 2.6.5000.0 Microsoft Azure Data Lake Tools for Visual Studio Azure Functions and Web Jobs Tools 17.4.86.38009 Azure Functions and Web Jobs Tools Azure Stream Analytics Tools for Visual Studio 2.6.5000.0 Microsoft Azure Stream Analytics Tools for Visual Studio C# Tools 4.4.0-1.22403.13+ebbf56c257fb4d3128d3487ef525d92e9f94b412 C# components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used. Common Azure Tools 1.10 Provides common services for use by Azure Mobile Services and Microsoft Azure Tools. Extensibility Message Bus 1.4.1 (main@2ee106a) Provides common messaging-based MEF services for loosely coupled Visual Studio extension components communication and integration. Microsoft Azure Hive Query Language Service 2.6.5000.0 Language service for Hive query Microsoft Azure Stream Analytics Language Service 2.6.5000.0 Language service for Azure Stream Analytics Microsoft Azure Tools for Visual Studio 2.9 Support for Azure Cloud Services projects Microsoft JVM Debugger 1.0 Provides support for connecting the Visual Studio debugger to JDWP compatible Java Virtual Machines Mono Debugging for Visual Studio 17.4.0 (d412e49) Support for debugging Mono processes with Visual Studio. NuGet Package Manager 6.4.0 NuGet Package Manager in Visual Studio. For more information about NuGet, visit https://docs.nuget.org/ Project System Tools 1.0 Tools for working with C#, VisualBasic, and F# projects. Razor (ASP.NET Core) 17.0.0.2237602+b10895cdddd2286fe448ca44dfeb6f9c2c4f6abd Provides languages services for ASP.NET Core Razor. SQL Server Data Tools 17.0.62207.28050 Microsoft SQL Server Data Tools ToolWindowHostedEditor 1.0 Hosting json editor into a tool window TypeScript Tools 17.0.10727.2001 TypeScript Tools for Microsoft Visual Studio Visual Basic Tools 4.4.0-1.22403.13+ebbf56c257fb4d3128d3487ef525d92e9f94b412 Visual Basic components used in the IDE. Depending on your project type and settings, a different version of the compiler may be used. Visual F# Tools 17.4.0-beta.22376.4+0e8a2053cc2fc7d3801205a873e0afd3b6a1235e Microsoft Visual F# Tools Visual Studio IntelliCode 2.2 AI-assisted development for Visual Studio. VisualStudio.DeviceLog 1.0 Information about my package VisualStudio.Mac 1.0 Mac Extension for Visual Studio VSPackage Extension 1.0 VSPackage Visual Studio Extension Detailed Info Xamarin 17.4.0.95 (main@de68f64) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android. Xamarin Designer 17.4.0.32 (remotes/origin/d17-4@439b92ed7b) Visual Studio extension to enable Xamarin Designer tools in Visual Studio. Xamarin Templates 17.4.1 (0b2b400) Templates for building iOS, Android, and Windows apps with Xamarin and Xamarin.Forms. Xamarin.Android SDK 13.0.99.36 (main/b4998c8) Xamarin.Android Reference Assemblies and MSBuild support. Mono: dffa5ab Java.Interop: xamarin/java.interop/main@032f1e71 SQLite: xamarin/sqlite/3.39.2@40e8743 Xamarin.Android Tools: xamarin/xamarin-android-tools/main@9c641b3 Xamarin.iOS and Xamarin.Mac SDK 15.13.0.6 (af0b05917) Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support. ```

Build Logs

msbuild.zip

Example Project (If Possible)

WcfExample.zip

Copied from original issue xamarin/xamarin-macios#15866

[chamons]

From @chamons on Tue, 06 Sep 2022 15:07:56 GMT

Given all of the assemblies in the stack trace are BCL (Base Class Library), in particular System.ServiceModel.*, I believe this should be looked at by the dotnet folks first.

[dotnet-issue-labeler[bot]]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[danmoseley]

I think this belongs in dotnet/wcf

[DuncWatts]

Note this works fine in a .NET 6 WPF application, it's only the .NET 6 iOS build that has this issue so far. I've yet to try it on Android or MacOS.

[danmoseley]

cc @radical @steveisok since this may be iOS specific and therefore not a WCF issue - perhaps they have seen something similar.

[chamons]

Given the lack of any Xamarin iOS types in the stack is the reason I bounced it over to the runtime at first.

[chamons]

I just checked, and this works fine on macOS Desktop with NET6.

[chamons]

This appears to be the location of the exception.

[steveisok]

@simonrozsival Please give this a look and see if you can find anything.

[simonrozsival]

I managed to reproduce the issue locally. When the exception that @chamons pointed out is throw, the state is:

• bytesRead == 0
• _contentLength == 348
• offset == 325
• count == 23
• buffer contains 325 bytes corresponding to a UTF-8 encoded string <?xml version="1.0" encoding="utf-8"?><soap:Envelope ...</soap:Envelope>

The raw HTTP response is:

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Content-Encoding: gzip
Vary: Accept-Encoding
Server: Microsoft-IIS/10.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
X-Powered-By-Plesk: PleskWin
Date: Thu, 08 Sep 2022 07:57:57 GMT
Content-Length: 348

<?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><AddResponse xmlns="http://tempuri.org/"><AddResult>3</AddResult></AddResponse></soap:Body></soap:Envelope>

As far as I can tell the actual size of the body is 325 bytes. I think the ProtocolException is the correct behavior in this case because the server responded with incorrect Content-Length (even though it is not consistent with legacy Xamarin which seems to ignore this mismatch). The code in WCF that checks the mismatch is 7 years old and the ReadAsync code in NSUrlSessionHandler+NSUrlSessionDataTaskStream hasn't changed in 4 years so this doesn't seem like a recent regression.

@chamons can you share the macOS desktop project so I can try to check why that behaves differently?

[DuncWatts]

@simonrozsival I've added a WPF .NET 6 application to my example that isn't throwing a ProtocolException if that's of any help.

WcfExampleWithWPF.zip

[DuncWatts]

I've run Fiddler against the WPF build and in this instance it's returning Content-Length: 325 so assume it must be something in how it's making the request.

Request headers:

POST /calculator.asmx HTTP/1.1
Host: www.dneonline.com
Cache-Control: no-cache, max-age=0
SOAPAction: "http://tempuri.org/Add"
Accept-Encoding: gzip, deflate
Content-Type: text/xml; charset=utf-8
Content-Length: 161

Response headers:

HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: text/xml; charset=utf-8
Vary: Accept-Encoding
Server: Microsoft-IIS/10.0
X-AspNet-Version: 2.0.50727
X-Powered-By: ASP.NET
X-Powered-By-Plesk: PleskWin
Date: Thu, 08 Sep 2022 09:50:39 GMT
Content-Length: 325

[simonrozsival]

@DuncWatts That's interesting. I ran the request a couple more times and I got three different Content-Length values for the same request (identical headers to yours) with a body length of 325 bytes: 262, 325, 348. It seems there's something wrong with that server.

Anyway, I found a workaround that might help. If you can, add this property to your iOS project: <UseNativeHttpHandler>false</UseNativeHttpHandler>. If you depend on the behavior of NSUrlSessionHandler somewhere, you can will need to instantiate it yourself and pass it to HttpClient explicitly through its constructor. This may not be suitable for all projects though.

The SocketsHttpHandler, that is used when the native one is disabled, behaves differently: the response's content.Headers.ContentLength doesn't have a value (even though the response contains it) so the ReadChunkedBufferedMessageAsync method is called instead of ReadBufferedMessageAsync. The ReadChunkedBufferedMessageAsync method ignores the Content-Length header and reads the whole response body.

https://github.com/dotnet/wcf/blob/acb1157835acda59bde51cb1e64f1340619379e7/src/System.Private.ServiceModel/src/System/ServiceModel/Channels/HttpResponseMessageHelper.cs#L178-L183

I'm not sure if this is the desired behavior of SocketsHttpHandler or not, but it's definitely an inconsistency between the platforms. There were some improvements to the Content-Length behavior recently (for example https://github.com/dotnet/runtime/pull/62541) so I'll try it with the latest preview .NET 7 later as well.

EDIT: So I ran the example in a console app with the latest changes to the SocketsHttpHandler in .NET 7 P7 and the quirk with missing ContentLength in the response is still there.

[simonrozsival]

@karelz is the behavior of SocketsHttpHandler and the HttpResponseMessage it returns described above the expected behavior or is that a bug?

[mconnew]

WCF has the ability to enable modifying the HttpMessageHandler stack. There's a blog post here that demonstrates how to modify the behavior of the HttpClientHandler where you can wrap it if needed. If you need to keep using the native implementation, you could wrap it with an implementation which removes the Content-Length header once the response has been received before returning the response message to WCF. This will cause WCF to switch to a mode where it keeps reading the content until it either reaches the end, or it hits the configured MaxReceivedMessageSize byte limit.

[mconnew]

@simonrozsival, @DuncWatts, @karelz, which repo is the best place to track this? I think it's been sufficiently investigated to know this is not an issue WCF's implementation or usage of HttpClient.

[simonrozsival]

If @karelz confirms this is a bug then we should move it to dotnet/runtime.

[karelz]

@simonrozsival can you please restate what exactly is the SocketsHttpHandler behavior that you suspect may be a bug? (I tried to gather it from previous comment, but it was not 100% clear to me sadly :() Thanks!

If we find a bug in some component, I would recommend to create a new bug with specific details instead of this one which is IMO not super-clear at the moment. (there is of course always an option it is just me being slow ;))

[simonrozsival]

@karelz when HttpClient is used with SocketsHttpHandler, the response message's content.Headers.ContentLength doesn't have a value even though the HTTP response contained the Content-Length header. This doesn't happen when NSUrlSessionHandler is used instead. I think I'll just go ahead and create a new issue in dotnet/runtime with the details.

[karelz]

Yeah, that looks like a bug to me. Please file it. With info if it is specific to iOS, or general. And ideally with a standalone repro for us to act on. Thanks! cc @MihaZupan @ManickaP @CarnaViire

[DuncWatts]

Removing the Content-Length header using a DelegatingHandler works a charm thank you. The previous workaround of setting UseNativeHttpHandler to false did work but triggered a lot of errors with the linker which I was struggling to resolve.

[simonrozsival]

tl;dr: The root of the problem is gzip compression of the response. The server works correctly. It's not a bug in .NET. It's possibly a bug in WCF and/or NSUrlSessionHandler.

@karelz I won't open any issue in dotnet/runtime after all. Sorry for the confusion 😄

---

I did some more digging and I realized that my previous conclusions were incorrect. I overlooked the fact that the response is gzipped and so the Content-Length contains the size of the compressed payload (interesting in this case the gzipped payload is 348 B instead of the original 325 B). WCF sets the AutomaticDecompression property of the handler and it leaves all the decompression implementation details to the handler.

The .NET implementation (`) makes thecontent.Headers.ContentLengthvalue inaccessible, while theNSUrlSessionHandlerimplementation returns the size of the compressed payload. WCF assumes that when theContentLengthvalue is available, it's the size of the plaintext content and not necessarily the exact value of theContent-Length` header value.

@mconnew The docs of the ContentLength property say that it should contain the value of Content-Length and the Mac/iOS implementation sticks to the exact wording of the docs. I think WCF should ignore the value if the payload was compressed instead of relying on .NET's internal implementation. Or is there some documentation that would suggest it's a bug in the Mac/iOS implementation.

[MihaZupan]

wcf shouldn't be validating Content-Length here, it's either redundant or just wrong.

If decompression happened, or the server returned both the content length and chunked transfer encoding, or some handler somewhere changed the content/headers, this verification logic will be wrong.

SocketsHttpHandler will enforce the correctness of the response content length (it has to - it's the core of the protocol). If the server sent us too many bytes, you won't see them when reading the content stream. If the server sent us too few bytes, the ReadAsync on the content stream will throw.

[MihaZupan]

Ah, I see you just commented that decompression was indeed the issue. This should be fixed in wcf. You should let the underlying handler handle the protocol correctness. WCF shouldn't be enforcing the value of the Content-Length as it can be different from the number of bytes read from the content.

[mconnew]

What was the behavior with HttpWebRequest? We are using the exact same logic on .NET Framework.

[psudodev]

I am facing similar issue on Maui Android on .net 7 @chamons did you find any work around for this