Migration from HttpWebRequest to HttpClient for net452

[rahuldutta90]

Hi,

I am migrating our ADLS sdk from httpwebrequest to HttpClient because httpclient is better performant in netcore and provides more flexibility and also it is recommended. However I have seen regression in performance on post requests (append) for httpclient in net452.

Usage before: new HttpWebrequest instance per request. Current usage: Single httpClient instance with WebRequestHandler for all requests

I have done single threaded append tests (post requests) also multthreaded tests and both show there is a regression (max 1.25 times) in performance with webrequesthandler (httpclient).

I have kept all the settings same like ServicePointManager.UseNagleAlgorithm = false, ServicePointManager.Expect100Continue = false and set the ServicePointManager.Defaultconnection limit for the multithreaded test.

Our sdk is run by quite a few customers on netframework so I would want to avoid this regression. Can someone comment why is there a difference and if I should do something else to get equal performance?

[caesar-chen]

What's the .NET Core's version? Is it greater that 2.1?

ServicePointManager is a no-op in .NET Core for either HttpWebRequest or HttpClient classes.

[rahuldutta90]

This is for .net framework (net452) not netcore. For NetCore I already confirmed the benefit of httpclient over httpwebrequest. For net452 I am seeing this regression.

[karelz]

We do not track .NET Framework problems on GitHub. It's the same set of people though.

Given that HttpClient is implemented over HttpWebRequest on .NET Framework, it is most likely a problem in the settings you're using. I would recommend to isolate it into a simple minimal repro and make it BenchmarkDotNet to demonstrate the difference. It will IMO reveal the difference of settings on the way.

We can keep it opened for a while here to track the final resolution.

[rahuldutta90]

Working on it, will paste the sample code and test result by tomorrow.

[rahuldutta90]

@karelz Following is the httpclient and httwebrequest simple code (Please ignore other application layer codes since I have copied it from my sdk). I have run this test in a Azure VM in same region as my adl account to send requests just to count out network issues.

HTTPWEBREQUEST:

internal class HttpWebRequestExample
    {
        private static void AssignCommonHttpHeaders(HttpWebRequest webReq, AdlsClient client, RequestOptions req, string token, string opMethod)
        {
            webReq.Headers["Authorization"] = token;
            webReq.UserAgent = client.GetUserAgent();
            webReq.ServicePoint.UseNagleAlgorithm = false;
            webReq.ServicePoint.Expect100Continue = false;
            webReq.Headers["x-ms-client-request-id"] = req.RequestId;
            webReq.Method = opMethod;
        }

        internal static async Task<double?> MakeSingleCallAsync(string urlString, string method,
            ByteBuffer requestData, ByteBuffer responseData, AdlsClient client, RequestOptions req, OperationResponse resp)
        {
            string token = null;
            if (string.IsNullOrEmpty(urlString))
            {
                return null;
            }

            try
            {
                // Create does not throw WebException
                HttpWebRequest webReq = (HttpWebRequest)WebRequest.Create(urlString);

                // If security certificate is used then no need to pass token

                token = await client.GetTokenAsync().ConfigureAwait(false);

                AssignCommonHttpHeaders(webReq, client, req, token, method);
                Stopwatch timer = Stopwatch.StartNew();
                if (!method.Equals("GET"))
                {
                    if (requestData.Data != null)
                    {
                        using (Stream ipStream = await webReq.GetRequestStreamAsync().ConfigureAwait(false))
                        {
                            await ipStream.WriteAsync(requestData.Data, requestData.Offset, requestData.Count).ConfigureAwait(false);
                        }
                    }
                    else
                    {
                        webReq.ContentLength = 0; 
                    }
                }

                using (var webResponse = (HttpWebResponse)await webReq.GetResponseAsync().ConfigureAwait(false))
                {
                    resp.HttpStatus = webResponse.StatusCode;
                    resp.RequestId = webResponse.Headers["x-ms-request-id"];
                }
                timer.Stop();
                return timer.ElapsedMilliseconds;

            }// Any unhandled exception is caught here
            catch (Exception e)
            {

            }

            return null;
        }
    }

HTTPCLIENT:

  internal class HttpClientExample
    {
        private static HttpClient AdlsHttpClient = null;
        static HttpClientExample()
        {
            var handler= new WebRequestHandler();
            AdlsHttpClient = AdlsHttpClient = new HttpClient(handler, false); 

        }

        private static void AssignCommonHttpHeaders(HttpRequestMessage webReq, AdlsClient client, RequestOptions req, string token, string opMethod)
        {
            webReq.Headers.TryAddWithoutValidation("Authorization", token);
            webReq.Headers.TryAddWithoutValidation("User-Agent", client.GetUserAgent());
            webReq.Headers.TryAddWithoutValidation("x-ms-client-request-id", req.RequestId);
            webReq.Method = new HttpMethod(opMethod);
        }

        internal async static Task<double?> MakeSingleCallAsync (string urlString, string method, ByteBuffer requestData, ByteBuffer responseData, AdlsClient client, RequestOptions req, OperationResponse resp)
        {
                string token = null;

                try
                {
                    // Create does not throw WebException
                    HttpRequestMessage webReq = new HttpRequestMessage();
                    webReq.RequestUri = new Uri(urlString);

                    // If security certificate is used then no need to pass token
                    var httpClient = AdlsHttpClient;
                    Stream postStream;
                    Stopwatch watch = Stopwatch.StartNew();
                    token = client.GetTokenAsync().GetAwaiter().GetResult();
                    watch.Stop();

                    if (!method.Equals("GET"))
                    {
                        if (requestData.Data != null)
                        {
                            postStream = new MemoryStream(requestData.Data, requestData.Offset, requestData.Count);
                            webReq.Content = new StreamContent(postStream);
                        }
                        else
                        {
                            postStream = new MemoryStream();
                        }
                        webReq.Content = new StreamContent(postStream);
                    }
                    AssignCommonHttpHeaders(webReq, client, req, token, method);

                        try
                        {
                            Stopwatch timer = Stopwatch.StartNew();
                            using (var webResponse = await httpClient.SendAsync(webReq))
                            {
                                resp.HttpStatus = webResponse.StatusCode;
                                if (webResponse.Headers.Contains("x-ms-request-id"))
                                {
                                    resp.RequestId = webResponse.Headers.GetValues("x-ms-request-id").FirstOrDefault();
                                }
                                if ((int)resp.HttpStatus >= 300)
                                {
                                // exception
                                return null;
                                }

                            }
                    timer.Stop();
                    return timer.ElapsedMilliseconds;
                        }
                        catch (HttpRequestException e)
                        {
                        }
                    // Any unhandled exception is caught here
                }
                catch (Exception e)
                {

                }

                return null;
            }

    }

And this is how I call the tests and capture latencies over 40 requests:

static void HttpTest(string path, AdlsClient client, bool ishttpclient)
        {
            string guid = Guid.NewGuid().ToString();
            string urlstring = $"https://{client.AccountFQDN}/webhdfs/v1{path}?op=CREATE&filesessionid={guid}&overwrite=true&leaseid={guid}&write=true&syncFlag=DATA";
            string text1 = RandomString(4*1024*1024);
            byte[] textByte1 = Encoding.UTF8.GetBytes(text1);
            double latency = 0;
            if (ishttpclient)
            {
                latency+=HttpClientExample.MakeSingleCallAsync(urlstring, "PUT", default(ByteBuffer), default(ByteBuffer), client, new RequestOptions(), new OperationResponse()).GetAwaiter().GetResult().Value;
            }
            else
            {
                latency+=HttpWebRequestExample.MakeSingleCallAsync(urlstring, "PUT", default(ByteBuffer), default(ByteBuffer), client, new RequestOptions(), new OperationResponse()).GetAwaiter().GetResult().Value;
            }
             int n = 40; int offset = 0;
            for (int i = 0; i < n; i++)
            {
                string syncflag = (i == n - 1) ? "CLOSE" : "DATA";
                urlstring = $"https://{client.AccountFQDN}/webhdfs/v1{path}?op=APPEND&filesessionid={guid}&append=true&leaseid={guid}&write=true&syncFlag={syncflag}&offset={offset}";
                if (ishttpclient)
                {
                    latency+=HttpClientExample.MakeSingleCallAsync(urlstring, "POST", new ByteBuffer(textByte1, 0, textByte1.Length), default(ByteBuffer), client, new RequestOptions(), new OperationResponse()).GetAwaiter().GetResult().Value;
                }
                else
                {
                    latency += HttpWebRequestExample.MakeSingleCallAsync(urlstring, "POST", new ByteBuffer(textByte1, 0, textByte1.Length), default(ByteBuffer), client, new RequestOptions(), new OperationResponse()).GetAwaiter().GetResult().Value;                }
                offset += textByte1.Length;
            }
            Console.WriteLine((ishttpclient ? "HttpClient: " : "HttpWebRequest: ") + latency / (n + 1));
        }

Only config changes I have made is: ServicePointManager.UseNagleAlgorithm = false; ServicePointManager.Expect100Continue = false;

And here is the latencies I saw:

HttpClient: 277.024390243902 HttpWebRequest: 282.048780487805

HttpClient: 284.268292682927 HttpWebRequest: 277.243902439024

HttpClient: 284.268292682927 HttpWebRequest: 277.243902439024

HttpClient: 274.80487804878 HttpWebRequest: 265.951219512195

HttpClient: 287.414634146341 HttpWebRequest: 278.780487804878

HttpClient: 433.09756097561 HttpWebRequest: 275.219512195122

HttpClient: 434.585365853659 HttpWebRequest: 274.829268292683

HttpClient: 260.780487804878 HttpWebRequest: 268.073170731707

[rahuldutta90]

@karelz I also think that I am missing something for httpclient. LEt me know what I am missing here.

[rahuldutta90]

@karelz did you get a chance to take a look?

[caesar-chen]

On .NET Core, HttpWebRequest is based on HttpClient, while on Framework it's the other way around.

@davidsh Do you have insights on why Post request is slower with HttpClient than HttpWebRequest in Framework?

[davidsh]

@davidsh Do you have insights on why Post request is slower with HttpClient than HttpWebRequest in Framework?

There is nothing specific to POST requests vs. GET requests with regards to performance differences between HttpClient API and HttpWebRequest API.

However, HttpClient APIs fundamentally use Task async mechanisms. Your example code above is making async Task dispatch worse by using synchronous blocking .GetAwaiter().GetResult() pattern. That is not a good best-practice of using HttpClient APIs. You will observe lower performance due to using a pattern like this.

If you are trying to achieve high throughput by doing multiple requests, you should use async await patterns in your code.

Diagnosing the true bottleneck with performance issues is best done using the PerfView application which will help pinpoint where the slowdown is (i.e. network waits vs. CPU time vs. .NET Task dispatch etc.). See: https://github.com/Microsoft/perfview

[rahuldutta90]

@davidsh You mean to say that httpwebrequest does not use the same task async mechanisms, so with httpclient this becomes worse?

This above code was a bare minimal repro sample code and we do use similar pattern in our SDK. I will try to change the pattern in future.

There is another interesting behavior I saw. So we have this perf test of our sdk code (which I havent shared here and based on this test only I filed this issue at the begining) posts 4 MB data on concurrent threads (128) (the pattern is synchronous blocking which I agree might be bad for httpclient) and uploads total 195GB of data. Using that same application code, I see better performances in netcoreapp2.0 than net452. In fact I tried to target the app to net461 which gives even better performance (with rest other parameters exactly same - environment, application code).

I am not sure but were there changes in httpclient in .net framework between net452 and net461 that can give this perf difference?

[davidsh]

@davidsh You mean to say that httpwebrequest does not use the same task async mechanisms, so with httpclient this becomes worse?

HttpWebRequest in .NET Framework internally uses the Begin/End APM mechanism for async calls. So, it doesn't use as much of the Task mechanisms.

I am not sure but were there changes in httpclient in .net framework between net452 and net461 that can give this perf difference?

There were many bug fixes etc between those versions. But to diagnose your performance issues, you should look into the PerfView tracing etc. to narrow down where the time is actually spent.

[davidsh]

If this is still a problem for you in .NET Framework, please open an issue here:

https://developercommunity.visualstudio.com/spaces/61/index.html