Deprecate HttpListener

[karelz]

System.Net.HttpListener does not support many modern protocols. It is compat-only (i.e. only critical fixes, no new improvements, enhancements). It's useful for low-volume basic server requests, but does not scale to modern requirements of HTTP servers (incl. performance).

Use KestrelHttpServer instead.

[valeri2]

But Kestrel on Unix does not directly support the Windows authentication. https://github.com/aspnet/KestrelHttpServer/issues/2168 In my project, which I want to transfer to Unix, I use a HttpListener with Windows authentication.

[karelz]

@valeri2 Even Kestrel doesn't want to implement Windows authentication on Linux due to security concerns: https://github.com/aspnet/KestrelHttpServer/issues/2168#issuecomment-345033067. The less reason to implement it in HttpListener.

[springy76]

I would not consider HttpListener deprecated. Its directly tied to http.sys driver which constantly gets updated with newer IIS or OS versions since IIS itself heavily depends on it. Kestrel only can bind to entire ports while using http.sys/HttpListener many applications can simultaneously bind to port 80 as long as they use non-conflicting paths.

[karelz]

@springy76 not all HttpListener implementations are based on http.sys (e.g. UWP, Linux/Mac on .NET Core). We might eventually migrate even the Windows implementation out of http.sys to remove inconsistencies between platforms and to simplify maintenance - it is something we plan to consider. Even if http.sys is maintained, it does not mean that new APIs will be exposed from HttpListener, or that HttpListener will opt-in into new options, settings and APIs in http.sys.

From .NET point of view, HttpListener is legacy without active work on it.

[masonwheeler]

@karelz Wait a sec. Someone points out how being based on http.sys provides very useful functionality that they're relying on, that Kestrel can't duplicate, and your response is "meh, we're probably going to remove that anyway"? Seriously?!?

Also, how is it that you regard something that drags in a dependency on ASP.Net Core as a good alternative to a simple, lightweight server class with no dependencies outside of the standard library? Sorry, but no. From the perspective of a developer actually using it, that's simply insane!

[karelz]

@masonwheeler we need to weight in usefulness vs. maintenance cost. Is that particular feature so much more important to EVERY developer vs. other investments. Maintaining something, keeping it up to latest standards, etc. does cost something - see the cost in Kestrel. Using your arguments we would not be able to remove libcurl Http client ever, because it may have some useful corner-case features, we can't provide easily in SocketsHttpHandler.

If you want to help change things, I would recommend to focus on facts, and usage scenarios / commonality (e.g. how many customers truly depend on specific feature?)

Also note, that the fact we do not evolve HttpListener further, that it is compat-only API is a fact since its introduction into .NET Core 2.0. The deprecation is just clear public statement of where we are and where we have been for last year. It sets expectations. If you want something modern, it is not HttpListener.

[masonwheeler]

@karelz

If you want to help change things, I would recommend to focus on facts, and usage scenarios / commonality

I chose HttpListener specifically because it does not drag dependencies on a massive, bloated ASP.Net framework into your project. Saying now that it's deprecated in favor of something that does is no good. Deprecation is supposed to mean "don't use this because we've got a better alternative." Providing a worse alternative is a very bad use of the concept. The Khronos Group is infamous for not understanding this; please don't be like them.

Also note, that the fact we do not evolve HttpListener further, that it is compat-only API is a fact since its introduction into .NET Core 2.0. The deprecation is just clear public statement of where we are and where we have been for last year. It sets expectations. If you want something modern, it is not HttpListener.

Yes, I understand that that's your position. I'm saying that this is a bad position and needs to be reconsidered. If there are new and better core HTTP server features available, what's stopping you from making them available to everyone even if they don't want to bloat up their project with unwanted frameworks and unnecessary IIS crud?

[Mike-E-angelo]

So, late to the party here. Please consider the magic that is happening over at Ooui, which allows you to pipe the rendering of .NET elements straight into a webpage. Additionally, you can perform this same magic in a native-hosted/store scenario (UWP as an example). This is all supported at present through the use of HttpListener. If I am understanding this correctly, this issue is saying that we would remove the use of System.Net and instead add a dependency to Microsoft.AspNetCore.Server.Kestrel in our native-hosted/store scenarios to get the same functionality? It would be incredibly valuable to know that what "just works" with HttpListener presently with Ooui also "just works" with KestrelHttpServer, too.

[karelz]

First a little bit of history: Long time ago, HttpListener was found to be insufficient in the areas of modern feature richness, modern API surface and performance characteristics. ASP.NET team forked the code base and evolved it into WebListener, which then evolved into Kestrel. The moment this evolution path started, HttpListener became maintenance-only (aka non-modern aka obsolete).

If current HttpListener feature set is sufficient for your needs, you can still use it. We are just warning users to not expect modern evolution, APIs and performance improvements in the space. If you need any of that, we suggest to consider Kestrel or other alternatives.

@masonwheeler I chose HttpListener specifically because it does not drag dependencies on a massive ... Providing a worse alternative is a very bad use of the concept.

HttpListener may have fewer dependencies, for which you pay by limited capabilities (e.g. feature set, perf). Overall we are convinced that Kestrel is the better alternative from many point of views, especially those which are most important to majority of our customers.

@masonwheeler I'm saying that this is a bad position and needs to be reconsidered. If there are new and better core HTTP server features available, what's stopping you from making them available to everyone even if they don't want to bloat up their project with unwanted frameworks and unnecessary IIS crud?

As I suggested above, if you want us to reconsider, we will need some hard data on usage, limitations of Kestrel, etc. So far I am aware of only 1 feature which is not better in Kestrel. Its usefulness seems to be fairly limited. Maintaining multiple HTTP servers is cost, it does not come for free. The cost is high (you have to do everything twice at minimum), it also fractures ecosystem by providing multiple official ways of doing the same thing. Investing into Kestrel to fulfill all the needs of various HTTP server scenarios seems to be overall better investment. BTW: AFAIK the IIS dependency in Kestrel is just one of the options, it is not tied to it.

@Mike-EEE this issue is saying that we would remove the use of System.Net and instead add a dependency to Microsoft.AspNetCore.Server.Kestrel in our native-hosted/store scenarios to get the same functionality?

If your projects are satisfied with current limited HttpListener capabilities "as is", you can keep using it. If you need more, you might want to consider Kestrel (although I don't think it is available in UWP). HttpListener is not going to disappear, however, you should not expect it to evolve further. It has reached its full potential.

It would be incredibly valuable to know that what "just works" with HttpListener presently with Ooui also "just works" with KestrelHttpServer, too.

That sounds like a very good idea. I am sure quite a few developers would appreciate that. I hope some Kestrel experts will be able to provide such guidance.

FYI: For context, check some of the discussion here: https://github.com/dotnet/designs/issues/9 (just please do not reopen the discussion there - it does not belong on the uber-issue). If there are additional real-world usage patterns of HttpListener, I'll be happy to hear details.

[Mike-E-angelo]

Cool... thanks @karelz ... I am starting to dig and do some analysis around this space, myself. In my view it is incredibly important as at a glance it totally (and finally) satisfies the request for a Ubiquitous.NET. That is, using this magic bridge/connection, you can enable a world where it's one .NET codebase and one .NET viewbase total in your solution, rendered through a WebView (as presented/demonstrated above). That is a big deal and I do not want to jeopardize it. 😄

So it does sound like at the very least we will still be able to use HttpListener, but it will be deprecated? I am sure that will feature all the expected warnings in code as such.

If you need more, you might want to consider Kestrel (although I don't think it is available in UWP).

So I thought that UWP supported .NET Standard 2.0? That is the TargetFramework of Microsoft.AspNetCore.Server.Kestrel so it should "just work" right? Or am I missing something (again 😆)?

[karelz]

So it does sound like at the very least we will still be able to use HttpListener, but it will be deprecated? I am sure that will feature all the expected warnings in code as such.

Yes, I am not aware of any instance when we deprecate things and they immediately stop working :). The warnings are soft, easy to disable and opt-in currently - read more in this repo about the platform compat tooL; https://github.com/dotnet/platform-compat

So I thought that UWP supported .NET Standard 2.0? That is the TargetFramework of Microsoft.AspNetCore.Server.Kestrel so it should "just work" right?

I'd ask Kestrel folks where and how they are supported. IMO Kestrel needs to listen on ports which is not supported in UWP. Not sure if it can work without that (you have reached the limits of my knowledge).

BTW: Also note, that there are currently limitations on various platforms:

• HttpListener does not support HTTPS in UWP and on Linux - https://github.com/dotnet/corefx/issues/14691 (because it is C# implementation, not based on http.sys which is not available on the platforms).
• WebSocket not supported on Win7 - https://github.com/dotnet/corefx/issues/13663#issuecomment-313783837.

[Mike-E-angelo]

Kestrel needs to listen on ports which is not supported in UWP

So to add a little more context here, the HttpListener in Ooui is listening on a port and works in UWP. This is because it isn't intended for any external connections, and it's only listening for the process in which it is running. This is a supported scenario in UWP, so it would be interesting to see if Kestrel works as well when listening within its own process.

[springy76]

and it's only listening for the process in which it is running

I wonder how this is being checked?

[Mike-E-angelo]

@springy76 It's probably better stated that it doesn't accept any connections from external requests, so as a result it only accepts requests/connections running from within the process from which the application is running. That is my understanding, at least. Also, while I have gotten this to work locally on UWP, it hasn't been officially verified as a deployed product.

As such, I do have an outstanding question on Xamarin's forums, but if I do not get an answer there, I will have to brogrammer-up and get some HelloWorld's deployed to get a definitive answer. Unfortunately, Droid development hasn't exactly been friendly to me, and I haven't even touched iOS.

If someone has some valuable knowledge around these scenarios it would be greatly appreciated if you could share. 👍

[clrjunkie]

@masonwheeler

See discussion at: https://github.com/dotnet/designs/issues/9

[HendricksonJS]

@valeri2 , it seems to me that Kerberos (e.g. Active Directory) is one of the few reasons a developer would want to run an application on Windows vs. Linux. The Linux community by and large doesn't want to deal with Kerberos, whereas Windows has been pretty good at handling it for a long time. If Kerberos is a requirement (and tbh I would use if I could for an internal or enterprise app), you should stick with Windows.

[boeremak]

I would also like it if you could keep the HttpListener as it is and maybe evolve it as a no-frills HTTP Api/Server alongside the other ones. Seeing how many times it has been ported by other people already there seems to be plenty of demand for it.

As for my own use: I like that in 3 lines of code I can listen for a connection and do almost anything with it without needing a lot of dependencies or learning a new paradigm.

There are some things that can easily be made better, like the certificate format it expects. And some other things I am sure you are already aware of.

In other words, I think you'd make a lot of people happy, including me, if you keep HttpListener around and Microsoft would still support it.

[pablocar80]

Hello @karelz , your recommendation is not to use HttpClient and use Kestrel that has been moved into ASP.NET Core. However, I don't fully understand how to go from one to the other. I have a desktop application that listens on port 443 and replies web requests, and not full blown ASP.NET Core project with all the configuration and setup that a web server requires, frameworks, paradigms and all its baggage. Is there a way to make a console application to respond to HTTP requests? or are we adding the constraint that only ASP.NET Core projects can do that?

[karelz]

@pablocar80 AFAIK you can write minimal Kestrel server with little overhead, baggage, etc. Check out their docs, I am not expert on the topic.

[jkotas]

The docs are at https://docs.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-2.2 . The minimal sample linked from the docs is not a full blown ASP.NET project, but it still carries some of the baggage with it: https://github.com/aspnet/Docs/blob/master/aspnetcore/fundamentals/servers/kestrel/samples/2.x/KestrelSample/Startup.cs#L4

cc @davidfowl

[pablocar80]

Would it make sense for the DotNetCore team to offer the HttpClient class as a thin wrapper that under the hood uses Kestrel? This would help with migration A LOT.

[pablocar80]

@jkotas thank you for the links. I checked out the documentation and the references needed are specific to NET Core. Ideally we should have HttpClient or its replacement compiling under NET Standard.

[davidfowl]

Nothing in the docs show a minimal example. There are different levels of minimal:

1. Using the webhost/generic host
   • With or without a startup class (the generic host has better startup performance as part of 3.0)
2. Wire up the server instance yourself without a host.

The latter isn’t pretty but we could spend some cycles making a nicer API to get to the pre request callback if that was a concern.

[pablocar80]

Hello @davidfowl ! If I understood correctly then option 2 is what HttpListener is for.

Right now, if you have HttpListener, the Portability Analysis tool tells you that you're good to go.

Then you realize that not really -- the .NET Core implementation doesn't support HTTPS which is a reasonable requirement to put a system in production.

Then you look up the issue on why HTTPS isn't supported, and it's because there's this thread that says that HttpListener should be abandoned altogether.

So there's really a disconnect between the happy land of the portability analyzer that tells you you're ready to jump, and the reality that HttpListener in NET Core compiles yet isn't really implemented.

[clrjunkie]

The latter isn’t pretty but we could spend some cycles making a nicer API to get to the pre request callback if that was a concern.

Suggestion:

Spend those cycles on an API that mimics the .NET HttpListner API. The API does not have to serve as a foundational API for upper ASP.NET layers, but rather be considered an optional slim "Adapter" class aimed for developers who feel comfortable with that API definition.

How does that sound?

[davidfowl]

Exposing the HttpListener API from Kestrel IMO doesn’t make any sense. It would make more sense to just “fix” HttpListener but to make it production ready would be a wasted effort (wasting cycles fixing hard networking and security issues I’m not places etc). Now you could argue that Kestrel and HttpListener could share the same underlying core but there’s tons of code in the upper layers that actually handle the protocol control flow that wouldn’t be shared. HttpListener is also a pull model and ASP.NET Core exposes a push model for handling requests.

There are other layering concerns with replatting one on the other (like namespace and assembly dependencies). Even the core of kestrel is pluggable and things like TLS are plugged in via this extensibility (also multiple transports) and I’m not sure there’s an appetite to push this down into the lower levels.

Layering the other way (building Kestrel on top of HttpListener) would be wasteful layering wise and it’s already possible plug server implementations into ASP.NET Core wholesale so this would be pointless IMO.

All that said I’m not sure what the best call is here because we don’t have true replacement that’s built into the “core” shared framework that is a “production ready” (whatever that means here) server. That’s a bigger strategic decision that needs to be discussed.

[davidfowl]

cc @DamianEdwards

[clrjunkie]

Exposing the HttpListener API from Kestrel IMO doesn’t make any sense.

Why?

It would make more sense to just “fix” HttpListener

Why?

Now you could argue that Kestrel and HttpListener could share the same underlying core but there’s tons of code in the upper layers that actually handle the protocol control flow that wouldn’t be shared.

The .NET HttpListener class does not implement any protocol control flow. Thats handeled by lower level Windows components which HttpListener “wraps”

HttpListener is also a pull model and ASP.NET Core exposes a push model for handling requests.

What is Kestrel “Push model” ?

There are other layering concerns with replatting one on the other (like namespace and assembly dependencies). Even the core of kestrel is pluggable and things like TLS are plugged in via this extensibility (also multiple transports) and I’m not sure there’s an appetite to push this down into the lower levels.

How do these concerns impact a user facing HttpListener API? What dependecies exist?

[davidfowl]

Why

It would be a performance regression to use another API which has a different header and HttpContext structure.

Why

If this is about API porting then HttpListener is compatible with what was there on .NET Framework.

The .NET HttpListener class does not implement any protocol control flow. Thats handeled by lower level Windows components which HttpListener “wraps”

Sure it is. On Linux the entire protocol is implemented. It’s uninteresting to discuss the windows http.sys implementation as we would never base our implementation on that since it is tied to the OS and requires a new OS to get fixes and features. It’s also inconsistent with the Linux behavior (see prefix registration and HTTPS). There’s a fork of HttpListener in ASP.NET Core called HttpSysServer that doesn’t pretend to be an abstraction. We also don’t pay for another abstraction as we gutted it and bind directly to the native http abstractions to avoid the indirection.

So yes the entire control flow for both HTTP/1 and HTTP/2 needs to be implemented in managed code.

What is Kestrel “Push model” ?

With kestrel and ASP.NET Core in general you register a callback (usually a piece of middleware) that gets invoked per request. Unlike HttpListener the caller is responsible for implementating the accept loop (PS accepting on connection at a time may not be optimal).

How do these concerns impact a user facing HttpListener API? What dependecies exist?

Not sure what you mean. The entire discussion is about supporting the existing API, not about a replacement right?

[pablocar80]

The entire discussion is about supporting the existing API, not about a replacement right?

I personally would be open to change my code as long as we get to a new standard that works everywhere. The class doesn't need to be HttpListener, it could even be a higher abstraction class that already receives requests and dispatches async calls to request controllers.

But in any case, it needs to (1) compile in NET Standard, (2) implement the HTTP protocol, (3) allow to read the header and content of requests, (4) allow to override and send custom response headers and write the content of the response.

What I would NOT want is a web server that reads the URL and expects to find files on the disk or programs with a matching name, or that requires the user to install IIS/Apache, or that I need to change the project type to an ASP.NET flavor, etc.

[davidfowl]

Does it need to be included in netstandard? Does it need to run on .NET Framework? Is the goal compatibility or performance or “edge readiness”? What platforms does it need to run on? Does it need to an extensibility model (like middleware) so that static file support can be added) what about logging?

[pablocar80]

Does it need to be included in netstandard? Does it need to run on .NET Framework?

NET Standard has been critical to me to work migrating. This is because I have been able to convert libraries to NET Standard and keep compiling the current version in the NET Framework. If it wasn't for NET Standard you would have a "NET Core migration" project in parallel, that takes a lot of time, while the codebase evolves in two fronts. So NET Standard is an amazing thing to keep everything together.

Is the goal compatibility or performance or “edge readiness”?

I don't know what "edge readiness" is.

For compatibility, it's really frustrating when you move to another platform and something doesn't work, and you have to debug and research specifically for that platform. Other languages that are cross platform give you a reasonable confidence that if your program works in one operating system then it will work in another (Java, Kotlin, Golang, Rust, Python).

For performance, here's an open source web server developed in C# on top of TcpListener, and the performance is very good, complies easily with C10K: uhttpsharp

What platforms does it need to run on?

For me, the answer is Windows, Linux and Mac computers.

Does it need to an extensibility model (like middleware) so that static file support can be added) what about logging?

Like HttpListener, it should have none of that built-in. The programmer handles the requests and decides what to reply. If they want to put a static file in the response that's up to them. If they want to log somewhere they add a line of code to log with library of choice in your code that processes the request.

[davidfowl]

Sounds like you’re looking for a compatibility story and I think HttpListener is that but it’s not our web server story for .NET Core.

We have a cross platform sever and it’s called kestrel + asp.net core and it scales from very small libraries to very large applications with a consistent application model.

HttpListener is a lower level API with 0 extensibility originally designed for expose http.sys to managed code. This issue IMO is about compatibility between .NET Framework and .NET Core. Maybe it’s best to do the bare minimum to expose the right set of APIs to fix whatever is broken and then open another isssue to pinpoint the need for a non asp.net based webserver built into some future version of netstandard

[clrjunkie]

It would be a performance regression to use another API which has a different header and HttpContext structure.

Maybe, but how can you be so sure that such “hit” is unacceptable to a point where it doesn’t make sense - for users? Nuget shows 33,003 downloads for “NETStandard.HttpListener” so its pretty clear that demand exist for the API. Moreover, given that HttpListener Request/Response API exposes standard Http Structures (e.g query string params, methods, cookies) I’m quite confident you can avoid most reparsing/copy penalties by passing those structures from kestrel as is so the hit should be negligible.

If this is about API porting then HttpListener is compatible with what was there on .NET Framework.

This is about having one underlying server implementation, consistent across platforms, activity developed, exposing an alternate API. One is simpler than Two.

Sure it is. On Linux the entire protocol is implemented. It’s uninteresting to discuss the windows http.sys implementation as we would never base our implementation on that since it is tied to the OS and requires a new OS to get fixes and features. It’s also inconsistent with the Linux behavior (see prefix registration and HTTPS). There’s a fork of HttpListener in ASP.NET Core called HttpSysServer that doesn’t pretend to be an abstraction. We also don’t pay for another abstraction as we gutted it and bind directly to the native http abstractions to avoid the indirection. So yes the entire control flow for both HTTP/1 and HTTP/2 needs to be implemented in managed code.

That’s all beside the point – HttpListener API surface stands orthogonal to who handles control flow and that separation is clearly evident in the Windows implementation. Point being, having an Adapter class convert “Kestrel API” to “HttpListener API” would be equivalent to how currently the .NET HttpListener class serves as an Adapater class which converts the “Windows HttpServer Win32 API” to a .NET OO API. From .NET HttpListener standpoint “Control Flow Handling” is “Uninteresting” and I see no reason why it can’t stay that way if you move the underlying Http Server implementation from a “Windows” implementation to a “Kestrel” implementation.

With kestrel and ASP.NET Core in general you register a callback (usually a piece of middleware) that gets invoked per request. Unlike HttpListener the caller is responsible for implementating the accept loop (PS accepting on connection at a time may not be optimal).

Wait. Let’s discuss “Kestrel” in separate from “ASP.NET” (because they are separate). From what I have come to learn there is a tradition where network servers are architectured around the Socket model which is essentially a blocking request queue model providing callback registration as a user abstraction. Why isn’t kestrel following this proven model, especially if it was not designed to be used by users directly but rather through ASP.NET abstractions?

Not sure what you mean. The entire discussion is about supporting the existing API, not about a replacement right?

I don’t quite understand your arguments in this context. You wrote about: “namespace and assembly dependencies”, “extensibility”, “TLS”. How do these hinder implementing an alternate user-facing API?

[pablocar80]

@davidfowl thank you for taking the time to review this request, even though your profile says you're on vacation. It seems that you have reached a decision on this and we won't change it. For me, HttpListener is extensible because you can do pretty much everything the HTTP protocol allows you to do. Do you think there is any chance we will see a low level HTTP library like HttpListener in .NET Standard? I think of if as the equivalent of TcpListener but with HTTP built on top of it. It's what I will be doing on my own since I won't get it from the platform.

[davidfowl]

@clrjunkie

Moreover, given that HttpListener Request/Response API exposes standard Http Structures (e.g query string params, methods, cookies) I’m quite confident you can avoid most reparsing/copy penalties by passing those structures from kestrel as is so the hit should be negligible.

I'm not sure how you would pass those structures as is when the ASP.NET Core HTTP abstractions are different. Can you clarify? It would also be possible for you to test this easily by just plugging the existing HTTP Listener implementation to measure the performance difference. Having worked on webserver performance over the past 5 years, I disagree with the assessment that it'll be negligible but happy to be proven wrong 😄.

That’s all beside the point – HttpListener API surface stands orthogonal to who handles control flow and that separation is clearly evident in the Windows implementation. Point being, having an Adapter class convert “Kestrel API” to “HttpListener API” would be equivalent to how currently the .NET HttpListener class serves as an Adapater class which converts the “Windows HttpServer Win32 API” to a .NET OO API. From .NET HttpListener standpoint “Control Flow Handling” is “Uninteresting” and I see no reason why it can’t stay that way if you move the underlying Http Server implementation from a “Windows” implementation to a “Kestrel” implementation.

What's not clear? We're discussing implementation details.

Wait. Let’s discuss “Kestrel” in separate from “ASP.NET” (because they are separate). From what I have come to learn there is a tradition where network servers are architectured around the Socket model which is essentially a blocking request queue model providing callback registration as a user abstraction. Why isn’t kestrel following this proven model, especially if it was not designed to be used by users directly but rather through ASP.NET abstractions?

I'm not sure if we have a deep dive on how Kestrel an ASP.NET Core's abstractions are designed but I don't think this is the right place to discuss it. Maybe this will help clarify things in the implementation (https://github.com/aspnet/AspNetCore/issues/4772). There is no request queuing model (other than using async/await).

I don’t quite understand your arguments in this context. You wrote about: “namespace and assembly dependencies”, “extensibility”, “TLS”. How do these hinder implementing an alternate user-facing API?

namespace and assemblies have to do with the current ASP.NET Core types that exist at the connection/socket layer. Kestrel's HTTP implementation is built on top of "connection middleware" (https://github.com/aspnet/AspNetCore/blob/562d119ca4a4275359f6fae359120a2459cd39e9/src/Servers/Kestrel/Core/src/Internal/HttpConnectionMiddleware.cs#L15) which goes back to what I mentioned above (the bedrock issue) above how the stack comes together. What that boils down to is that there's no split of layering where you have a set of System.* namespaces only. Source would need to be copied. You could imagine an alternative world where the ConnectionContext and friends would live in System.Networking or something and the HTTP stack was a piece of middleware that was built on top.

As for the extensibility and TLS, historically, supporting HTTPS with HttpListener meant using admin privileged tools on windows to register prefixes and certs (e.g. http://blog.boxofbolts.com/ssl/windows/owin/guide/2015/06/29/https-self-hosted-windows/) AFAIK there's no user facing API to give HttpListener a certificate. That's the user facing API that I assume needs to be added.

@pablocar80

Do you think there is any chance we will see a low level HTTP library like HttpListener in .NET Standard

I don't know. If you're asking if there's a plan to expose an API like this then there isn't currently and if there was it would be in a future netstandard version that wasn't compatible with .NET Framework.

[clrjunkie]

@davidfowl The underlying Architecture behind Windows Http API is beautiful! The idea of having separate request queues identified by Url and Port independent from consumer threads represents a clean separation of concerns. Where one would choose to handle the request be it “callback”, “handler”, “controller” or what not is matter of style not architecture. I honestly don’t understand why you guys decided not to architect kestrel the same even though the goal is not to provide an O/S level API.

[davidfowl]

@davidfowl The underlying Architecture behind Windows Http API is beautiful! The idea of having separate request queues identified by Url and Port independent from consumer threads represents a clean separation of concerns. Where one would choose to handle the request be it “callback”, “handler”, “controller” or what not is matter of style not architecture.

I agree it's great but I don't think it's relevant to the discussion. Adding a request queue wouldn't affect the public API in anyway. Callbacks, handlers and controllers are all push based callbacks. You register a method that the server "calls back" so I'm not sure what that has to do with anything.

It's also why it's possible to plug HTTP.Sys into ASP.NET Core and the application logic doesn't change. Have you used ASP.NET Core at all?

PS: We're pretty off topic. Maybe best to stop the discussion here.

[clrjunkie]

I agree it's great but I don't think it's relevant to the discussion. Adding a request queue wouldn't affect the public API in anyway. Callbacks, handlers and controllers are all push based callbacks. You register a method that the server "calls back" so I'm not sure what that has to do with anything.

Everything! It appears your "Push Model" is an integral part of Kestrel as well as ASP.NET Core meaning there is an architectural dependency among the two. IMO that's probably the single main reason you find exposing an HttpListener API to be difficult - it does not "fit" your model - its more general. If the upper level ASP.NET Core abstractions where designed to consume Http Requests from Kestrel you could still provide a callback/controller API to ASP.NET users and it would have been trivial to develop an HttpListener interface for those users who like doing the consuming part themselves.

[davidfowl]

Everything! It appears your "Push Model" is an integral part of Kestrel as well as ASP.NET Core meaning there is an architectural dependency among the two. IMO that's probably the single main reason you find exposing an HttpListener API to be difficult - it does not "fit" your model - its more general. If the upper level ASP.NET Core abstractions where designed to consume Http Requests from Kestrel you could still provide a callback/controller API to ASP.NET users and it would have been trivial to develop an HttpListener interface for those users who like doing the consuming part themselves.

Honestly it's not insurmountable, it's just code and there are existing patterns for doing this. I was merely explaining the implementation differences that exist today and why I wouldn't build one of top of the other (including the fact that the performance would be worse). Again, I'm not sure what this adds to the discussion about deprecating HttpListener...

[clrjunkie]

Again, I'm not sure what this adds to the discussion about deprecating HttpListener...

If you change Kestrel to support a "Pull Model" there would be no need to Deprecate HttpListener but rather have it co-exit, just evolved...

[boeremak]

Maybe the amount of people that seem to like a low level HTTP server that let's you do anything you want with incoming requests should tell you that there are valid use cases for it.

I don't want kestrel in my micro-services. I just had a use case for HTTP requests in the Unity engine, I won't be running Giraffe in it, but a HTTP listener with some async workflows works a treat and doesn't mess around with the main game loop.

[clrjunkie]

I don’t think they don’t recognize valid use-cases, but rather they don’t have a cost metric for the learning effort and keeping up-to date with ASP.NET abstractions compared to HttpListener that is regardless if you can solve the problem using a subset of the ASP.NET API. From my experience the cost is very high when all you need is a familiar interface for doing local/remote IPC.

[quixoticaxis]

This thread, IMHO, does not mention the alternative to HttpListener. "Use KestrelHttpServer instead" is a bit broad and is not really an alternative:

1. The Kestrel main page says that it is http server for ASP.Net, which some people don't want to depend on;
2. The closest thing you can find throughout the tutorials by Microsoft is the creation of an ASP.Net application with no built-in middleware and a single custom middleware layer that provides the single handler. Which is also not an alternative because, on one hand, it raises the abstraction to the level of the whole application and, on the other hand, is a pain to implement (for example, it's a chore to utilize DependencyInjection extension when creating middleware, it works though);

If the class is deprecated, may we have either the clear alternative description or the clear statement that the class is deprecated without the alternative? I can understand why .Net Core team can be (is) focused on web application scenarios, but the "Use KestrelHttpServer instead" is the same as saying "We can sell you a car" to a person that comes to buy a single wheel.

My two cents: my team does not develop "web applications", the "API surface" term has no meaning for our system, for us any connection is valid, so we need (as some people who commented above) a simple "TcpListener-style" abstraction with HTTP support that can be used locally 1) without raising the functionality to the "application level", 2) without imposing any external "coding doctrines" and 3) without any other features we don't need like model validation, routing, authorization, serialization helpers, and so on.

[davidfowl]

Something like this?:

https://github.com/davidfowl/BasicKestrel/tree/master/BasicKestrel

[quixoticaxis]

@davidfowl thank you for the code. This is one approach someone can take.

For those who's interested but didn't do it, the other one is the creation of a single middleware class with InvokeAsync calling a user-provided asynchronous delegate; both are described here.

Unfortunately, your example code depicts the issues (well, not issues per se, but the pain points for the people who need a listener-only kind of abstraction) I've been talking about:

1. Even barebones example introduced the dependencies on ASP.Net Core which bring a bunch of application level features (for example, the generation of the error pages). Yes, we can easily opt-out of this behavior, but it is still there in our dependency graph;
2. I have a feeling that in general ASP.Net Core abstractions are meant to be used as dependency roots, not terminal dependencies, while HttlListener naturally can be injected as a terminal dependency. For example, to provide custom logic for the Run asynchronous callback, the most easy and obvious (IMHO) way to go is to inject application logic into the listener class. When using HttpListener the easy way to go is to inject the listener into the logic. This difference can be hidden under the additional user code, but it's there. It's hard to explain clearly but I hope you understand. This is what I meant when I wrote "it raises the abstraction to the level of the whole application".

[davidfowl]

Even barebones example introduced the dependencies on ASP.Net Core which bring a bunch of application level features (for example, the generation of the error pages). Yes, we can easily opt-out of this behavior, but it is still there in our dependency graph;

Can you clarify what "still there in the dependency graph" means? I might have an idea but I'd like to understand what added "cost" you are describing.

I have a feeling that in general ASP.Net Core abstractions are meant to be used as dependency roots, not terminal dependencies, while HttlListener naturally can be injected as a terminal dependency.

I don't know what that means, perhaps a code sample would clear things up for me.

For example, to provide custom logic for the Run asynchronous callback, the most easy and obvious (IMHO) way to go is to inject application logic into the listener class. When using HttpListener the easy way to go is to inject the listener into the logic. This difference can be hidden under the additional user code, but it's there. It's hard to explain clearly but I hope you understand. This is what I meant when I wrote "it raises the abstraction to the level of the whole application".

Same as above, show me some sample code and I'll try to provide you what what I think is the alternative. I don't think it raises the level of abstraction to the level of the "whole application", that's just the impression people have because of the history of ASP.NET (tied to IIS etc).

[pablocar80]

Hello! for anyone new coming to this thread -- I can confirm that I was able to migrate everything with HttpListener to Net Standard using Kestrel some months ago. It required lots of work refactoring but it was possible.

[Mike-E-angelo]

Hello! for anyone new coming to this thread

Yeah since we're updating everyone, I ended up dumping UWP/native altogether and am now on Blazor, which has that WebSocket magic built-in. This doesn't account for native hosted scenarios just yet but we have that identified and discussed in the Blazor repo so that's good enough for me ATM. 😁

That aside, I have to give everyone on the .NET team so much credit and respect for how .NET has shaped up as of late. The generic hosting model, EF, Blazor, Asp.NET Core, Razor Components, EVERYTHING!!! Really, truly, it's like developing back in the days of Silverlight again. You should unexpectedly and suddenly announce its unexplained deprecation and sunset for old times' sake.

(No don't actually do that please).

Bad jokes aside, kudos to everyone there. I keep meaning to make a blog post about it but random GitHub issue praises is all I can muster BECAUSE I AM DEVELOPING SO HARD THESE DAYS. 😅

[quixoticaxis]

@davidfowl

1. It forces us to write a lot of non essential code that we do not need (see the snippet below). It also brings a bunch of .dll-s along, and loads some of them in runtime presumably without any real need. For our project this is of no importance though.

2. I can't provide the real code, but the snippet we had looked more or less like this:

   /* somewhere deep inside the calculations */
   if (notEnoughData)
   {
   try
   {
      var listener = new HttpListener();
      SetupPrefixes(listener);
      listener.Start();
      await OrderNDataPackagesAsync(dataPackagesToOrder);
      foreach (var i in Enumerable.Range(dataPackagesToOrder))
      {
           var context = await listener.GetContextAsync();
           /* some processing and returning 200 OK at the end of the iteration */
       }
   }
   finally
   {
       listener.Close();
   }
   }

   The code sometimes needs the additional data and it orders it via http call, then receives the required number of packages through the listener, closes the listener and proceeds. It can't get the data as responses to the requests because it's being requested from the system that is out of our control (and it can only push data to the endpoint). To remove HttpListener I had to (at least I didn't find any better/easier way) host WebApi with a single controller with a single post method, then inject our whole service into the controller and write some safeguards, so we'd skip the data if none was requested. All of it was done only to receive a few data dumps when they are needed (an they are not needed often). The number of lines of code (with all configuration, code in the controller and safeguards) skyrocketed, and now the service looks like some kind of an insane web abomination that processes some requests and ignores the others. That what I was talking about: we don't have a server, nor do we need one, but we're kinda forced to have one. IMHO, bringing the web server infrastructure (and code to configure it) on board is a bit of an overkill for such a simple task. Bottom line: we ported the code, but I would not call Kestrel/ASP.Net viable alternative for the reasons I've tried to explain above.