search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.62k stars 4.56k forks source link

Networking stack - Technical roadmap #43495

Closed karelz closed 3 years ago

karelz commented 7 years ago

Summary

Our investment in the .NET networking space will focus in the following areas

Foundation: Sockets, SSL, DNS support.

Primary Goal: Provide near-native performance and rock-solid reliability.

Key investments:

Web Stack: HttpClient, ClientWebSocket

Primary goals: Ease of use; feature-rich; extensible; track emerging standards; match or exceed competitor's performance.

Key investments:

Tracking: https://github.com/dotnet/corefx/issues/21452

Emerging Technology

Primary goals: Demonstrate leadership in support for new protocols and capabilities

Key investments:

Maintenance Components: *WebRequest, Mail, HttpListener

Primary goals: Preserve existing customer investments Key investments: None.

Details

(1) Foundational

Key components

Characteristics

Requirements

Current Status

Windows

Linux

Key investments to make

(2) Web stack

Key components

Characteristics

Requirements

Current Status

Windows

Linux

ASP.NET

Key investments to make

(3) Legacy/maintenance components

Key components

Characteristics

Key investments to make

(4) Future/bleeding edge components

Not covered in this version.

Key Feedback

TODO (@karelz)

Credentials & Status

Author: @geoffkizer

Reviewed by CoreFX team: @stephentoub @davidsh @CIPop @Priya91 @wfurt @DavidGoll @karelz Reviewed by partner teams experts: @davidfowl (ASP.NET), @Tratcher (ASP.NET), @mikeharder (ASP.NET), @marek-safar (Mono/Xamarin), @mconnew (WCF), Windows networking experts, @tmds (RedHat), @omajid (RedHat) Reviewed by community networking experts: @benaadams, @NickCraver, @onovotny, @mgravell, @Drawaes

Reviews happened in stages in April. Feedback will be incorporated in updates to this doc. The delay between original review and publishing the roadmap now is purely lack of time to incorporate feedback into the doc & publish it.

Current status: Community review / feedback

Announcements: Twitter and CoreFX repo

davidfowl commented 7 years ago

I think a “Kestrel Http Server” outside CoreFx is an awkward decision. If it’s so “generic” why aren’t you including it in the standard library, like Go does? IMHO, this whole “Kestrel” branding is so irrelevant for what obviously needs to be part of the standard class-library.

Right now the HttpListener is a compatibility play and I agree that having a server as part of the "core" stack is important in general (it's standard in pretty much every other platform). But honestly, I don't see that many people complaining about a Kestrel dependency because they're writing applications and using the framework on top. As a fan of "the right layering" I sympathize with your request, you just want a simple listener in the framework, in the System namespace, not the ASP.NET namespace.

As I wrote in a separate issue, you need to provide an adapter class that converts between Kestrel API and Http Listener API , so people can benefit from a SINGLE Http server implementation. Your whole “Http Listener legacy” story is nothing I would even consider using.

Something like this but not exactly this. Once we do more refactoring it's possible the sharing can be much lower than what HttpListener exposes (which IMO is too high level and not very efficient). I actually don't think the HttpListener API is good one in general...

I think a “Kestrel Http Server” outside CoreFx is an awkward decision. If it’s so “generic” why aren’t you including it in the standard library, like Go does? IMHO, this whole “Kestrel” branding is so irrelevant for what obviously needs to be part of the standard class-library.

@omariom

What I'd like to see is building blocks used by other buildings blocks and all these building blocks used to build servers (including Kestrel).

I'll post more details soon but that's the plan. Work already started to split the transport layer from Kestrel in 2.0. That work will continue in 2.1 and beyond.

benaadams commented 7 years ago

Yes. I don't want anything besides Http Header parsing

Isn't that kinda what Kestrel does? Just need to write a single middleware (which would then work on all aspnet servers, including custom).

Is the issue that you get DI and need to use startup/hosting?

Or do you want to also drop comms and just have parsing?

clrjunkie commented 7 years ago

@davidfowl

Right now the HttpListener is a compatibility play

IMO this whole “compatibility play” approach is fundamentally flawed. People who are using HttpListener directly would except an equivalent in terms of stability, performance and feature set on Unix and not some radically different implementation that was coded by Novell for Mono ages ago (which god knows who ever used it)

honestly, I don't see that many people complaining about a Kestrel dependency because they're writing applications and using the framework on top.

Of course! I bet 99.9% of the people who used ASP.NET.CORE till now speak the “MVC Action, IoC, etc” parlance. You are now entering a NEW market – the Unix market. You are on the selling side! There is plenty of evidence that built in http server support as part of the framework is a very strong selling point – that was Go’s primary value proposition from day one and the Go Pac Man is eating!!

@benaadams

Is the issue that you get DI and need to use startup/hostin

Those are definitely things I would like to be tossed out, I want to do ‘new Kestrel()’ or better new HttpServer() and hookup a callback that gives me a simple api to the HttpRequest headers and body stream as well as the HttpResponse stream, that’s it, I don’t want anything else!

benaadams commented 7 years ago

@clrjunkie not sure I understand as you only need to include a single package; and the package is included as part of the dotnet install for 2.0, to do what I think you are after api-wise e.g. MinHttpServer

And then setting up a HttpServer (that is using Kestrel) and hook up a callback

public class Program
{
    public static void Main(string[] args)
    {
        WebHost.CreateDefaultBuilder(args)
            .UseStartup<Program>()
            .Build()
            .Run(); // or RunAsync();
    }

    // This method gets called by the runtime, to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app)
    {
        // Add handler to the HTTP request pipeline.
        app.Run(HandleRequest);
    }

    private async static Task HandleRequest(HttpContext context)
    {
        // Handle HTTP request
        var data = System.Text.Encoding.ASCII.GetBytes("Hello");

        await context.Response.Body.WriteAsync(data, 0, data.Length);
    }
}

The contents of Main and Configure could be hid in a library that exposed a class called HttpServer and it would be far faster than HttpListener on Windows is, both on Windows and Linux.

Is it the namespaces you don't like as its under Microsoft rather than System?

clrjunkie commented 7 years ago

@benaadams

Thanks for providing the snippet! This precisely shows what I don't want to see.

Look how much abstraction one needs to be aware of compared to HttpListener

_listener = new HttpListener(); _listener.Prefixes.Add(url) _listener.Start();

// Assume a loop
var context = await _listener.GetContextAsync();

// Assume ReadInputStringAsync() is Extenstion Method on HttpListenerRequest
var input = await context.Request.ReadInputStringAsync();

// Assume WriteOutputStringAsync() is Extenstion Method on HttpListenerResponse
await context.Response.WriteOutputStringAsync("Hello World");

I don’t want know anything about a WebHost , IApplication builder, Configure Method or any other tricks. I don’t need any fluent API's or any shortcuts. I want the ability to have total control over the instantiation of the Server object without any “helpers” and I want to deal with only one entry-point object. If/When I decide to step into the server code I don’t want to pass-through any builder/filter abstraction or what not..

At this point I don’t care if you call it "Kestrel" or "Castle". The “branding” you give to such Listener class is meaningless to me (though IMO, you be better off with people using the term .NET Core Http Server and not “Kestrel” which doesn’t say anything) but I do want the Listener to be part of CoreFx and not as a separate project I don’t want to add any package for this fundamental facility. I don’t want this generic facility to be further developed with ASP.NET in mind.

abatishchev commented 7 years ago

but I do want the Listener to be part of CoreFx and not as a separate project I don’t want to add any package for this fundamental facility

Why not? Separate package can evolve, be deployed and bug-fixed faster. So you won't need to wait for the next release of Core for some enhancements to be shipped. Your idea is same as classic .NET Framework with its well-known issues and flaws, so well-known that Core uses the opposite approach. A preference to have 1 rather than 5 packages is just a preference and basically a prejudice. Also if I build an offline processing library, I don't care about HTTP at all, it's not a 'fundamental facility' for me, it bloats the base package, it must be separate.

clrjunkie commented 7 years ago

Separate package can evolve, be deployed and bug-fixed faster. So you won't need the next release of Core for some enhancements to be shipped.

Old thinking, false assumption. Foundational components don't evolve rapidly, bugs can appear anywhere in the FW and nowadays when you encounter a bug you don't wait for anyone. If you can fix it yourself you compile from source.

it bloats the base package, it must be separate.

Are you serious?

abatishchev commented 7 years ago

If you can fix it yourself you compile from source.

Then you don't need anything listed above. Just do it yourself and "compile from source", huh?

Are you serious?

Yes, I need neither HTTP nor UI nor anything else in my 'pure' C# (0 dependencies, just C#, LINQ and Expression Trees) data processing library.

poke commented 7 years ago

I do want the Listener to be part of CoreFx and not as a separate project

A big point about .NET Core was its modular approach, so you only get what you actually want. Arbitrarily putting some features into its very core just because you personally believe it’s a fundamental functionality is really against the point and ignoring all other use cases.

I don’t want this generic facility to be further developed with ASP.NET in mind.

ASP.NET Core is the driver behind this. We would be nowhere near the performance we have right now with Kestrel if it wasn’t a separate server built as part of ASP.NET Core. Yes, Kestrel is coupled in a way to ASP.NET Core’s design foundations, but again, this just just driving both projects along.

Thanks for providing the snippet! This precisely shows what I don't want to see.

As Ben wrote, you can easily wrap this into a different library, call it whatever, and ensure that your expected public API works that way. There is no reason why this shouldn’t be possible, or why this couldn’t be actually published as a package sometime. You do not care about the internals of HttpListener, so why do you suddenly care with Kestrel?

And no, I certainly don’t agree that a generic name “.NET Core HTTP Server” would be better than Kestrel. By now, Kestrel is kind of an established name for a lightweight server offering a high performance. Hiding that behind a generic name which means literally nothing and could be anything does not serve anything but instead will just lead to confusion when people find different .NET Core HTTP servers which do not happen to be that one.

I generally don’t get what you are actually trying to achieve here. To me it seems like you are just fighting over a name and a public API.

clrjunkie commented 7 years ago

@abatishchev

Then you don't need anything listed above. Just do it yourself and "compile from source", huh?

Don't understand your implied conclusion.

Yes, I need neither HTTP nor UI nor anything else in my 'pure' C# (0 dependencies, just C#, LINQ and Expression Trees) data processing library.

If your developing a library you are not bloating your code, you are referencing. Are you concerned about Http Client being part of corefx as well?

@poke

A big point about .NET Core was its modular approach, so you only get what you actually want. Arbitrarily putting some features into its very core just because you personally believe it’s a fundamental functionality is really against the point and ignoring all other use cases.

C'mon, I gave an example of how Go includes an Http Server as part of it's standard library and your turning it specifically personal... I'll leave MS PM's to do the proper research with respect to other languages.

ASP.NET Core is the driver behind this. We would be nowhere near the performance we have right now with Kestrel if it wasn’t a separate server built as part of ASP.NET Core. Yes, Kestrel is coupled in a way to ASP.NET Core’s design foundations, but again, this just just driving both projects along.

I don't understand what you mean by "driver" or how this all relevant to discussion which is about an independent Http Listener. Sound more like your defending ASP.NET for whatever reason, well guess what I'm not attacking it I just want something independent of it.

You do not care about the internals of HttpListener, so why do you suddenly care with Kestrel?

Who said I don't care about the internals..@#! I specifically mentioned that I may debug it.

And no, I certainly don’t agree that a generic name “.NET Core HTTP Server” would be better than Kestrel. By now, Kestrel is kind of an established name for a lightweight server offering a high performance.

Said who?

Hiding that behind a generic name which means literally nothing and could be anything does not serve anything but instead will just lead to confusion when people find different .NET Core HTTP servers

Means a lot! .NET Core Http Server

I generally don’t get what you are actually trying to achieve here. To me it seems like you are just fighting over a name and a public API.

I suggest you experiment a bit with HttpListener on the full .NET Framework first..

clrjunkie commented 7 years ago

To all of you who rushed to press the thumbs down button, I say you haven’t thoroughly considered the implications and your vote is against your best interest.

Full .NET Framework deployment and patching issues have nothing to do with the inclusion of Http Server support in the standard class library and are totally orthogonal concerns.

One of the main pain points with the Full .NET FX is that depending whether you have an SSD drive or not, it takes anywhere between 10 – 30 min to uninstall and install a new version, not to mention patches and that's all due to “Installer Suger”. For .NET Core the most simple and straight forward approach would be delivering it as simple ZIP file where “Installing” is a simple matter of unzipping to directory on the path and uninstalling would be deleting the directory! So installing an update for the entire shared .NET Core should be no more than seconds.

Furthermore, while the clean separation of core classes into separate projects has tremendous value from a software maintenance standpoint, it has many negatives when those are mapped to distinct packages, especially from a testing standpoint. Given some of the responses here I suspect a fundamental misconception that a shared core monolith package (which internally is project separated) is a bad thing. You are wrong. When you have one package you don’t need to deal with individual assemblies version hell and the package is tested as a whole. This has tremendous value with respect to the overall stability of the base the framework. Furthermore, I’m pretty sure you all know that Kestrel is dependent on many dozens of CoreFX based classes so now every time a new Kestrel pops-up the team needs to test it with all past CoreFX versions or specify a particular set of assembly versions for a release? IMO that’s insane !

.NET Core is specificity targeted toward Server applications which in practice is synonym with Http based applications. In a "Network world" / “cloud first” era, built-in Http support in standard libraries is becoming the norm. Heck, even NodeJS has a built-in Http Listener as part of the node binary which all NodeJS Web Frameworks depend on. What more evidence do you need?

roji commented 7 years ago

@clrjunkie without agreeing or disagreeing with anything you say, I think it would be a good idea to separate what you're saying into two subjects:

  1. Whether .NET Core should have a monolithic/expansive/single-package BCL, much like .NET Framework, where additional functionality (such as an HTTP server) would be added, or a more modular nuget-based approach where additional functionality is added as additional nugets. While this can be an interesting discussion, it does seem like that ship has sailed to a large extent, and that .NET Core has taken the more modular nuget-based approach. Example include nuget packages such as Microsoft.Extensions.Logging (whose equivalent in Java is in the core), Microsoft.Extensions.Configuration and others. In any case, it seems this question is out of scope for this issue specifically.

  2. Whether an HTTP server should exist separately from ASP.NET (and how it should be branded).

The above two points seem completely orthogonal to me. I do have an opinion on them, but I'll just react to two things you said:

.NET Core is specificity targeted toward Server applications which in practice is synonym with Http based applications.

I'd avoid saying statements like this, there are many many developers out there using .NET Core without doing any web.

Heck, even NodeJS has a built-in Http Listener as part of the node binary which all NodeJS Web Frameworks depend on. What more evidence do you need?

The fact that NodeJS has some feature doesn't seem very compelling for .NET to do the exact same thing. Specifically, since NodeJS is so web-geared (much more than .NET) it's little surprise that it includes an HTTP server.

svick commented 7 years ago

@clrjunkie

To all of you who rushed to press the thumbs down button, I say you haven’t thoroughly considered the implications and your vote is against your best interest. … You are wrong.

I don't think telling people they are wrong or that you know better than them is going to help you convince them.

Furthermore, I’m pretty sure you all know that Kestrel is dependent on many dozens of CoreFX based classes so now every time a new Kestrel pops-up the team needs to test it with all past CoreFX versions or specify a particular set of assembly versions for a release? IMO that’s insane!

How is that different from any other .Net Standard library? If you're saying that developing a .Net Standard library is insane, then something somewhere is very wrong with the platform.

clrjunkie commented 7 years ago

@roji

I think it would be a good idea to separate what you're saying into two subjects:

This is not something I started but the votes appear to be about this issue specifically.

I'd avoid saying statements like this, there are many many developers out there using .NET Core without doing any web.

The .NET Core story is a "Server story" from its first commit. Until MS states otherwise that's the Story (not your own interpretation of the mission statement). Furthermore, I suggest you avoid using the word "many" without any backing.

The fact that NodeJS has some feature doesn't seem very compelling for .NET to do the exact same thing.

So as Go and probably many others, which are on the rise and have captured high profile services.

clrjunkie commented 7 years ago

@svick

I don't think telling people they are wrong or that you know better than them is going to help you convince them.

I don't think pressing a thumb down button without providing an explanation or discussing the matter contributes to anything.

How is that different from any other .Net Standard library? If you're saying that developing a .Net Standard library is insane, then something somewhere is very wrong with the platform.

I didn't say that. Please read my post again.

roji commented 7 years ago

The .NET Core story is a "Server story" from its first commit. Until MS states otherwise that's the Story (not your own interpretation of the mission statement).

Server != HTTP Server

Furthermore, I suggest you avoid using the word "many" without any backing.

It's definitely not less unfounded/unbacked than saying "Server applications which in practice is synonym with Http based applications".

@karelz, in light of this discussion I think there's a case for reopening this issue to discuss all the points you originally posted... This issue doesn't really make any sense for that anymore.

karelz commented 7 years ago

@roji

in light of this discussion I think there's a case for reopening this issue to discuss all the points you originally posted... This issue doesn't really make any sense for that anymore.

I am not sure I understand what you mean: This issue is open and the discussion is happening. Do you expect more / something different? (Not pushing back, just trying to clarify)

clrjunkie commented 7 years ago

@roji

Server != HTTP Server

You right. the story is ASP.NET!

Tell me, what part of the Techempower benchmark (which the team so loves) measures, SMTP, H323?

Plain Text Http!.

roji commented 7 years ago

in light of this discussion I think there's a case for reopening this issue to discuss all the points you originally posted... This issue doesn't really make any sense for that anymore.

I am not sure I understand what you mean: This issue is open and the discussion is happening. Do you expect more / something different? (Not pushing back, just trying to clarify)

@karelz just that this issue seems to have turned to an exclusive (and heated) discussion of having an HTTP server separately from ASP.NET (and possibly the modular vs. monolithic nature of .NET Core...!), whereas that was just one point in your original list. There are many other important points there that need to be discussed, but this issue no longer seems like a good place for that...

karelz commented 7 years ago

It's a good point that it might be more productive overall to take the server shipping discussion offline into another issue and come back with its result once it settles.

Voting: If you agree with splitting the server shipping discussion into another issue, please upvote my comment. If you disagree, please downvote my comment.

clrjunkie commented 7 years ago

@karelz

It's a good point that it might be more productive overall to take the server shipping discussion offline into another issue and come back with its result once it settles.

I disagree. There are many other topics on list and in IMO the Http Server topic should not be discussed outside the "Network Stack Roadmap". Any other networking topic can also "dominate" this issue. If you decide to separate the topics then I suggest opening an issue for each one having a single design document file incorporating all decisions, otherwise I see no reason to move the Http Server discussion specifically.

jnm2 commented 7 years ago

I don't think pressing a thumb down button without providing an explanation or discussing the matter contributes to anything.

I hear this a lot from people, and never about upvotes, only downvotes. I think they're equivalent. It does contribute information about consensus without cluttering the thread with redundant and possibly tedious commenting, if someone already made the point.

clrjunkie commented 7 years ago

@jnm2

I hear this a lot from people, and never about upvotes, only downvotes

If you hear this a lot maybe there is merit and things are not that symmetric :)

Personally, I find upvotes kinda of a 'yay' and in any case I'm now not particular fond of either unless used specifically for survey. Anyway, IMO neither help developing a conversation in any meaningful way.

jnm2 commented 7 years ago

Anyway, IMO neither help developing a conversation in any meaningful way.

It gives a gauge of what the community wants. It was designed to be a survey and it's good at being that.

Drawaes commented 7 years ago

@karelz @clrjunkie I think it should be split off, then those that want to discuss it in depth can feel free. I have hesitated in jumping in on this specific issue so as to not clutter the roadmap issue. To answer

I disagree. There are many other topics on list and in IMO the Http Server topic should not be discussed outside the "Network Stack Roadmap". Any other networking topic can also "dominate" this issue.

I would say to that, if any other issue becomes a "hot topic" or one that requires further indepth debate then, yes it too should be moved. For instance HttpClient design/behavior has already been moved. This stops a 10k comment issue that no one will ever read all of.

clrjunkie commented 7 years ago

Coming to think of it, I have nothing more to say on the subject. However you continue discussing this is fine by me.

benaadams commented 7 years ago

Look how much abstraction one needs to be aware of compared to HttpListener ...

@clrjunkie I've written a rough HttpListener cross-platform Core 2.0 library that only uses the libraries that are installed with dotnet in the pre-jitted runtime store; and using it looks like this (sample included):

public static async Task MainAsync()
{
    var listener = new HttpListener();
    listener.Prefixes.Add("http://localhost:5000");
    listener.Start();

    while (true)
    {
        using (var context = await listener.GetContextAsync())
        {
            var input = await context.Request.ReadInputStringAsync();

            await context.Response.WriteOutputStringAsync("Hello World");
        }
    }
}

Its based on top of ASP.NET Core; but as its included with dotnet, to all intents and purposes is part of .NET Core's "standard" library; even if it isn't part of .NET Standard (i.e it is an optional reference on top of .NET Standard 2.0).

For .NET Core the most simple and straight forward approach would be delivering it...

This is already the case for .NET Core 2 as you can see from the other downloads page; and this includes ASP.NET Core.

When you have one package you don’t need to deal with individual assemblies version hell and the package is tested as a whole

And ASP.NET Core provides a single package reference Microsoft.AspNetCore.All which covers the full .NET Core stack including .NET Standard 2.0 and ASP.NET Core; if you don't want to worry about what individual packages you need to install and you want to use .NET Core's installed pre-jitted runtime store - and all the versions of this package will have been tested together and the correct matching versions.

even NodeJS has a built-in Http Listener ...

And you need to reference it with requires; just like bringing in ASP.NET Core; same with golang. ASP.NET Core is .NET Core's standard http server.

IMO this whole [HttpListener] “compatibility play” approach is fundamentally flawed.

On Windows HttpListener is not a server in its own right; it is a wrapper on the Windows HTTP Server Version 2.0 API also commonly referred to as http.sys

HttpListener directly would except an equivalent in terms of stability, performance and feature set on Unix

Which is a very hard ask as; as stated above HttpListener is a wrapper over a Window's component; its not .NET code; so it can't be "ported" to Linux in that sense.

As @davidfowl has mentioned the ASP.NET Core servers are being further decomposed and their core networking components using Span and Buffer are being pushed back down into corefx, so they can be reused for other things; but fundamentally ASP.NET provides the standard http server for .NET Core; its what its lowest level is build on.

I bet 99.9% of the people who used ASP.NET.CORE till now speak the “MVC Action, IoC, etc” parlance.

And unlike classic ASP.NET/System.Web; ASP.NET Core has been well layered so you only need to use as much of the stack as you need. If you just need the Http Sever you can use just that and I have several projects when only use that much of the stack; I have others which go all the way and also use MVC model binding, web api and Razor.

Also ASP.NET Core has been built with a very open design, so any component in it can be exchanged for your own or a third party's as long as it conforms to the interface contract. You can even switch out the http parsing for an alternate and still use the same server.

For example it has two exchangeable HTTP servers which are KestrelHttpServer and HttpSysServer (which uses the same Windows api as HttpListener so only works on Windows) and no doubt their will be other interchangeable servers from other providers in the future. Much like node and golang.

The method to choose which server component you want to use is very flexible and you can do it via environment var, cli parameter, config file setting, runtime switch or some other method that suits - could even stick the choice in an XML file and download it from the network at startup - if that was your thing.

ASP.NET Core really isn't as bloated as classic ASP.NET/System.Web which provided an unbroken compatibility chain to classic ASP and carrying with it 21 years of api history and organic design. Its a lean fresh start that builds upon all that knowledge; while skipping out the mistakes and legacy.

clrjunkie commented 7 years ago

@benaadams

I've written a rough HttpListener cross-platform Core 2.0 library that only uses the libraries that are installed with dotnet in the pre-jitted runtime store;

What you wrote is a start of an HttpListener to Kestrel Adapter class which I’ve been advocating for, over and over and over. (Apparently it didn’t take that long to code...) I do have some concerns about the implementation but that's not something I’d want to get into now.

Saying that, I repeat: I don’t want any ASP.NET hosting magic under the hood. I don’t want any WebHost/Service/Configuration/Middleware abstractions at the core of the implementation, I want to have an explicit view of who calls during server instantiation and configuration. Same as WebListener (a.k.a HttpListener) doesn’t include any of these as part of it's API surface (e.g There is no need for a WebHost or what not to run WebListener) All these "Hosting" abstraction should be part of ASP.NET and not the HTTP Server. Don’t impose on me to learn and deal with things that I don’t need or want just because they benefit ASP.NET architecture in some way.

This is already the case for .NET Core 2 as you can see from the other downloads page; and this includes ASP.NET Core.

Great! I didn’t notice that. This shows that lengthy installs/updates to the entire .NET Core is a non-issue.

And ASP.NET Core provides a single package reference Microsoft.AspNetCore.All which covers the full .NET Core stack including .NET Standard 2.0 and ASP.NET Core; if you don't want to worry about what individual packages you need to install and you want to use .NET Core's installed pre-jitted runtime store - and all the versions of this package will have been tested together and the correct matching versions.

Again, what does it mean that they both come together? Why then is ASP.NET not part of .NET Standard? (Like ASP.NET is part of Full .NET FX)

Anyway, I advocate for an HTTP Server that’s independent of ASP.NET – meaning HTTP Server without any ASP.NET code what so ever, and that should be part of CoreFX like HttpClient is.

And you need to reference it with requires; just like bringing in ASP.NET Core; same with golang. ASP.NET Core is .NET Core's standard http server.

I'm puzzled by your whole point. What does referencing a “namespace” have to do with this issue !@?

In Go you also need to import the “fmt” package to print to the console, so what? what are you saying?

Fact is that any NodeJS binary has HTTP Server code in it (NGINX Http parsing code actually). You don’t need any external dependency to write an Http handler. Also every Go install has an HTTP Server as part of it’s standard library.

On Windows HttpListener is not a server in its own right; it is a wrapper on the Windows HTTP Server Version 2.0 API also commonly referred to as http.sys

Oh please, have you read the post I referenced above https://github.com/dotnet/corefx/issues/13610 Give me a bit more credit… I have “some” experience with Http Api.

Which is a very hard ask as; as stated above HttpListener is a wrapper over a Window's component; its not .NET code; so it can't be "ported" to Linux in that sense.

You either missed the point or I should have been more descriptive. I asked for a an HttpListener API that has ONE HTTP Server Implementation across platforms. I didn’t ask to port Http.sys to Linux. I don’t see any reason why anyone would insist that the managed HttpListener in .NET Core specifically be implemented using a "kernel mode driver". Yes, there is HttpCfg.exe for configuring Http.sys on windows but I'm very very skeptical if .NET developers rely on it's indirect relationship to HttpListener or depend on a configuration set by it and in any case they won’t get it on Linux or any other platform.

but fundamentally ASP.NET provides the standard http server for .NET Core; its what its lowest level is build on.

I pretty confident that factoring out all ASP.NET hosting abstractions so to have a true independent low-level HTTP Server is something that can take place within the refactoring effort.

And unlike classic ASP.NET/System.Web; ASP.NET Core has been well layered so you only need to use as much of the stack as you need…

Sorry Ben, this all reads to me like technical marketing. I'm not concerned about ASP.NET overall architecture. WebListener is a living proof you don't need any ASP.NET hosting/configuring abstractions to "new up" an HTTP Server. Quite frankly, I’m surprised you are not backing this. If I recall correctly you once stated you where using Http Listener in your system prior to Kestrel.

clrjunkie commented 7 years ago

Regarding my note on the "insignificance" of httpcfg (or netsh) I should have specifically stated "from a programmability standpoint". Obviously those are needed to configure SSL for Http API on windows, but then again that is more of an administrative tasks external to the code being developed and an alternative process can easily be documented.

svick commented 7 years ago

@clrjunkie

I don’t want any ASP.NET hosting magic under the hood. […] Same as WebListener (a.k.a HttpListener) doesn’t include any of these as part of it's API surface […] Don’t impose on me to learn and deal with things that I don’t need or want just because they benefit ASP.NET architecture in some way.

I don't understand. If the "hosting magic" was under the hood of the wrapper, it wouldn't be part of the API surface. So why would you have to learn and deal with it?

Why then is ASP.NET not part of .NET Standard?

One reason is that it would mean that every platform that implements .Net Standard would have to include ASP.NET Core, including UWP, Xamarin and .Net Framework. Even if you think the only reason for .Net Core to exist is ASP.NET Core, the same cannot be said about .Net Standard.

clrjunkie commented 7 years ago

@svick

I don't understand. If the "hosting magic" was under the hood of the wrapper, it wouldn't be part of the API surface. So why would you have to learn and deal with it?

Because I'm not into buying any black box.

I want the ability to maintain it, improve it, possibly contribute to it, without having to deal with what I believe are unnecessary abstractions at this layer.

One reason is that it would mean that every platform that implements .Net Standard would have to include ASP.NET Core, including UWP, Xamarin and .Net Framework.

For this topic specifically, I would agree upfront that a discussion should be held as a separate issue, but in short: So what? Why would you want to limit the possibility of incorporating ASP.NET in client applications? Having ASP.NET layered on top of a general purpose socket based HTTP Server virtually makes it available on any possible device and may allow developing applications you might not even envision.

poke commented 7 years ago

Because I'm not into buying any black box.

I want the ability to maintain it, improve it, possibly contribute to it, without having to deal with what I believe are unnecessary abstractions at this layer.

But you did buy the HttpListener before which used a very-black-box Windows API which you couldn’t look into?

Why would you want to limit the possibility of incorporating ASP.NET in client applications?

It is not limited. That’s why it’s a netstandard library. It can be used in any environment that implements netstandard. That’s the whole point. Making ASP.NET part of netstandard would mean that every implementation of netstandard would have to ship with its own included version of ASP.NET Core. That would slow down development a lot.

The big benefit about netstandard libraries is that we do not need to touch all these environment all the time to expand them. We have separate packages for everything we want to add.

svick commented 7 years ago

@poke

That’s why it’s a netstandard library. It can be used in any environment that implements netstandard.

I believe it's not actually that simple. Most of ASP.NET Core are indeed just .Net Standard libraries, including Kestrel. But Kestrel also depends on libuv, which is a native library and its stable version does not have a version for ARM on Linux, for example (there is a preview version that does, though).

But for any platform that has a working version of libuv, ASP.NET Core, including Kestrel, should indeed just work.

clrjunkie commented 7 years ago

@poke

But you did buy the HttpListener before which used a very-black-box Windows API which you couldn’t look into?

Expected :)

Eureka! No More to closed source HTTP Server implementation!

adamhathcock commented 7 years ago

@clrjunkie

Expected :)

Eureka! No More!

You should have because I almost kept replying.

After reading this, I have no idea what you’re advocating and I feel trolled by kept up with this issue. I subscribed wanting to keep tabs with progress but instead got an aimless series of disagreements.

Please, make a new issue.

Drawaes commented 7 years ago

It should be in a new issue and here is why, I actually thing the initial suggestion is a good one, however it shouldn't be done yet. ASP.NET and .NET CORE are moving fast, Span, Pipelines (if they go main stream) HTTPS, Sockets Everywhere (and transports) HTTP2 parsing etc are all in flux and being actively worked on. If and when those pieces come together and stabilise there is then scope to abstract them out into the common library (as @davidfowl mentioned) and at that point I would love to see an "http listener", and we would have all the building blocks to do it pretty simply. That is why this should be a separate issue, with maybe a list of the bits needed (or steps) to get there and track all those individual issues.

clrjunkie commented 7 years ago

I would strongly be in favor of having a detailed design document and an accompanying issue dedicated towards discussing the implementation of a new socket based HttpListener. The only reason I made my case here is to push for establishing the requirement.

I suggest that such "design issue" start by calling for community feedback on HttpListener's API surface to discuss what limitations exist (if any) as well as proposed changes/enhancements.

benaadams commented 6 years ago

I would strongly be in favor of having a detailed design document and an accompanying issue dedicated towards discussing the implementation of a new socket based HttpListener.

HttpListener is an unclear design. Is the context from GetContextAsync dispatchable, or do you have to finish with the current context before requesting the next one so is it single threaded?

Also a reimplementation of HttpListener so it was shared managed code cross platform would break the compat bar as it would likely deviate from any quirks in Window's HTTP api (especially since it would no longer be able to port and path share as http.sys can with IIS and other applications)

So it is essentially a request for an entirely new web server and api to be built from scratch; that would not maintain some of the benefits of HttpListener.

Will limit reply, so as not to spam group; but is already a new server and its called WebHost (which can be implemented via Kestrel, HttpSysServer (previously WebListener), or 3rd party.) All open source so you debug and contribute; also outside of .NET Standard so you can use it on platforms that implement .NET Standard without having to wait for them to integrate the next version of .NET Standard to use it - which would be the case if they were part of it.

clrjunkie commented 6 years ago

HttpListener is an unclear design.

To clarify, consider "HTTP Listener" (space is significant) or "HTTP Server".

Point being an HTTP Server API independent of ASP.NET

So to picture this, one possible layering might be:

[System.Net.HttpListener]* [System.Net.HttpServer] [System.Net.Sockets]

* For compact . Connection state management/Protocol Parsing/Request-response API surface No ASP.NET abstractions.

I would expect existing Kestrel code be easily transferable to a System.Net.HttpServer to cover the implementation requirements for HTTP 1.1, so in that sense I have no idea what's the problem.. As I wrote several times I currently have no specific concern about Kestrel internal implementation (which is something I will definitely look into if it's API surface will be refactored to exclude ASP.NET abstractions)

Is the context from GetContextAsync dispatchable, or do you have to finish with the current context before requesting the next one so is it single threaded?

Good question. I would be happy to share my experience if/when a proper design discussion takes place.

Also a reimplementation of HttpListener so it was shared managed code cross platform would break the compat bar as it would likely deviate from any quirks in Window's HTTP api (especially since it would no longer be able to port and path share as http.sys can with IIS and other applications)

With respect to port share, your technically right. Now go search on the Internet and tell me what your gut feeling tells you about the popularity of this feature in the .NET world.

Anyway, I think you underestimate the requirement for compatibility across platforms, port sharing is not provided on any other platform by the current "Compatibility Play" approach.

IMO it all boils down into making the right trade-offs and I argue that an HttpListener implementation which is based on a shared code base is the better choice for numerous reason, the current compatibility play will be a bone stuck in the team's/customer's throat for many years.

My gut feeling tells me that port sharing scenarios are rather "exotic" and for those specific cases where it's absolutely a must I bet an IIS Proxy solution will due. Then again, that's my hunch.

but is already a new server and its called WebHost (which can be implemented via Kestrel, HttpSysServer (previously WebListener), or 3rd party.)

WebHost is an abstraction for a "Web Server" for ASP.NET users! While it may have it's place for ASP.NET (I have no opinion about this) from a platform api standpoint I say this is "Architecture Astronaut".

HttpSysServer/WebListener/HttpListener (rename it as you like) IS a concrete Web Server which has a straight forward API HTTP Server API. There is zero dependency on ASP.NET/WebHost for using using it.

What/Where exactly is the Kestrel API Surface? How can one use it without the WebHost abstraction?

clrjunkie commented 6 years ago

@benaadams

you can use it on platforms that implement .NET Standard without having to wait for them to integrate the next version of .NET Standard to use it - which would be the case if they were part of it.

I don't buy this counter argument. Both NodeJS and GO require a recompile of/with, their respective runtime in case of a bug in their built-in "HTTP Listener" and both are used by mega services ! Show evidence anyone is complaining about this, or better, go look at their release notes and show that issues with their "HTTP Listener" are a primary motivation for a release compared to other issues.

svick commented 6 years ago

@clrjunkie As far as I know, NodeJS and Go don't have a standard library specification (they have just the standard library). I think a better comparison is C++, which does have standard library specification similar to .Net Standard and multiple implementations. And there, you have to wait for your implementation to catch up to the specification.

clrjunkie commented 6 years ago

@svick

Actually, this distinction, in the context of .NET, is something that I find hard to understand. Who has the motivation to re-implement the BCL and not reuse Microsoft’s implementation. I mean what does all this .NET Standard mean outside Microsoft and why wouldn’t the .NET Core code base be shared by all “.NET implementations”?

svick commented 6 years ago

@clrjunkie You don't need to go outside Microsoft. When .Net Standard 2.0 is released later this year, as far as I know, only .Net Core and .Net Framework will support it right away. .Net Framework, UWP and Xamarin will all implement it later.

Even though they share some source code (to varying degrees), it doesn't mean they start supporting new version of .Net Standard at the same time.

And the same would likely apply to any non-MS implementation.

roji commented 6 years ago

@clrjunkie

Actually, this distinction, in the context of .NET, is something that I find hard to understand. Who has the motivation to re-implement the BCL and not reuse Microsoft’s implementation. I mean what does all this .NET Standard mean outside Microsoft and why wouldn’t the .NET Core code base be shared by all “.NET implementations”?

At some point this line of comments really makes no more sense (at least to me) - in the actual, real .NET world you have .NET standard and its various implementations, regardless of what you understand or think about it. This is not something that is going to change - at least not in the short term - and it's a waste of everyone's time to start discussing it in an issue about the .NET networking stack.

Today, in July 2017, the upcoming .NET Standard 2.0 will have no ASP.NET (or an HTTP API). Regardless of what anyone thinks about that, saying that "HTTP/web should be part of .NET Standard" basically means there's no ASP.NET anywhere until something like .NET Standard 2.1 (or 3 or whatever) comes out, which is clearly not an option.

Moving forward, I've looked the comments over and I really can't find any clear, substantial arguments of what would be gained by including HTTP/web/ASP.NET in .NET Standard (and no, saying that Golang/NodeJS do it is not an argument). One point does makes sense to me, and that's the ensured compatibility between the HTTP/web API and the rest of the BCL - since they are released as a whole testing is simpler. The flip side of this argument is of course the freedom of the HTTP/web stack to publish whenever it wants, at its own rhythm, rather than being held back by the .NET Standard release schedule. This is relevant both for bug fixes (imagine having to wait for .NET Standard 2.1 just because you've found some bug in your HTTP stack), and for features. You say that there's no evidence of Golang/NodeJS having suffered because of the HTTP/web stack being part of their core framework; maybe, I have no idea what their core framework release schedule looks like. But in any case, there's no evidence that dissociating the HTTP/web stack from the core framework has any adverse effects - your "compatibility testing" argument isn't any more substantial or backed up. At the end of the day, the HTTP/web stack is clearly a large enough to merit its own separate existence and release/versioning policy - but that's just my personal opinion.

The discussion about a separate HTTP server API that isn't related to ASP.NET (and whether WebHost can be considered that API), is IMHO the only discussion that's actually relevant to have in this issue. As I'm not familiar enough with those .NET APIs I don't really have anything to add to that discussion.

roji commented 6 years ago

@svick

You don't need to go outside Microsoft. When .Net Standard 2.0 is released later this year, as far as I know, only .Net Core will support it right away. .Net Framework, UWP and Xamarin will all implement it later.

While it's true that UWP support for .NET Standard 2.0 will come later, AFAIK .NET Framework 4.6.1 will fully support it (see https://github.com/dotnet/standard/blob/master/docs/netstandard-20/README.md).

clrjunkie commented 6 years ago

@roji Your comments here suggest to me that you have a limited understanding of the HTTP Server echo-system and the problem domain and this not the place to extend the discusion beyond what people involved in this space are already aware of. Now while everyone are free to make their point, respond, and challenge others, I ask you not to "declare" what I personaly think or not think! Furthermore, you are not a Microsoft spokes person, roadmaps plans can all be subject to change. Only MS is i the position to talk about it.

benaadams commented 6 years ago

Actually, this distinction, in the context of .NET, is something that I find hard to understand. Who has the motivation to re-implement the BCL and not reuse Microsoft’s implementation.

Microsoft have reimplemented it many times (part of the problem); Mono, Unity; Xamarin on Mac, iOS, Android, Samsung for Tizen

roji commented 6 years ago

Your comments here suggest to me that you have a limited understanding of the HTTP Server echo-system and the problem domain and this not the place to extend the discusion beyond what people involved in this space are already aware of

My last comment said almost nothing HTTP, but rather about the value of discussing the logic of .NET Standard, and about the pros and cons of bundling vs. unbundling a major component in .NET Standard. If you actually have something concrete to say (as opposed to a very vague accusation of me not knowing what I'm talking about), I'd be glad to hear it.

Now while everyone are free to make their point, respond, and challenge others, I ask you not to "declare" what I personaly think or not think!

I've reread my comment and I have no idea where I declared what you think or don't. I understand what you're saying, it's just that your arguments make little sense to me.

Furthermore, you are not a Microsoft spokes person, roadmaps plans can all be subject to change. Only MS is i the position to talk about it.

Of course. But this is a community, and you've hijacked a thread about the networking stack to vent out your rants about .NET Standard not making sense, and about how things that aren't bundled into it should be bundled into it. You're simply off-topic.

clrjunkie commented 6 years ago

@benaadams

Microsoft have reimplemented it many times (part of the problem); Mono, Unity; Xamarin on Mac, iOS, Android, Samsung for Tizen

Is'nt the .NET Core shared code base the solution?

clrjunkie commented 6 years ago

@roji

and you've hijacked a thread about the networking stack to vent out your rants about .NET Standard not making sense,

Well, excuse me ..you know, sometimes thing are closely related, if you don't see it doen't mean there is no place for it. Anyway I haven't noticed you posted anything here on any other networking topic on the list! Nobody is prohibiting you to start a new topic.What’s your problem?