Closed karelz closed 3 years ago
roji
commented
7 years ago As the tone of this discussion is rapidly deteriorating, I'd say it's time for someone from Microsoft to weigh in and decide whether there's value in discussing these proposals, i.e.:
benaadams
commented
7 years ago @> What/Where exactly is the Kestrel API Surface? How can one use it without the WebHost abstraction?
It uses the Microsoft.AspNetCore.Hosting.Server.Abstractions interfaces which are the onerous interfaces:
interface IServer : IDisposable
{
IFeatureCollection Features { get; }
Task StartAsync<TContext>(IHttpApplication<TContext> application, CancellationToken cancellationToken);
Task StopAsync(CancellationToken cancellationToken);
}IHttpApplication and TContext is either defined by you, or WebHost sets it up to be something that creates an HttpContext with Request and Response etc
interface IHttpApplication<TContext>
{
TContext CreateContext(IFeatureCollection contextFeatures);
Task ProcessRequestAsync(TContext context);
void DisposeContext(TContext context, Exception exception);
}But you can grab the components via the Features collection of IServer with the ones Kestrel implementing directly being IHttpRequestFeature, IHttpResponseFeature, IHttpUpgradeFeature, IHttpConnectionFeature, IHttpRequestLifetimeFeature, IHttpRequestIdentifierFeature, IHttpBodyControlFeature, IHttpMaxRequestBodySizeFeature, IHttpMinRequestBodyDataRateFeature, IHttpMinResponseDataRateFeature. Other things like caching, compression, SSL/TLS, Websockets, IIS integration, logging, authentication, file serving, MVC and even Routing are optional addins further up-stack.
WebHost details with reading stuff from configs; setting up a easy HttpContext api to use, adding instrumentation and diagnostics (if switched on) integrating any of the optional components you've added etc. It also sets up the "middleware pipeline"; which sounds more complex than it is; as its a set of functions that get called in order; e.g. do you want auth to run before file serving, caching to run after then routing?
The middleware part allows them to do a little more than a straight function using their constructors, where they can specify the components they are dependent on e.g. does it need to be provided the common logging method to allow unified logging? However you don't need to specify any middleware (such as authentication, compression, logging etc) and it won't set them up.
How much extra does WebHost and other middleware add? Well the plain text Techempower tests are set up with the MVC tests also, so have the whole shebang set up and configured in the same app and Kestrel still does pretty well.
If you want to do it all without WebHost you can rummage around in Kestrel's tests and go through the steps of setting something up that's very similar to WebHost anyway.
What I and others have pointed out
What I'm interpreting from your responses is
System.*
clrjunkie
commented
7 years ago @benaadams Apparently you're holding a parallel discussion on the topic via twitter, therefore I suggest you reach out to your followers for future feedback on this topic as I will not be commenting anymore.
@davidfowl You’re right, my arguments are not “Technical” per-se. They are “Architectural”. But thanks for sharing your thoughts.. saved me precious time!!
Good Luck!
clrjunkie
commented
7 years ago "The Server APIs project will provide core capabilities in areas such as networking and security so Swift programs no longer need to frequently rely on platform-specific C libraries to provide this functionality. As a result, developers will be able to create frameworks and server applications using pure-Swift code, without the need to also have systems programming skills and knowledge of multiple platforms."
"HTTP and WebSockets: Provide low level HTTP parsing, including HTTP, HTTP/2 and WebSocket support making it possible, in conjunction with the security and networking APIs, to create secure HTTP and WebSocket servers."
mconnew
commented
7 years ago @clrjunkie, there are two reasons I'm aware of why HTTP.SYS (and HttpListener by extension) exists. First is so that different applications on a machine can all service http requests on the same port(s). The second is to offload Windows authentication to HTTP.SYS. As it's a driver, it runs with system privileges. This is important when using Kerberos authentication, as only when running with system privileges can a server use the default HTTP SPN. If you need either of those, using a user mode HTTP stack won't work for you and you need to be using HTTP.SYS. .Net Core is cross platform which means it can't use HTTP.SYS on non Windows platforms. HTTP.SYS also has capabilities that other HTTP stacks won't have, and those stacks will have features that HTTP.SYS doesn't have. This means you can't design an api into netstandard/netcore which exposes those HTTP.SYS specifics. If I were to build an HTTP server on top of swift, it won't be able to authenticate Kerberos users unless I run the whole process as a system user. Sure, you could create a class written in swift which wraps the HTTP.SYS api's and provides this capability, but then you aren't using the built in server.
clrjunkie
commented
7 years ago @mconnew Interesting point about Kerberos, it's been a while since I've used it.. Nevertheless I think the initial question with respect to that is whether Kerberos Authentication is a .NET Standard requirement?
As far as I recall there is a Kerberos Authentication in HttpWebRequest as well. How is that supported in .NET Standard? How would that be supported by a .NET Standard implementation on Linux?
https://docs.microsoft.com/en-us/dotnet/api/system.net.httplistener?view=netstandard-2.0
Also, since work on HttpListener is fairly new I wonder if the decision to add it was in response to customers asking for the Kerberos support..
BTW, Isn't NegotiateStream supposed to handle this in user mode?
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363764(v=vs.85).aspx
With respect to port share, as I wrote previously it's all about making trade-offs and I believe for those scenarios an IIS Proxy setup is a viable workaround, heck maybe even for Kerberos!!
I would agree in general that API compat is King & Queen.
That is of course until “Winter comes” !
About Transactional NTFS
“[Microsoft strongly recommends developers utilize alternative means to achieve your application’s needs. Many scenarios that TxF was developed for can be achieved through simpler and more readily available techniques. Furthermore, TxF may not be available in future versions of Microsoft Windows. For more information, and alternatives to TxF, please see Alternatives to using Transactional NTFS.]”
https://msdn.microsoft.com/en-us/library/windows/desktop/aa363764(v=vs.85).aspx
So Assuming Kerberos/Port Share an absolute MUST API, I would say .NET FULL FX - ONLY.
mconnew
commented
7 years ago @clrjunkie, when using Kerberos authentication, both ends have an identity. The client identity is usually your user account which is the ambient security context when using HttpWebRequest. When HttpWebRequest requests a Kerberos ticket to access a service, it has to provide the identity of the remote system it is communicating with. The ticket it receives is tied to the remote service. When you send that ticket to the remote service, only a process with permissions to the identity used to obtain the ticket can verify it. The default spn (HTTP/hostname in the case of HTTP) isn't usable except by a system account (System or Network Service). This means a user process can't verify a Kerberos token issued for HTTP/hostname as user accounts don't have permission to that SPN. HTTP.SYS allows a user process a way to authenticate via Kerberos as it can do the authentication on your behalf as it's a kernel mode driver. This means swift and any other user mode HTTP implementation can not authenticate using Kerberos unless your process is running as a system account, which is a bad idea. So basically, if you want to authenticate using Kerberos, you either have to have a process running in a system account or use HTTP.SYS. NegotiateStream allows the client to specify an arbitrary SPN for the remote end, which can even be a UPN. So if your service is running as the non-privileged user foo@contoso.com, the client end provides the remote identity UPN of foo@contoso.com to NegotiateStream which in turn obtains a ticket for your current identity which can be verified by foo@contoso.com so everything works.
Drawaes
commented
7 years ago Isn't it true though that is the default spn permissions you can of course delegate spn permissions to any account. If that isn't the case I am not sure how 50% of my apps work :)
mconnew
commented
7 years ago @Drawaes, I glossed over a few details. The machine account is identified as HOST/hostname, the HTTP service class SPN by default is aliased to HOST/hostname. You can use the setspn commandline tool to change which account the HTTP service class is aliased to. So you can set HTTP/hostname to be an alias of foo@contoso.com which means when the client requests a ticket for HTTP/hostname, it is actually granted a ticket for foo@contoso.com which can then be used by the user process. But this means now only that user is able to authenticate Kerberos authenticated HTTP requests to HTTP/hostname. IIS won't be able to authenticate any requests unless you change the app pool identity to use the same user. If you didn't use setspn, then you might have your app configured to use "Windows Authentication", which means Negotiate. Negotiate goes through (negotiates) a list of authentication mechanisms, which in practice means try Kerberos, if that doesn't work, try NTLM. Often people are falling back to using NTLM without knowing.
Drawaes
commented
7 years ago In practice I never use the Http/hostname anyway :) all my apps are behind loadbalancers with a DNS per app anyway.. but yeah it can be an issue if you stop other apps using it. Anyway this isn't my fight as we are moving away from Kerb apart from Client -> frontend web boxes and moving to server/client TLS certs for server to server.
clrjunkie
commented
7 years ago @mconnew
This means swift and any other user mode HTTP implementation can not authenticate using Kerberos unless your process is running as a system account, which is a bad idea.
That’s not entirely accurate. While I totally agree that running a Server process as “System” is a big NO NO, running it as “Network Service” allows Kerberos communication (as you mentioned) and is a legitimate setup (I would argue that “Network Service” was invented for that purpose). There is a BIG difference between a “System” account and “Network Service” account where the latter has no Administrator privileges. Also, running a process under “Network Service” is a popular configuration choice for many Microsoft Server products, see SQL Database Engine:
Having said that, I would suggest to exclude Swift from this particular discussion so as to not steer away from the central question which is: Assuming we go along with a new implementation for HttpListener, how would we maintain backwards compatibility with respect to Kerberos Authentication.
The way I see it the primary issue is that currently HTTP.SYS calls certain Windows API’s to have Windows do the Kerberos authentication where in a new managed implementation those API's would need to be called directly. So assuming Kerberos Authentication is NOT a .NET Standard requirement (e.g It's a Windows only requirement) and given we really don’t know how many/if any .NET HttpListener application use Kerberos, it may be worth to consider an initial release (.NET Core Only) which does not support it and implement it if there is customer push-back. With respect to sharing the implementation with FULL .NET FX, here it might make sense to have both implementations and allow users to select either one via an App.Config configuration setting.
Drawaes
commented
7 years ago The windows Api's in question are Schannel the same as used today for TLS on windows in corefx. They are a piece of cake to implement and I have done so to make Kerberos work with "raw" kestrel. The interface is also very similar to GASPI (I think that is the acronym but on a beach right now so not going to bother looking it up) on *nix and I have support for Kerberos on linux. It's not a windows specific technology.
clrjunkie
commented
7 years ago Great!!
clrjunkie
commented
7 years ago Here is an example of what I would like to see as part of the .NET Core BCL, but based on a high-performance user-mode HttpListener.
clrjunkie
commented
7 years ago After reading the issue again, I suspect a terminology trap which admittedly I fell into on one or more occasions. I wish to clarify. Windows uses the term “HTTP Server Api” to describe something quite different from what’s commonly referred to by Go, NodeJS and others, hence the trap. What Windows refers to as “HTTP Server Api” are HTTP Request / Response Api's that are not tied to any particular threading pattern where others use the term “HTTP Server” to describe an Api that dispatches incoming HTTP Requests to users code. In familiar terms they offer “Windows HTTP Server Api” style functionality + thread management for dispatching requests to user handlers. So for lack of a better term I call the Go/NodeJS HTTP Server offering: “HTTP Server Handler”.
In .NET the "HttpListener" class simply exposes the “Windows HTTP Server Api” in OOP form, so in essence "HttpListener" IS the "Windows HTTP Server Api” equivalent in .NET
So to summarize: I propose that .NET Core BCL include an “HTTP Server Handler” built on a newly implemented “HttpListener” having the same "Windows HTTP Server API" semantics without any ASP.NET abstractions. I see no reason why this new “HttpListener” not be based on Kestrel internal implementation. However, if good arguments are made against being bound to "HttpListener" existing Api surface I would argue for a new “.NET HTTP Server Api” (also based on Kestrel) to use as the underlying shared-code base for the “Legacy HttpListener” and for a new “HTTP Server Handler”.
Furthermore, I would suggest discussing HttpListener’s Api continued viability for different use cases using the project I linked to. I know beauty is in the eyes of the beholder and I’m not trying to force personal taste on anyone or promote my own abstraction but I do think it can serve as solid reference during a technical debate on what’s good or bad about HttpListener Api surface. With respect to legalities: This is my own work period. I wrote it before knowing anything about NodeJS or Go. I do not want/need any sort of credit or implied credit and I have no problem contributing the project code to the .NET foundation as a sample showing various use-cases around HttpListener, in which case I would prefer having my alias removed from the project source.
With respect to ASP.NET/WebHost, as I mentioned before I have no opinion about that, actually it appears that at least in Java land, Web Frameworks which work with different "HTTP Server Handlers" are a common thing.
VladimirAkopyan
commented
6 years ago I get where @clrjunkie is coming from. The idea is that it should be easy to build applications that can talk to each-other, which are not primarily servers - console / gui/ whatever applications. As it stands, ASP + kestrel carry quite a lot of stuff you have to understand, like kestrel's threadpool, etc. All the guidance instructs you to use them as frameworks, which means my application is "obeys asp's rules". It's clear that @clrjunkie is a bit lost in ASP + kestrel and does not understand how to peel away layers he does not need, and it's kind of similar for me.
It's not a technical problem, it's a library vs framework and developer understanding problem. @benaadams the library you linked would be the kind of thing some folks would find helpfull.
krispenner
commented
6 years ago I'm not sure the format here, is anyone just adding their suggestions to this? In another post @davidsh suggested to post here.
If you would like to raise this as a new design issue/discussion, please contribute to dotnet/designs#9
I would like to comment on #23422 which is now closed - not sure if you will see my comment on there.
Using WebClient or HttpRequest with auto follow redirects enabled throw exceptions when the redirection is considered a security downgrade such as https > http. However this seems to be more common than expected where a user requests http and gets redirected to https and then back to http (Yahoo among others). WebClient was meant to be very simple and quick to download resources from the web and it can't follow some simple redirects in .NET Core due to this restriction even when the user was not asking for https. Would it not possible to store the user's original URL requested and if the scheme was http initially then you can follow any https > http redirects that come up along the redirect pipeline as the user was never wanting https in the first place so I don't see this being a security risk. This would fix a lot of issues dev's are experiencing with this. Thoughts?
Cheers
shaggygi
commented
6 years ago Referencing https://github.com/dotnet/corefx/issues/24742 since it seems like in the same bucket.
karelz
commented
6 years ago @alexwiese QUIC fits into Emerging Technology section and was brought to our attention earlier, updating top post to list it.
Currently we are focusing on first 2 sections - Foundations (Sockets, SSL) and Web Stack (ManagedHandler). https://github.com/dotnet/corefx/issues/24742 tracks our plans for .NET Core 2.1 milestone (via ZenHub Epics and dependencies).
Dotnet-GitSync-Bot
commented
3 years ago I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.
alexwiese
Summary
Our investment in the .NET networking space will focus in the following areas
Foundation: Sockets, SSL, DNS support.
Primary Goal: Provide near-native performance and rock-solid reliability.
Key investments:
Web Stack: HttpClient, ClientWebSocket
Primary goals: Ease of use; feature-rich; extensible; track emerging standards; match or exceed competitor's performance.
Key investments:
Tracking: https://github.com/dotnet/corefx/issues/21452
Emerging Technology
Primary goals: Demonstrate leadership in support for new protocols and capabilities
Key investments:
Maintenance Components: *WebRequest, Mail, HttpListener
Primary goals: Preserve existing customer investments Key investments: None.
Details
(1) Foundational
Key components
Characteristics
Requirements
Current Status
Windows
SocketAsyncEventArgsSocketAsyncEventArgs; this applies to Linux too)SocketAsyncEventArgsbug that had been known in community for yearsLinux
SocketAsyncEventArgsKey investments to make
Span<T>be a perf win for Sockets? Ideally, yes, but details are unclear at this point(2) Web stack
Key components
HttpClient(System.Net.Http)ClientWebSocket(System.Net.WebSockets)HttpListener, which is legacy.HttpWebRequestand related (e.g.ServicePoint), which is legacy.Characteristics
Requirements
Current Status
Windows
HttpClientandClientWebSocketbuilt on native WinHttp APIsClientWebSocketnot supported on Win7 due to missing OS supportLinux
HttpClientbuilt on libcurlClientWebSocketimplemented in managed code, with hard-wired, websocket-specific HTTP protocol handlingASP.NET
Key investments to make
Move to managed-only implementation of
HttpClientandClientWebSocketfor both Linux and Windows - see https://github.com/dotnet/corefx/issues/21452HttpClientprototype:ASP.NET
Enhance web stack to enable key customer asks and fill gaps
UseNagleAlgorithm,DnsRefreshTimeout, etc)(3) Legacy/maintenance components
Key components
Mailclient (System.Net.Mail)Http/Ftp/FileWebRequest(System.Net.Requests, System.Net.ServicePoint)HttpListener(System.Net.HttpListener)Characteristics
HttpWebRequest=>HttpClient)FtpWebRequest)Mailclient =>MailKit)Key investments to make
(4) Future/bleeding edge components
Not covered in this version.
Key Feedback
TODO (@karelz)
Credentials & Status
Author: @geoffkizer
Reviewed by CoreFX team: @stephentoub @davidsh @CIPop @Priya91 @wfurt @DavidGoll @karelz Reviewed by partner teams experts: @davidfowl (ASP.NET), @Tratcher (ASP.NET), @mikeharder (ASP.NET), @marek-safar (Mono/Xamarin), @mconnew (WCF), Windows networking experts, @tmds (RedHat), @omajid (RedHat) Reviewed by community networking experts: @benaadams, @NickCraver, @onovotny, @mgravell, @Drawaes
Reviews happened in stages in April. Feedback will be incorporated in updates to this doc. The delay between original review and publishing the roadmap now is purely lack of time to incorporate feedback into the doc & publish it.
Current status: Community review / feedback
Announcements: Twitter and CoreFX repo