search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
14.92k stars 4.64k forks source link

Runtime developer can run Networking tests without Loopback server capability (on mobile/browser platforms) #42852

Closed lewing closed 3 years ago

lewing commented 3 years ago

Some platforms don't support listening to incoming requests but do support making outgoing requests. Most of the existing http functional tests expect to be able to start up a loopback server to respond to incoming request which means the tests cannot run on the more restricted platforms.

Plan:

ghost commented 3 years ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

akoeplinger commented 3 years ago

Some of the http cert and websocket tests use an echo server hosted in Azure, presumably we could do the same for the basic http scenarios: https://github.com/dotnet/runtime/blob/6072e4d3a7a2a1493f514cdf4be75a3d56580e84/src/libraries/Common/tests/System/Net/Configuration.Http.cs#L44

wfurt commented 3 years ago

If the tests can run under platform emulator using containers may be better - like we do for stress and enterprise authentication. Dependency on external servers was problematic in the past and all the tests using them are marked as Outerloop.

stephentoub commented 3 years ago

Some of the http cert and websocket tests use an echo server hosted in Azure

Some, yes. Two problems:

  1. Going off-box results in significantly more instability of the tests, which is why most of our networking tests that do so have been relegated to outerloop, and why we've pushed to move any tests that don't require going remote to not do so. Even so, there have been discussions in the past about making it configurable.
  2. Lots of test use shared in-memory state to coordinate between the client and server, in some cases just as a way to pass data back and forth, but in other cases actually synchronizing execution to ensure proper ordering for the what the test is validating. That's much harder (if not impossible) to do well with a remote server.
wfurt commented 3 years ago

for the state, can we use the new connection hook and use transport other than TCP socket? Pipe, memory stream or something else?

halter73 commented 3 years ago

I'm not sure what these tests are using for the server-side, but I've started using the new SocketsHttpHandler connection hook and an in-memory stream (based on System.IO.Pipelines) in some Kestrel tests.

https://github.com/dotnet/aspnetcore/blob/925926c6156e56c6414ad004e5f1e54ced8c29c4/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2EndToEndTests.cs#L49-L61

stephentoub commented 3 years ago

Yes, we could use the connection callback for SocketsHttpHandler to use something other than sockets for testing... but that would only apply to SocketsHttpHandler. It wouldn't address the browser-based handler that is currently the crux of @lewing's issue. If we were to expose similar callbacks on HttpClientHandler (which wraps the browser-based handler), then maybe we could, but I don't know if the browser-based handler would even be able to support it... I suspect not... but maybe in that case supplying such a connection callback would actually opt you out of the browser-based handler? I'm waving my hands.

lewing commented 3 years ago

We've resolved some other blocking issues for testing the networking side in CI for Browser but we still need some way to test actual network requests even if they are going to a local endpoint.

geoffkizer commented 3 years ago

I don't really see any alternative here other than remote server tests. These platforms don't support doing loopback and they don't support connection-level hooks. They basically only support talking to a remote server.

As mentioned above, we do have some remote server tests, but in general they are not factored in a way that would make it easy to run only these tests (and not the loopback tests). One thing that would help here would be to refactor these tests so remote server tests are separated from loopback tests; see also https://github.com/dotnet/runtime/issues/30205

The remote server tests we have today are fairly limited, but there's no reason we couldn't add some more here if we wanted. I don't think we need anywhere near the coverage for mobile/browser scenarios that we want for SocketsHttpHandler, because we are mainly relying on the platform HTTP implementation in these cases, whereas SocketsHttpHandler is implementing HTTP itself and thus we want to do extensive on-the-wire validation (which, as @stephentoub pointed out above, is only possible when using a local, in-memory server).

wfurt commented 3 years ago

The "loopback" can possibly be implemented externally IMHO. Let say we have external helper outside of emulator proxying connection. The "loopback" can connect to it and tell client what port to connect to. We would end-up with two stream both in process so I think most of the current logic would work.

https://github.com/dotnet/runtime/blob/4a9ae3229fbc1f90f7b2e8bfd2c5e258d5fde8b4/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs#L27-L47

All the helper would need to do is to pick a port, give it back to the code above and splice any data to/from the first connection to it.

Is UnixDomainSocket (or something similar) viable option @lewing? We could hijack the actual TCP and we could steer it to "loopback" via other means. Once again we would end up with test process having both sides locally available. I don't know if the platforms handlers could work on UDS but curl can AFAIK so there is at least some precedent.

geoffkizer commented 3 years ago

I'm not quite understanding what you are proposing, and I'm not sure how it addresses the "close coordination" issue. Maybe some code examples would help?

wfurt commented 3 years ago

I put together crude prototype @geoffkizer I made small changes to Loopback: https://github.com/dotnet/runtime/compare/main...wfurt:remoteLoop?expand=1

I pick test that uses LoopbackServer.CreateClientAndServerAsync(...) before running the test I set HELPER=192.168.0.106:10000 I started the helper on 192.168.0.106. The loopback can run successfully without any changes and without listening and incoming socket.

With that, we should be able to run existing tests without modifications. Timing my be different but if the helper runs on Helix machine where the emulator runs, everything will be done without hitting any real network. It would be up to the logic starting the emulator to also start the loopback helper and set the environment (or let tests know via some other means.

Let me know if that make some more sense.

BTW here is the simplified helper I used for testing

helper.cs ```c# using System; using System.Net; using System.Net.Sockets; using System.Threading.Tasks; namespace helper { class Program { static async Task relayData(Socket s1, Socket s2) { using (NetworkStream ns1 = new NetworkStream(s1, ownsSocket: true)) using (NetworkStream ns2 = new NetworkStream(s2, ownsSocket: true)) { Task t1 = ns1.CopyToAsync(ns2); Task t2 = ns2.CopyToAsync(ns1); // TBD errors and propagate half-close Task.WaitAll(t1, t2); Console.WriteLine("relayData all done"); } } static void Main(string[] args) { byte[] port = new byte[2]; int listenPort = 10000; if (args.Length > 0) { listenPort = int.Parse(args[0]); } var listener = new Socket(AddressFamily.InterNetworkV6, SocketType.Stream, ProtocolType.Tcp); listener.DualMode = true; listener.Bind(new IPEndPoint(IPAddress.IPv6Any, listenPort)); listener.Listen(1); while ( true ) { Socket s1 = listener.Accept(); Console.WriteLine("Got new connection from Loopback {0} {1}", s1.LocalEndPoint, s1.RemoteEndPoint); var l = new Socket(s1.AddressFamily, SocketType.Stream, ProtocolType.Tcp); if (((IPEndPoint)s1.LocalEndPoint).Address.IsIPv4MappedToIPv6) { l.DualMode = true; } l.Bind(new IPEndPoint(((IPEndPoint)s1.LocalEndPoint).Address, 0)); l.Listen(1); // Since we binded, the new listener has port assign. int p = ((IPEndPoint)l.LocalEndPoint).Port; port[0] = (byte)(((IPEndPoint)l.LocalEndPoint).Port & 0xff); port[1] = (byte)(((IPEndPoint)l.LocalEndPoint).Port >> 8); // send it to loopback so client knows where to connect to. s1.Send(port); // This will block until HttpClient connects Socket s2 = l.Accept(); l.Close(); // Send one byte to signal presense of connected client. // We may send whole RemoteEndPoint if that is ever interesting. s1.Send(port, 0, 1, SocketFlags.None); Task.Run(() => relayData(s1, s2)); } } } } ```
karelz commented 3 years ago

@wfurt is the ask only about emulators, or also about real HW? (AFAIK Mono used to have lab full of real phones) Either way, your scheme should work - we would be just hitting network. The infra would have to take care of deploying simultaneous server somewhere of course ...

wfurt commented 3 years ago

yes, this should work for real HW as well. Just the coordination may be trickier and I don't know much about the deployment at the moment, (I did some experiments with the emulators in the past)

What should be the next step @karelz? I can do more polishing and verification but I think we should confirm directions first.

karelz commented 3 years ago

I would suggest discussion with @lewing to see if they like the approach and how it would integrate with their infra.

geoffkizer commented 3 years ago

Maybe I'm missing something, but this still requires Socket.Connect support, and that doesn't exist on browser platforms, right?

wfurt commented 3 years ago

That it true for iWatch AFAIK @geoffkizer

Some platforms don't support listening to incoming requests but do support making outgoing requests.

Perhaps @lewing, @akoeplinger and @marek-safar should clarify what capabilities we can expected. The proposal is based on assumption that we can connect directly outbound.

akoeplinger commented 3 years ago

The proposal is based on assumption that we can connect directly outbound.

You can connect outbound but watchOS and the browser don't allow access to regular raw BSD sockets, so you can only go through the platform's HTTP/WebSocket request APIs where the APIs are at the level of "request https://example.com, get back stream/bytes". There's no way to connect at the socket level.

wfurt commented 3 years ago

are there any other platforms where sockets exist but not inbound? Or is this really only about the two platforms you mention @akoeplinger?

The WebSocket could conceptually work IMHO as well, as long as we get API where we can send and read arbitrary data. The encapsulation would be more complicated but I think we could still avoid dependency on external services and big test rewrite.

akoeplinger commented 3 years ago

are there any other platforms where sockets exist but not inbound? Or is this really only about the two platforms you mention @akoeplinger?

None that I'm aware, at least not for .NET 6.

wfurt commented 3 years ago

Will that be plumbed to our WebSocket API or would we need to use it in some other way? I'm wondering if there is path forward for experiments.

akoeplinger commented 3 years ago

Not sure, @lewing do you know?

pavelsavara commented 3 years ago

Hi @wfurt, @karelz ,

I started working on solving this problem for the browser/mobile. In the first phase I'm not solving generic loopback socket, just the echo services. The idea is to host the echo server in the xharness process.

I have very simple draft, in which I re-implemented 2 endpoints. It works fine but it has two problems:

I would like to be able to reuse the endpoint handler code and keep it in the runtime repo. I thought that I would extend xharness to be able to consume assembly with the echo handlers as binary assembly via commandline switch.

To be able to do that I would have to upgrade code of the existing CoreFxNetCloudService from ASP.NET 4.5.1 to .NET core/kestrel. Unfortunately the current project doesn't even compile for me.

Right now I'm thinking that I would create new kestrel library project side by side with WebServer and make it work with xharness.

As it would eventually become functionally 1:1 with the old implementation, you could take over the new code and deploy it to Azure instead of the current one.

Does that work for you ? Do you have any suggestions ?

karelz commented 3 years ago

@pavelsavara I don't think the source code you found is used for our Azure endpoints. I believe the latest version of the source code now lives in https://github.com/davidsh/corefx-net-endpoint which we have longer-term plan to move into dotnet org (I don't think it happened yet).

Can you check that source code? If it is usable for your efforts, we can move it into Runtime repo. That may save you some effort ...

wfurt commented 3 years ago

I'm not sure if the echo belongs to xharness. Many tests still need close coordination and as far as I understand, outbound connections work so using the Azure endpoint (or what ever else) should work. What we discussed with @lewing was idea of plumbing socket proxy to xharness so we can sort of emulate inbound connections. Perhaps we can chat on Teams to get in sync @pavelsavara ?

premun commented 3 years ago

I am not sure how much you are talking about mobile devices here and how much about WASM only, but please bare in mind that on Apple devices we have in Helix (iOS 14), local network access is not possible. You cannot open any connection to a local IP. This started with iOS 14 this year, so it's something new too.

We go around this by creating a TCP tunnel through the USB cabel and then XHarness and the phone connect to a port on their localhost. I don't think there is such limitation for public network.

Maybe I am off as I don't understand the whole scenario but just something to keep in mind ^^

pavelsavara commented 3 years ago

We spoke with @wfurt yesterday and clarified that echo in xharness is just first step. It would unblock wasm developers to be able to create new unit tests. To to create necessary server side for testing wasm specific websocket scenarios locally, without Azure deployment.

My https://github.com/dotnet/runtime/pull/52642 doesn't cover full scope of the original CoreFxNetCloudService, but I don't need it now. So I created https://github.com/dotnet/runtime/issues/52693 to describe the gaps I know about.

@premun we briefly spoke about iOS limitations on Mono team meeting yesterday. @marek-safar suggested that Wasm and Android platforms are good first steps. As we discussed offline with @premun, we could enable TCP tunnel as later step for iOS.

About @wfurt idea of "asking external server to create the loopback", that could work with both Azure and xharness hosted server. For wasm, the unit test is running in the browser, where we don't have full socket, nor pipe. So we would have to implement the "ask"/coordination part with websockets. Again, this is later step from my perspective.

karelz commented 3 years ago

@pavelsavara I am a bit lost in the overall plan. What happens when and who does it? Would it make se to agree on a plan and then keep up-to-date version in the top post?

pavelsavara commented 3 years ago

@karelz I updated the issue description with my plan. I will talk to @lewing to confirm priorities. Thanks for your feedback.

wfurt commented 3 years ago

What exactly do you mean by 'local network' @premun? Does it mean that you can access remote networks but not directly connected one? I assume that is some limitation of the emulator, right?

premun commented 3 years ago

@wfurt this is for iOS devices. Simulators seem fine but I didn't try it much. I haven't tried remote networks. For those you might need some permission which you can put in the application's manifest.

Details on the local network access: https://support.apple.com/en-mk/HT211870 https://developer.apple.com/videos/play/wwdc2020/10110/

You cannot connect to things that "don't look localhost" but I don't understand how Apple tells it's "local". We were originally having a direct connection and we were trying to connect to things like these (which ended up in the dialog):

The original intention was so that you cannot scan devices around and figure out location or fingerprint people's home...

wfurt commented 3 years ago

good to know @premun. I think the work @pavelsavara is doing can still work. Aside from external server, he is going to do simple prototype with "Loopback", Once that ready you can perhaps check if that would work on iOS devices and provide more feedback.

pavelsavara commented 3 years ago

Both loopback and echo server are now also implemented as kestrel middleware for Xhareness process. Many tests are now enabled for WASM platform.

@mdh1418 is looking at Android.

mdh1418 commented 3 years ago

Closing this issue as the wasm side is now complete. Separating mobile work into another issue https://github.com/dotnet/runtime/issues/54626