search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.27k stars 4.73k forks source link

Ubuntu Server 20.04 SO_REUSEPORT throws System.Net.Sockets.SocketException (95): Operation not supported #73920

Open okarpov opened 2 years ago

okarpov commented 2 years ago

Description

Enabling SO_REUSEPORT throws exception on Ubuntu Server 20.04

Operation not supported System.Net.Http.HttpRequestException: Operation not supported 
 ---> System.Net.Sockets.SocketException (95): Operation not supported

Reproduction Steps

// Default socket creation logic
Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
socket.NoDelay = true;

// Enable SO_REUSE_UNICASTPORT:
socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseUnicastPort, 1);

try
{
    await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false);
    return new NetworkStream(socket, ownsSocket: true);
}
catch (Exception ex)
{
    if (Log.IsEnabled)
    {
        Log._Log(this, ex.Message, ex);
    }
    socket.Dispose();
    throw;
}

Expected behavior

should reuse ports as per SO_REUSEPORT feature

Actual behavior

Operation not supported System.Net.Http.HttpRequestException: Operation not supported 
 ---> System.Net.Sockets.SocketException (95): Operation not supported

Regression?

No response

Known Workarounds

No response

Configuration

Ubuntu server 20.04 .Net Core 6

Other information

No response

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details
### Description Enabling SO_REUSEPORT throws exception on Ubuntu Server 20.04 Operation not supported System.Net.Http.HttpRequestException: Operation not supported ---> System.Net.Sockets.SocketException (95): Operation not supported ### Reproduction Steps ``` // Default socket creation logic Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp); socket.NoDelay = true; // Enable SO_REUSE_UNICASTPORT: socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseUnicastPort, 1); try { await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false); return new NetworkStream(socket, ownsSocket: true); } catch (Exception ex) { if (Log.IsEnabled) { Log._Log(this, ex.Message, ex); } socket.Dispose(); throw; } ``` ### Expected behavior should reuse ports as per SO_REUSEPORT feature ### Actual behavior Operation not supported System.Net.Http.HttpRequestException: Operation not supported ---> System.Net.Sockets.SocketException (95): Operation not supported ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration Ubuntu server 20.04 .Net Core 6 ### Other information _No response_
Author: okarpov
Assignees: -
Labels: `area-System.Net`
Milestone: -
wfurt commented 2 years ago

it seems like that API is Windows specific. cc: @tmds @antonfirsov You can probably use SocketOptionName.ReuseAddress on Linux. It will set both...

strace -e network -f  /tmp/socket/bin/Debug/net6.0/linux-x64/publish/socket
....
[pid 46509] setsockopt(40, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
[pid 46509] setsockopt(40, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
okarpov commented 2 years ago

it seems like that API is Windows specific. cc: @tmds @antonfirsov You can probably use SocketOptionName.ReuseAddress on Linux. It will set both...

strace -e network -f  /tmp/socket/bin/Debug/net6.0/linux-x64/publish/socket
....
[pid 46509] setsockopt(40, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
[pid 46509] setsockopt(40, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

Thx, I will check it

okarpov commented 2 years ago

it seems like that API is Windows specific. cc: @tmds @antonfirsov You can probably use SocketOptionName.ReuseAddress on Linux. It will set both...

strace -e network -f  /tmp/socket/bin/Debug/net6.0/linux-x64/publish/socket
....
[pid 46509] setsockopt(40, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_IPV6, IPV6_V6ONLY, [0], 4) = 0
[pid 46509] setsockopt(40, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
[pid 46509] setsockopt(40, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

does not work :(

okarpov commented 2 years ago 
System.AggregateException: One or more errors occurred. (Sockets on this platform are invalid for use after a failed connection attempt. )
 ---> System.Net.Http.HttpRequestException: Sockets on this platform are invalid for use after a failed connection attempt. 
 ---> System.PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
   at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()
   at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)

                    // Default socket creation logic
                    Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
                    socket.NoDelay = true;

                    // Enable SO_REUSE_UNICASTPORT:
                    //socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseUnicastPort, 1);
                    socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);

                    try
                    {
                        await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false);
                        return new NetworkStream(socket, ownsSocket: true);
                    }
                    catch (Exception ex)
                    {
                        if (Log.IsEnabled)
                        {
                            Log._Log(this, ex.Message, ex);
                        }
                        socket.Dispose();
                        throw;
                    }
wfurt commented 2 years ago

This is not what I see @okarpov. Can you pose complete stand alone repro? (instead of just fragment)

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details
### Description Enabling SO_REUSEPORT throws exception on Ubuntu Server 20.04 ``` Operation not supported System.Net.Http.HttpRequestException: Operation not supported ---> System.Net.Sockets.SocketException (95): Operation not supported ``` ### Reproduction Steps ```c# // Default socket creation logic Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp); socket.NoDelay = true; // Enable SO_REUSE_UNICASTPORT: socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseUnicastPort, 1); try { await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false); return new NetworkStream(socket, ownsSocket: true); } catch (Exception ex) { if (Log.IsEnabled) { Log._Log(this, ex.Message, ex); } socket.Dispose(); throw; } ``` ### Expected behavior should reuse ports as per SO_REUSEPORT feature ### Actual behavior ``` Operation not supported System.Net.Http.HttpRequestException: Operation not supported ---> System.Net.Sockets.SocketException (95): Operation not supported ``` ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration Ubuntu server 20.04 .Net Core 6 ### Other information _No response_
Author: okarpov
Assignees: -
Labels: `area-System.Net.Sockets`, `untriaged`
Milestone: -
ghost commented 2 years ago

This issue has been marked needs-author-action and may be missing some important information.

okarpov commented 2 years ago

Welcome to Ubuntu 20.04 LTS (GNU/Linux 5.4.0-29-generic x86_64)

65 updates can be applied immediately. 25 of these updates are standard security updates. To see these additional updates run: apt list --upgradable

New release '22.04.1 LTS' available. Run 'do-release-upgrade' to upgrade to it.

okarpov commented 2 years ago

This is not what I see @okarpov. Can you pose complete stand alone repro? (instead of just fragment)

will create repro

okarpov commented 2 years ago 
        static void Main(string[] args)
        {
            //repro

            var coo = new CookieContainer();
            var socketsHandler = new System.Net.Http.HttpClientHandler()
            {
                AutomaticDecompression = System.Net.DecompressionMethods.GZip | System.Net.DecompressionMethods.Deflate | System.Net.DecompressionMethods.Brotli,
                AllowAutoRedirect = false,
                SslProtocols = System.Security.Authentication.SslProtocols.Tls11 | System.Security.Authentication.SslProtocols.Tls | System.Security.Authentication.SslProtocols.Tls12 | System.Security.Authentication.SslProtocols.Tls13,
                ServerCertificateCustomValidationCallback = (s, e, a, b) =>
                {
                    return true;
                },
                CookieContainer = coo,
                UseCookies = true,
                MaxAutomaticRedirections = 50,
                MaxConnectionsPerServer = int.MaxValue,
                PreAuthenticate = false,
            };

            var field = typeof(System.Net.Http.HttpClientHandler).GetField("_underlyingHandler", System.Reflection.BindingFlags.Instance | System.Reflection.BindingFlags.NonPublic);
            var sockHandler = (System.Net.Http.SocketsHttpHandler)field?.GetValue(socketsHandler);

            sockHandler.ConnectCallback = async (context, cancellationToken) =>
            {
                // Default socket creation logic
                Socket socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
                socket.NoDelay = true;

                // Enable SO_REUSE_UNICASTPORT:
                //socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseUnicastPort, 1);
                socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);

                try
                {
                    await socket.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false);
                    return new NetworkStream(socket, ownsSocket: true);
                }
                catch (Exception ex)
                {
                    Console.WriteLine(ex);
                    socket.Dispose();
                    throw;
                }
            };

            sockHandler.EnableMultipleHttp2Connections = true;
            sockHandler.UseCookies = true;
            sockHandler.CookieContainer = coo;

            System.Net.Http.HttpClient wr = new System.Net.Http.HttpClient(socketsHandler, true);

            wr.DefaultRequestHeaders.Clear();
            wr.DefaultRequestHeaders.Accept.TryParseAdd("*/*");
            wr.DefaultRequestHeaders.AcceptEncoding.TryParseAdd("gzip, deflate, br");
            wr.DefaultRequestHeaders.AcceptLanguage.TryParseAdd("en-US,en;q=0.9");
            wr.DefaultRequestHeaders.ConnectionClose = false;
            wr.DefaultRequestHeaders.Connection.Add("keep-alive");
            wr.DefaultRequestHeaders.CacheControl = new System.Net.Http.Headers.CacheControlHeaderValue() { NoCache = true };
            wr.DefaultRequestHeaders.Pragma.TryParseAdd("no-cache");

            wr.DefaultRequestHeaders.Add("Upgrade-Insecure-Requests", "1");

            wr.DefaultVersionPolicy = System.Net.Http.HttpVersionPolicy.RequestVersionOrHigher;
            wr.Timeout = TimeSpan.FromSeconds(10);

            System.Net.Http.HttpRequestMessage req = new System.Net.Http.HttpRequestMessage(System.Net.Http.HttpMethod.Get, "https://www.google.com")
            {
                Version = System.Net.HttpVersion.Version30,
                VersionPolicy = System.Net.Http.HttpVersionPolicy.RequestVersionOrLower
            };

            req.Headers.ConnectionClose = false;
            req.Headers.UserAgent.TryParseAdd("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36");
            req.Content = new System.Net.Http.StringContent("", null, null);
            req.Content.Headers.ContentType = null;

            try
            {
                var r = wr.SendAsync(req, System.Net.Http.HttpCompletionOption.ResponseContentRead).Result;
                string response = r.Content.ReadAsStringAsync().Result;

                Console.WriteLine(response);
            }
            catch(Exception ex)
            {
                Console.WriteLine(ex);
            }

            return;
}

Linux:

System.PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
   at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()
   at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at ConsoleApp1.Program.<>c.<<Main>b__0_0>d.MoveNext() in \ConsoleApp1\Program.cs:line 53
System.AggregateException: One or more errors occurred. (Sockets on this platform are invalid for use after a failed connection attempt. (www.google.com:443))
 ---> System.Net.Http.HttpRequestException: Sockets on this platform are invalid for use after a failed connection attempt. (www.google.com:443)
 ---> System.PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
   at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()
   at System.Net.Sockets.Socket.ConnectAsync(SocketAsyncEventArgs e, Boolean userSocket, Boolean saeaCancelable)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ConnectAsync(Socket socket)
   at System.Net.Sockets.Socket.ConnectAsync(EndPoint remoteEP, CancellationToken cancellationToken)
   at ConsoleApp1.Program.<>c.<<Main>b__0_0>d.MoveNext() in \ConsoleApp1\Program.cs:line 53
--- End of stack trace from previous location ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(HttpRequestMessage request)
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttp2ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at ConsoleApp1.Program.Main(String[] args) in \ConsoleApp1\Program.cs:line 98

Windows:

wfurt commented 2 years ago

It seems like the socket is failing to connect - and not failing on socket options as originally reported. I think you should try it without any HTTP - just getting to socket nailed down first. And the code above is still just a fragment referencing wr. (with that I cannot use it ASIS without further modifications)

okarpov commented 2 years ago

its complete console main method - why you can not use it?

it works on Windows 11 with no errors and returns google page html

okarpov commented 2 years ago

changing the socket ConnectCallback to this - works on linux and windows with no errors

        sockHandler.ConnectCallback = async (context, cancellationToken) =>
        {
                   var entry = await Dns.GetHostEntryAsync(context.DnsEndPoint.Host, AddressFamily.InterNetwork, cancellationToken);
                    var s = new System.Net.Sockets.Socket(System.Net.Sockets.AddressFamily.InterNetwork, System.Net.Sockets.SocketType.Stream, System.Net.Sockets.ProtocolType.Tcp);
                    s.Bind(new System.Net.IPEndPoint(System.Net.IPAddress.Parse(this.ipaddress), 0));

                    s.NoDelay = true;

                    await s.ConnectAsync(context.DnsEndPoint, cancellationToken).ConfigureAwait(false);
                    return new System.Net.Sockets.NetworkStream(s, ownsSocket: true);
        }
okarpov commented 2 years ago

commenting out that two lines returns google html page on linux as well

                 // Enable SO_REUSE_UNICASTPORT:
                //socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseUnicastPort, 1);
                //socket.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);
okarpov commented 2 years ago

dotnet --info

Host (useful for support): Version: 6.0.6 Commit: 7cca709db2

.NET SDKs installed: No SDKs were found.

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET runtimes or SDKs: https://aka.ms/dotnet-download

okarpov commented 2 years ago

have just found this: https://github.com/dotnet/runtime/issues/24917

tmds commented 2 years ago 
System.PlatformNotSupportedException: Sockets on this platform are invalid for use after a failed connection attempt.
   at System.Net.Sockets.Socket.ThrowMultiConnectNotSupported()

is because you're calling Socket.ConnectAsync with a DnsEndPoint.

You need to resolve the hostname, and try to connect to each IPEndPoint creating a new Socket after each failed attempt (cfr https://github.com/dotnet/runtime/issues/73920#issuecomment-1215743144).

okarpov commented 2 years ago

i took this example from here: https://github.com/dotnet/runtime/issues/48219 where the similar issue (Reuse Port) seems was resolved

also it works on windows, so this is not obvious why it does not work in linux

okarpov commented 2 years ago

main issue is that i'm trying to achieve is making our application to Reuse Port to avoid port exhausting on sending a lot of outbound connections.

so, could you, please, provide any affordable solution?

CarnaViire commented 2 years ago

Triage: not critical for 7.0, moving to Future

okarpov commented 2 years ago

setting this sysctl net.ipv4.tcp_tw_reuse=1 seems resolve the issue partially and now i can set s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseAddress, 1);

but this still throws exception s.SetSocketOption(SocketOptionLevel.Socket, SocketOptionName.ReuseUnicastPort, 1);