Script execution hangs on http.client.HTTPSConnection.getresponse()

[punk88]

Description

Script hangs on read reasponse via http.client.HTTPSConnection.

Steps to Reproduce

I have script that send https request and read its response via http.client module:

conn = http.client.HTTPSConnection(host)

conn.request(
""GET"",
""/weatherforecast"", 
headers={
""Accept-Encoding"": ""identity"",
""content-type"": ""application/json""
})
response = conn.getresponse()

Expected behavior:

Script is executing without problems

Actual behavior:

Under certain conditions(which i describe below) script hang on line response = conn.getresponse()

Version Information

3.4.1

Additional information

I got this promplem under certain conditions, when response size is multiple of 2048 bytes

Stack trace on script hangs:

System.Net.Sockets.dll!Interop.Winsock.recv.____PInvoke|12_0
[Managed to Native Transition]
System.Net.Sockets.dll!Interop.Winsock.recv.____PInvoke|12_0(nint socketHandle, byte* pinnedBuffer, int len, System.Net.Sockets.SocketFlags socketFlags)
System.Net.Sockets.dll!Interop.Winsock.recv(System.Net.Sockets.SafeSocketHandle socketHandle, byte* pinnedBuffer, int len, System.Net.Sockets.SocketFlags socketFlags)
System.Net.Sockets.dll!System.Net.Sockets.SocketPal.Receive(System.Net.Sockets.SafeSocketHandle handle, System.Span<byte> buffer, System.Net.Sockets.SocketFlags socketFlags, out int bytesTransferred)
System.Net.Sockets.dll!System.Net.Sockets.Socket.Receive(System.Span<byte> buffer, System.Net.Sockets.SocketFlags socketFlags, out System.Net.Sockets.SocketError errorCode)
System.Net.Sockets.dll!System.Net.Sockets.NetworkStream.Read(System.Span<byte> buffer)
System.Net.Security.dll!System.Net.Security.SyncReadWriteAdapter.ReadAsync(System.IO.Stream stream, System.Memory<byte> buffer, System.Threading.CancellationToken cancellationToken)
System.Net.Security.dll!System.Net.Security.SslStream.EnsureFullTlsFrameAsync<System.Net.Security.SyncReadWriteAdapter>(System.Threading.CancellationToken cancellationToken)
System.Net.Security.dll!System.Net.Security.SslStream.ReadAsyncInternal<System.Net.Security.SyncReadWriteAdapter>(System.Memory<byte> buffer, System.Threading.CancellationToken cancellationToken)
System.Net.Security.dll!System.Net.Security.SslStream.Read(byte[] buffer, int offset, int count)
IronPython.Modules.dll!IronPython.Modules.PythonSsl._SSLSocket.read(IronPython.Runtime.CodeContext context, int size, IronPython.Runtime.ByteArray buffer)
Microsoft.Dynamic.dll!Microsoft.Scripting.Interpreter.FuncCallInstruction<IronPython.Modules.PythonSsl._SSLSocket, IronPython.Runtime.CodeContext, int, IronPython.Runtime.ByteArray, object>.Run(Microsoft.Scripting.Interpreter.InterpretedFrame frame)
Microsoft.Dynamic.dll!Microsoft.Scripting.Interpreter.Interpreter.Run(Microsoft.Scripting.Interpreter.InterpretedFrame frame)
Microsoft.Dynamic.dll!Microsoft.Scripting.Interpreter.LightLambda.Run5<System.Runtime.CompilerServices.CallSite, IronPython.Runtime.CodeContext, object, object, object, object>(System.Runtime.CompilerServices.CallSite arg0, IronPython.Runtime.CodeContext arg1, object arg2, object arg3, object arg4)
System.Linq.Expressions.dll!System.Dynamic.UpdateDelegates.UpdateAndExecute4<IronPython.Runtime.CodeContext, object, object, object, object>(System.Runtime.CompilerServices.CallSite site, IronPython.Runtime.CodeContext arg0, object arg1, object arg2, object arg3)
Microsoft.Dynamic.dll!Microsoft.Scripting.Interpreter.DynamicInstruction<IronPython.Runtime.CodeContext, object, object, object, object>.Run(Microsoft.Scripting.Interpreter.InterpretedFrame frame)
Microsoft.Dynamic.dll!Microsoft.Scripting.Interpreter.Interpreter.Run(Microsoft.Scripting.Interpreter.InterpretedFrame frame)
Microsoft.Dynamic.dll!Microsoft.Scripting.Interpreter.LightLambda.Run4<IronPython.Runtime.PythonFunction, object, object, object, object>(IronPython.Runtime.PythonFunction arg0, object arg1, object arg2, object arg3)
IronPython.dll!IronPython.Compiler.PythonCallTargets.OriginalCallTarget3(IronPython.Runtime.PythonFunction function, object arg0, object arg1, object arg2)
IronPython.dll!IronPython.Runtime.FunctionCaller<object, object, object>.Call3(System.Runtime.CompilerServices.CallSite site, IronPython.Runtime.CodeContext context, object func, object arg0, object arg1, object arg2)
...

In my opinion, problem in code in _ssl.cs on method read():

[Documentation(@"read(size, [buffer])
Read up to size bytes from the SSL socket.")]
            public object read(CodeContext/*!*/ context, int size, ByteArray buffer = null) {
                EnsureSslStream(true);

                try {
                    byte[] buf = new byte[2048];
                    MemoryStream result = new MemoryStream(size);
                    while (true) {
                        int readLength = (size < buf.Length) ? size : buf.Length;
                        int bytes = _sslStream.Read(buf, 0, readLength);
                        if (bytes > 0) {
                            result.Write(buf, 0, bytes);
                            size -= bytes;
                        }

                        if (bytes == 0 || size == 0 || bytes < readLength) {
                            var res = result.ToArray();
                            if (buffer == null)
                                return Bytes.Make(res);

                            // TODO: get rid of the MemoryStream and write directly to the buffer
                            buffer[new Slice(0, res.Length)] = res;
                            return res.Length;
                        }
                    }
                } catch (Exception e) {
                    throw PythonSocket.MakeException(context, e);
                }
            }

_sslStream initialization:

_sslStream = new SslStream(
                            new NetworkStream(_socket._socket, false),
                            true,
                            _callback,
                            CertSelectLocal
                        );

This code read data from NetworkStream(inside SslStream) in cycle(while) of 2048 bytes. When whole response size is multiple of 2048 bytes(in my case I got this problem on response size 4096 bytes) then _sslStream.Read(buf, 0, readLength) hangs. This known behavior of NetworkStream.Read() blocks until data becomes available or the connection is closed. See https://stackoverflow.com/a/6958290 and another issues. So, in my case I have 3 iterations: read 2048 bytes, read 2048 bytes, and read 0 bytes expected where I hangs.

Workaround in scripts for this problem is use header 'Connection': 'close' instead default 'keep-alive', so with 'close' server close connection and cycle in code ends. But need a solution without this hardcode in scripts

[slozier]

Thanks for the report! I've been able to reproduce the issue.

[stukselbax]

When release of package will be available with this fix?

[slozier]

I don't have concrete plans for a release at the moment. You should be able to grab nuget packages from CI (e.g. https://dotnet.visualstudio.com/IronLanguages/_build/results?buildId=101933&view=results) if you need the fix right away...