GSS failures in System.Net.Http.Functional.Tests on Ubuntu 22.04

[omajid]

Description

Running runtime tests on Ubuntu 22.04 (which adds OpenSSL 3.0 resulting in a number of changes under the hood), leads to a bunch of tests failing: https://dev.azure.com/dnceng/public/_build/results?buildId=1690650&view=ms.vss-test-web.build-test-results-tab&runId=46193442&resultId=189361&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

Some examples:

System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
---- System.IO.IOException : Unexpected EOF trying to read request header

    System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL]
      System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header)
      ---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure.  Minor code may provide more information (Crypto routine failure).
      ---- System.IO.IOException : Unexpected EOF trying to read request header
      Stack Trace:
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(88,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks)
        /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<<CreateClientAndServerAsync>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(669,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme)
        --- End of stack trace from previous location ---
        ----- Inner Stack Trace #1 (System.ComponentModel.Win32Exception) -----
        /_/src/libraries/Common/src/System/Net/Security/NegotiateStreamPal.Unix.cs(537,0): at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(128,0): at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(98,0): at System.Net.NTAuthentication..ctor(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs(169,0): at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1033,0): at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.cs(240,0): at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
        /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(532,0): at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(676,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__0>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)
        ----- Inner Stack Trace #2 (System.IO.IOException) -----
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(715,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync()
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(777,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1021,0): at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1081,0): at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content)
        /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(686,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<<Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader>b__1>d.MoveNext()
        --- End of stack trace from previous location ---
        /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task)

Reproduction Steps

From helix:

/root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Http.Functional.Tests.runtimeconfig.json --depsfile System.Net.Http.Functional.Tests.deps.json xunit.console.dll System.Net.Http.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 

Expected behavior

All tests pass

Actual behavior

Tests fail with GSS exceptions.

Regression?

Yes, the tests pass on older versions of Ubuntu currently running in CI

Known Workarounds

No response

Configuration

• dotnet/runtime main branch, commit 5a0564b01442f8ea9247e27c4fab85ee0d457265
• Ubuntu 22.04

Other information

No response

[ghost]

Tagging subscribers to this area: @dotnet/ncl See info in area-owners.md if you want to be subscribed.

Issue Details

---

### Description Running runtime tests on Ubuntu 22.04 (which adds OpenSSL 3.0 resulting in a number of changes under the hood), leads to a bunch of tests failing: https://dev.azure.com/dnceng/public/_build/results?buildId=1690650&view=ms.vss-test-web.build-test-results-tab&runId=46193442&resultId=189361&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab Some examples: ``` System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header) ---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (Crypto routine failure). ---- System.IO.IOException : Unexpected EOF trying to read request header ``` ``` System.Net.Http.Functional.Tests.SyncHttpHandler_HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(authScheme: "NTLM") [FAIL] System.AggregateException : One or more errors occurred. (GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (Crypto routine failure).) (Unexpected EOF trying to read request header) ---- System.ComponentModel.Win32Exception : GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (Crypto routine failure). ---- System.IO.IOException : Unexpected EOF trying to read request header Stack Trace: /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(88,0): at System.Threading.Tasks.TaskTimeoutExtensions.WhenAllOrAnyFailed(Task[] tasks) /_/src/libraries/Common/tests/System/Net/Http/GenericLoopbackServer.cs(38,0): at System.Net.Test.Common.LoopbackServerFactory.<>c__DisplayClass5_0.<b__0>d.MoveNext() --- End of stack trace from previous location --- /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(101,0): at System.Net.Test.Common.LoopbackServer.CreateServerAsync(Func`2 funcAsync, Options options) /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(669,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.Credentials_ServerChallengesWithWindowsAuth_ClientSendsWindowsAuthHeader(String authScheme) --- End of stack trace from previous location --- ----- Inner Stack Trace #1 (System.ComponentModel.Win32Exception) ----- /_/src/libraries/Common/src/System/Net/Security/NegotiateStreamPal.Unix.cs(537,0): at System.Net.Security.NegotiateStreamPal.AcquireCredentialsHandle(String package, Boolean isServer, NetworkCredential credential) /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(128,0): at System.Net.NTAuthentication.Initialize(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding) /_/src/libraries/Common/src/System/Net/NTAuthentication.Common.cs(98,0): at System.Net.NTAuthentication..ctor(Boolean isServer, String package, NetworkCredential credential, String spn, ContextFlagsPal requestedContextFlags, ChannelBinding channelBinding) /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.NtAuth.cs(169,0): at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken) /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/HttpConnectionPool.cs(1033,0): at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/AuthenticationHelper.cs(240,0): at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken) /_/src/libraries/System.Net.Http/src/System/Net/Http/SocketsHttpHandler/RedirectHandler.cs(30,0): at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) /_/src/libraries/System.Net.Http/src/System/Net/Http/HttpClient.cs(532,0): at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(676,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<b__0>d.MoveNext() --- End of stack trace from previous location --- /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) ----- Inner Stack Trace #2 (System.IO.IOException) ----- /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(715,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestHeaderBytesAsync() /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(777,0): at System.Net.Test.Common.LoopbackServer.Connection.ReadRequestDataAsync(Boolean readBody) /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1021,0): at System.Net.Test.Common.LoopbackServer.Connection.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) /_/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs(1081,0): at System.Net.Test.Common.LoopbackServer.HandleRequestAsync(HttpStatusCode statusCode, IList`1 headers, String content) /_/src/libraries/Common/tests/System/Net/Http/HttpClientHandlerTest.Authentication.cs(686,0): at System.Net.Http.Functional.Tests.HttpClientHandler_Authentication_Test.<>c__DisplayClass34_0.<b__1>d.MoveNext() --- End of stack trace from previous location --- /_/src/libraries/Common/tests/System/Threading/Tasks/TaskTimeoutExtensions.cs(120,0): at System.Threading.Tasks.TaskTimeoutExtensions.GetRealException(Task task) ``` ### Reproduction Steps From helix: ``` /root/helix/work/correlation/dotnet exec --runtimeconfig System.Net.Http.Functional.Tests.runtimeconfig.json --depsfile System.Net.Http.Functional.Tests.deps.json xunit.console.dll System.Net.Http.Functional.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing ``` ### Expected behavior All tests pass ### Actual behavior Tests fail with GSS exceptions. ### Regression? Yes, the tests pass on older versions of Ubuntu currently running in CI ### Known Workarounds _No response_ ### Configuration - dotnet/runtime main branch, commit `5a0564b01442f8ea9247e27c4fab85ee0d457265` - Ubuntu 22.04 ### Other information _No response_

Author:	omajid
Assignees:	-
Labels:	`area-System.Net.Http`, `untriaged`
Milestone:	-

[karelz]

Triage: Likely related to new image Ubuntu 22.04 (we do not have a queue yet). Seems to fail reliably. We should investigate.

[wfurt]

do you know @omajid if the image has the gss-ntlm package? Generally, I would think the OpenSSL is independent from Kerberos and GSS.

[omajid]

It does have gss: https://github.com/dotnet/dotnet-buildtools-prereqs-docker/blob/main/src/ubuntu/22.04/helix/amd64/Dockerfile

[wfurt]

Actually, you may be right about OpenSSL. It seems like md4 is no longer available from crypto

helixbot@9d96aeaca4ba:/$ openssl md4
Error setting digest
4037123E367F0000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:349:Global default library context, Algorithm (MD4 : 88), Properties ()
4037123E367F0000:error:03000086:digital envelope routines:evp_md_init_internal:initialization error:../crypto/evp/digest.c:237:

and

https://github.com/gssapi/gss-ntlmssp/blob/734e522c14a9821d7c03f2ce1691706d3d8131ad/src/crypto.c#L149-L153 While the gss-ntlmssp builds it is probably completely useless. (something to report ditto vendors..?)

For now, I think we detect presence of the page and we would skip tests as needed. Short term fix may be removing the ntlm package from docker image. That of course leaves NTLM auth broken.

The options would be to report/fix the package so it works with OpenSSL 3.x (e.g. add private fall-back implementation of md4) or switch to managed implementation #66879

cc: @filipnavara @bartonjs

[filipnavara]

Let's report upstream and see. Long term I am keen on making the Managed NTLM an option either through an app context switch, or as a fallback if gss-ntlmssp is not installed.

[omajid]

Should we flag this as part of Ubuntu 22.04 support? Looking at https://github.com/dotnet/core/issues/7038 it seems like everything is 100% functional?

[wfurt]

I'm not sure. This looks like distribution bug to me @omajid as the package they provide does not work.

[vcsjones]

It seems like md4 is no longer available from crypto

You need to load the legacy provider for that to work in OpenSSL 3. You can either do that in openssl.cnf, or from the command line, this should work:

echo hi | openssl md4 -provider legacy

However for the runtime, we explicitly load the "legacy" provider, so MD4 should be available.

https://github.com/dotnet/runtime/blob/fe0f600e46ebcaf25ad045b0469c6ff11b527381/src/native/libs/System.Security.Cryptography.Native/openssl.c#L1175-L1180

[vcsjones]

One way to tell is to change openssl.cnf to have the following provider_sect:

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

[filipnavara]

However for the runtime, we explicitly load the "legacy" provider, so MD4 should be available.

...but only after you use some crypto that initializes the OpenSSL native shim, right?

[vcsjones]

Ah, you're right. I thought we always loaded the legacy provider, but we only do it when you use an algorithm that is in the legacy provider:

https://github.com/dotnet/runtime/blob/899bf9704693661d6fe53fdb7f737f76447efa14/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/RC2Implementation.OpenSsl.cs#L23

Changing the openssl.cnf to load the legacy provider however would work, assuming the problem is the lack of the legacy provider's availability.

[wfurt]

This was fixed in gss-ntlm package. It is up to Ubuntu to pick up the fix. Big thanks to @simo5 who did the fix.

[karelz]

@wfurt why did you send it back to triage? If it is external (and addressed there already), I would recommend to close it with details which version of library is needed to make it work.

[wfurt]

We can (should?) improve platform detection and skip the tests as needed instead of failing. This will bite us once #67345 is merged. We may also choose to solve it via manage NTLM.

[karelz]

Triage: Platform detection needs to be improved to handle the case as well.

Note: This will be addressed once we have managed NTLM implementation - but there is no guarantee when it will happen.

[yaakov-h]

This isn't just a test-related issue. I just upgraded a system from Ubuntu 20.04 to Ubuntu 22.04 and a .NET 6 application could no longer use NTLM auth until I applied the workaround mentioned in https://github.com/dotnet/runtime/issues/67353#issuecomment-1085003764.

Should this be documented somewhere as a compatibility issue for developers / end-users?

[wfurt]

We could but it is really difficult to trace and keep in sync all Linux distributions and versions. We could add note that the functionality depends on underlying OS capabilities. But as I mentioned above it is really up to Ubuntu to pick up the package fixes. Perhaps you can open issue with them.

[wooncherk]

One way to tell is to change openssl.cnf to have the following provider_sect:

[provider_sect]
default = default_sect
legacy = legacy_sect

[default_sect]
activate = 1

[legacy_sect]
activate = 1

This helped. Thanks! :)

[SaravanakumBalach]

Hi,

We are planning to upgrade our .Net core MVC application from .Net 6 to .Net 8 version. It is written in C#. To prepare for that upgrade we first upgraded from ubuntu 20.04 to ubuntu 22.04. We target ubuntu 22.04 version jammy tag with amd64 architecture.

This is our base image in the Docker file and the following line updates our package list.

FROM artifactory.xyz.com/dockerhub-microsoft/dotnet/aspnet:6.0-jammy-amd64 AS base

&& apt-get update && apt-get install -y --no-install-recommends curl gss-ntlmssp tzdata \

When our application tries to authenticate and open SSRS reports we get "GSSAPI operation failed with error - Unspecified GSS failure. Minor code may provide more information (Crypto routine failure)"

We were previously using focal base image in 20.04 version and was able to render SSRS reports. Did something change with gss-ntlmssp package in 22.04 version? I was reading some other posts where it says this issue is related to incompatibilities between OpenSSL 3.0 and the older cryptographic algorithms involved in NTLM authentication. Any thoughts on how to fix this issue? Appreciate your kind response.

[filipnavara]

Any thoughts on how to fix this issue?

One of the fixes is literally the comment right above yours (enable the legacy crypto in OpenSSL). The other one is to get newer version of the gss-ntlmssp package or compile it yourself.

[wfurt]

One of the fixes is literally the comment right above yours (enable the legacy crypto in OpenSSL). The other one is to get newer version of the gss-ntlmssp package or compile it yourself.

there seems to be updated binaries in never Ubuntu. You may be able to get binaries from there with little bit of trickery (but I did not test it)

[wfurt]

and in .NET 8 they could force the managed implementation, right @filipnavara if all they need is NTLM?

[wfurt]

https://packages.ubuntu.com/search?keywords=gss-ntlmssp you want the 1.2.xxx version @SaravanakumBalach

[SaravanakumBalach]

Thanks @wfurt . I looked at it but it appears like 1.2.xxx version is available only with ubuntu 23.x and above. But we are targeting ubuntu 22.04 which only supports GSSAPI 0.7.0 version. @filipnavara How can I enable the legacy crypto in OpenSSL? Any other thoughts pls?

[wfurt]

right, but

curl -LO http://mirrors.kernel.org/ubuntu/pool/universe/g/gss-ntlmssp/gss-ntlmssp_1.2.0-1_amd64.deb
sudo dpkg -i gss-ntlmssp_1.2.0-1_amd64.deb

works because all dependencies are met

furt@ubu22:/tmp$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

https://packages.ubuntu.com/lunar/gss-ntlmssp