NegotiateStream fails when authenticating from linux client to windows server

[wimme]

Description

We're using NegotiateStream to authenticate clients on our server application. This works on Windows clients, but we would like to get it working on Linux clients as well.

Reproduction Steps

The server application always runs on a Windows Server OS and uses .NET Framework 4.6.2. In this case, the Windows Server is part of a local domain (I didn't test it on a server that is not in a domain).

The client application makes use of a library dll that gets build in both .NET Framework 4.6.2 and .NET Standard 2.0.

Clients using the .NET Framework 4.6.2 library dll on Windows works. Clients using the .NET Standard 2.0 library on .NET 8 applications on Windows works. Clients using the .NET Standard 2.0 library on .NET 8 applications on Linux does not work.

Linux clients are running:

• Ubuntu Server 22.04 LTS
• dotnet-runtime-8.0
• aspnetcore-runtime-8.0

Server implementation:

_authStream = new SSPIAuthStream();
_negotiateStream = new NegotiateStream(_authStream);

NegotiateStream stream = _negotiateStream;
stream.AuthenticateAsServer(
    (NetworkCredential)CredentialCache.DefaultCredentials,
    ProtectionLevel.None,                               // min. protectionlevel required
    TokenImpersonationLevel.Identification              // min. impersonationlevel required
);

if (stream.RemoteIdentity != null)
    name = stream.RemoteIdentity.Name;

authenticated = stream.IsAuthenticated;

Client implementation:

_authStream = new SSPIAuthStream();
_negotiateStream = new NegotiateStream(_authStream);

// determine client credentials
NetworkCredential credentials = null;
if (String.IsNullOrEmpty(userName))
    credentials = (NetworkCredential)CredentialCache.DefaultCredentials;
else
    credentials = new NetworkCredential(userName, password, domainName);

// launch client authentication process asynchronously.
_arAuthenticateAsClient = _negotiateStream.BeginAuthenticateAsClient(
    credentials,
    String.Empty,                                       // target name
    ProtectionLevel.None,                               // we don't need sign or encrypt since we'll ditch the stream once authenticated
    TokenImpersonationLevel.Identification,             // we only need to identify ourselves
    new AsyncCallback(callback_onAuthenticateComplete), // callback method when done
    null
    );

Expected behavior

I would expect when specifying a username/password that the authentication step works on Linux as well, just like on Windows.

Actual behavior

When a client running on Linux connects, the following error gets thrown on the server:

System.Security.Authentication.AuthenticationException: Authentication failed on the remote side (the stream might still be available for additional authentication attempts). ---> System.ComponentModel.Win32Exception: Unknown error (0xffffffff)
   --- End of inner exception stack trace ---
   at System.Net.Security.NegoState.ProcessReceivedBlob(Byte[] message, LazyAsyncResult lazyResult)
   at System.Net.Security.NegoState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.NegotiateStream.AuthenticateAsServer(NetworkCredential credential, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel requiredImpersonationLevel)

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[filipnavara]

It would help a lot if you could provide Wireshark captures, or otherwise capture the communication between the NegotiateStream instances (see #99227 for other diagnostic steps for a similar problem). Additionally, it's possible to enable .NET (https://github.com/filipnavara/httpclienttest/blob/9d3336a9aed944dbbdef8d0c4dc4007ea1163255/Program.cs#L33-L78) and Kerberos logs (KRB5_TRACE environment variable) for additional information.

[wimme]

Wireshark will be difficult, it's being transferred over an encrypted gRPC connection.

I added some logging inside the underlying stream used by the NegotiateStream that outputs the buffer when calling the read and write methods:

Client:

3/26/2024 12:51:36 WRITE: byte[] { 21, 1, 0, 0, 8 }
3/26/2024 12:51:36 WRITE: byte[] { 2, 3, 9, 128, 255, 255, 255, 255 }

Server:

03/26/2024 12:51:36 READ: byte[] { 21, 1, 0, 0, 8 }
03/26/2024 12:51:36 READ: byte[] { 2, 3, 9, 128, 255, 255, 255, 255 }

This means the connection works, we're getting the same data.

Comparing it with a Windows client, the 2nd line contains there a lot more data, and gets then followed by a WRITE on the server. Here with the Linux client, the server does not WRITE to the stream.

So I'm guessing the Linux client should send some more data initially?

Does this help?

[SteveSyfuhs]

You're going to need traces from the auth stack. It's likely failing there before it writes anything to the stream. The messages written to the stream also don't make a ton of sense. They're too short to be negotiate messages themselves, so there is no token present.

[wimme]

How do I get these auth traces? I've been trying with the KRB5_TRACE env variable, but it doesn't log anything when running our client .NET application.

If I run kinit with the env variable then it does output a lot of logging. (kinit works with our Windows domain accounts.) But I'm also trying to log in with a local Windows user, not necessarily a domain user (is Kerberos then even needed?).

I think I'm probably missing something obvious here, but I can't find any documentation on this...

[rzikm]

@filipnavara can you advise here please? I am not familiar with kerberos auth tracing

[wimme]

Any information regarding how this works is also welcome.

We're having users with Windows domain accounts (if there is a Windows domain) and with local Windows accounts of the Windows Server (just like how it works with NegotiateStream from Windows client machines). I'm hoping we don't need Kerberos with SPNs for this?

As a workaround we could transfer the credentials over our gRPC connection to the Windows Server and then verify these there. But ideally I'd prefer using the same flow as Windows clients.

[filipnavara]

KRB5_TRACE may not log anything if Kerberos is not used. I don't think that libgss (despite being part of the krb5 package) uses it for general tracing.

If NTLM is used then gss-ntlmssp package has to be installed on the system. It has its own logging capabilities by setting the GSSNTLMSSP_DEBUG environment variable and pointing it to a file. Notably some versions of Ubuntu shipped incompatible versions of OpenSSL (with disabled md4) and gss-ntlmssp, so just installing the package would not work out of the box. See https://github.com/dotnet/runtime/issues/67353#issuecomment-1085003764 for a workaround to that particular problem; it was fixed upstream, but I don't think Ubuntu ships the fixed packages.

[wimme]

I installed the gss-ntlmssp package and modified /usr/lib/ssl/openssl.cnf to enable the legacy provider.

With the GSSNTLMSSP_DEBUG environment variable I get the following:

[1712313101] ERROR: get_enterprise_name() @ src/gss_names.c:109 [1048576:0]
[1712313101] ERROR: string_split() @ src/gss_names.c:47 [1048576:0]
[1712313101] ERROR: string_split() @ src/gss_names.c:47 [1048576:0]
[1712313101] ALLOK: gssntlm_acquire_cred_from() @ src/gss_creds.c:400 [0:0]
[1712313101] ALLOK: gssntlm_release_name() @ src/gss_names.c:423 [0:0]

The underlying stream of NegotiateStream on the Ubuntu client doesn't write anything anymore.

[filipnavara]

What is the version of the gss-ntlmssp package on your system? The get_enterprise_name method doesn't even exist in the latest version, so it's hard to interpret the line number without having the correct source version.

[wimme]

I'm using Ubuntu Server 22.04 LTS, the version of gss-ntlmssp is 0.7.0-4build4. From the source code it seems to fail on parsing the domain name, but I get this exception when trying it with a local user (empty domain name) and a domain user ("company2"). Since the code has been heavily changed, it might be fixed in newer version...

[filipnavara]

local user (empty domain name)

FWIW the domain name is never really empty in NTLM. It's the machine name for local authentication. If you don't specify it explicitly during the initial authentication, then it's taken from the server's challenge message. You may want to try using different user name / domain combinations to see if it has any effect.

[wimme]

If I'm reading the code of get_enterprise_name in gss_names.c correctly then it seems like it expects a username@domain.x format. The ERROR lines go away when using username@machine.domain.local or username@domain.local (without providing a domain in NetworkCredential):

[1712671097] ALLOK: get_enterprise_name() @ src/gss_names.c:117 [0:0]
[1712671097] ALLOK: gssntlm_acquire_cred_from() @ src/gss_creds.c:400 [0:0]
[1712671097] ALLOK: gssntlm_release_name() @ src/gss_names.c:423 [0:0]

This still doesn't result in any negotiate messages though...

[wimme]

Are there any next steps I could try?

[filipnavara]

Are there any next steps I could try?

Well, you can try to enable the System.Net.Security tracing (https://github.com/filipnavara/httpclienttest/blob/9d3336a9aed944dbbdef8d0c4dc4007ea1163255/Program.cs#L33-L78), and you can try how it behaves with the ManagedNtlm override (https://github.com/AlexanderUsmanov/WcfClientNet8-issue/blob/57f5fbf3228ac18cd57211c8c913d13876deae57/WcfClientNet8/Program.cs#L30)

[wimme]

I'm not getting any logging with HttpEventListener (should I initialize this somewhere?), but the entire authentication flow fully works by using the ManagedNtlm override. I think we should make it configurable for end-users whether they want to use ManagedNtlm? Although I'm not sure why the default doesn't work for me.

[filipnavara]

I'm not getting any logging with HttpEventListener (should I initialize this somewhere?)

Yes, it needs to be initialized (and ideally disposed): https://github.com/filipnavara/httpclienttest/blob/9d3336a9aed944dbbdef8d0c4dc4007ea1163255/Program.cs#L15

I think we should make it configurable for end-users whether they want to use ManagedNtlm?

The configuration property _UseManagedNtlm (in .csproj) was added in .NET 9. The whole support for managed NTLM code path on Linux/macOS landed very late in the .NET 8 release cycle. It didn't receive enough testing to be a regular part of the product. On .NET 9 it's the new default for macOS/iOS and opt-in for Linux.

(The bar for backports to .NET 8 is set quite high, so not all the fixes for the managed NTLM code made it there yet; due to an unrelated bug on Apple platforms some backports may eventually be done, including the support for the .csproj switch.)

[wimme]

This generates indeed a lot of logging, it includes also our other https and grpc calls. I'm not sure if it reveals any info why the NTLM authentication fails? https://pastebin.com/nhJZGDvx

I'm not sure if it's worth investigating this further if it works with ManagedNtlm.

[filipnavara]

I'm not sure if it reveals any info why the NTLM authentication fails?

I don't see any meaningful info, unfortunately. That suggests we likely have a gap in the logging messages. It shows that InitializeSecurityContext was called which calls into the native GSSAPI library but the trace is lost after that.

Thanks for the log, anyway.

[rzikm]

Triage: putting it to Future for now. We will most likely need some more data and help from filipnavara to investigate further.

[norey]

Hey all, has there been any updates on this problem?

Essentially, I want to run the following snippet in Linux

...

Stream ret = client.GetStream();

TokenImpersonationLevel tkn = TokenImpersonationLevel.Identification;

NegotiateStream stm = new NegotiateStream(ret);

NetworkCredential creds = CredentialCache.DefaultNetworkCredentials;

stm.AuthenticateAsClient(creds, System.String.Empty, ProtectionLevel.EncryptAndSign, tkn);

MemoryStream ms = new MemoryStream();
BinaryWriter writer = new BinaryWriter(ms);

string data_stream = "0001000000FFFFFFFF01000000000000000C02000000...";
byte[] data = Convert.FromHexString(data_stream);

writer.Write(data);

var netWriter = new BinaryWriter(stm)

netWriter.Write(ms.ToArray());
...

It works fine on Windows, but fails on Linux (on the stm.AuthenticateAsClient() call ) as follows

$ dotnet --list-runtimes
Microsoft.AspNetCore.App 6.0.33 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 6.0.33 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.8 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

$ ./senddata
Unhandled exception. System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
 ---> System.ComponentModel.Win32Exception (0x80090302): Unknown error -2146893054
   --- End of inner exception stack trace ---
   at System.Net.Security.NegotiateStream.SendAuthResetSignalAndThrowAsync[TIOAdapter](Byte[] message, Exception exception, CancellationToken cancellationToken)
   at System.Net.Security.NegotiateStream.SendBlobAsync[TIOAdapter](Byte[] message, CancellationToken cancellationToken)
   at System.Net.Security.NegotiateStream.AuthenticateAsync[TIOAdapter](CancellationToken cancellationToken)
   at System.Net.Security.NegotiateStream.AuthenticateAsClient(NetworkCredential credential, ChannelBinding binding, String targetName, ProtectionLevel requiredProtectionLevel, TokenImpersonationLevel allowedImpersonationLevel)
   at Program.<Main>$(String[] args) in C:\Users\Administrator\source\repos\senddata\Program.cs:line 24
Aborted

Target framework for the project is .NET 8.0, compiled in visual studio 2022.