simo5
commented
9 months ago Can you please provide actual logs with the build you made?
Closed yevheniilavrenchuk closed 8 months ago
simo5
commented
9 months ago Can you please provide actual logs with the build you made?
simo5
commented
9 months ago FWIW, we build gssntlmssp in Fedora against OpenSSL 3.0 with no problems, and .NET is also using that afaik, so I need info specific to your build.
yevheniilavrenchuk
commented
9 months ago There are no logs from the app side, it is just an error: Response status code does not indicate success: 401 (Unauthorized).
We are upgrading from .NET6 to .NET8, faced that issue in our Alpine image. The way we are doing it is building gss-ntlmssp library separately and putting the library with the config into our image. With Alpine 3.17 it worked but stopped with Alpine 3.18.
Here is how I put the library file and a config into our image. Library is picked from the stage in Dockerfile which I described earlier:
...
COPY --from=gssntlmssp /usr/local/lib/gssntlmssp/gssntlmssp.so /usr/local/lib/gssntlmssp/gssntlmssp.so
COPY ./tools/mechntlmssp.conf /usr/etc/gss/mech.d/mechntlmssp.conf
...Here is a config file content:
# Mechanism Name Object Identifier Shared Library Path Other Options
gssntlmssp_v1 1.3.6.1.4.1.311.2.2.10 /usr/local/lib/gssntlmssp/gssntlmssp.soHere is a full log from docker build command:
yevheniilavrenchuk
commented
9 months ago I've found the solution. For those who are facing the same, for me this helped.
When you are building gss-ntlmssp library specifically for .NET 8 and Alpine 3.18, there is another package that needs to be installed additionally, it is krb5or krb5-libs.
My application was silent, only 401 error was thrown, but when I tried Invoke-RestMethod cmdlet inside pwsh that is preinstalled in .NET 8 Docker SDK image, I received this error:
Cannot load library libgssapi_krb5.so.2
Error: Error loading shared library libgssapi_krb5.so.2: No such file or directory
Invoke-RestMethod: Response status code does not indicate success: 401 (Unauthorized).After I installed krb5-libs package, error has gone and I've received 200 Status code with proper response message.
P.S. @simo5 thank you for quick response!
yevheniilavrenchuk
commented
9 months ago Hi @simo5
I'm reopening it again since I faced the problem that authentication is fixed, but it stops working after some number of requests to the server.
With a help of strace utility I managed to trace the chain a bit. My application is respecting mech.ntlmssp.conf which I put to /usr/etc/gss/mech.d/ path, there is a line which leads to gssntlmssp.so library file (/usr/local/lib/gssntlmssp/gssntlmssp.so), then looks like there is an instruction to use /usr/lib/ossl-modules/legacy.so library, since this is the last file my app is reading. After some successful requests, it throws exact the same 401 error.
I checked with Powershell as well, the same behavior:
$creds = Get-Credential LiveUser
for($i = 0; $i -lt 100; $i++){
irm "https://some-external-service.com/GPService/Tenants(DefaultTenant)/Companies(Fabrikam,%20Inc.)/TestCompany/Request" -Credential $creds -Verbose -Debug
}Here is a Dockerfile for building gss-ntlmssp library for Alpine 3.18:
FROM alpine:3.18
RUN apk add --no-cache git curl
RUN apk add --no-cache make m4 autoconf automake gcc g++ krb5-dev openssl-dev gettext-dev
RUN apk add --no-cache libtool libxml2 libxslt libunistring-dev zlib-dev samba-dev pkgconf
RUN git clone https://github.com/gssapi/gss-ntlmssp
WORKDIR gss-ntlmssp
RUN autoreconf -f -i
RUN ./configure --without-manpages --disable-nls
RUN make installHere are build logs of gss-ntlmssp library for Alpine 3.18:
Can you please help me with that? Thank you in advance!
simo5
commented
9 months ago Build logs are completely useless to debug a runtime issue. Is the .NET code being used as a client or as a server?
At the very least you would need to raise log levels and provide the HTTP server logs to see if there is any indication of what goes wrong.
What is the peer?
If this is server code where are credentials stored? Is this using winbindd integration or are you providing a password file?
If this is client code, how are you providing credentials? Is the server a standalone machine or a domain member?
yevheniilavrenchuk
commented
9 months ago Build logs are completely useless to debug a runtime issue. Is the .NET code being used as a client or as a server?
This is a client that is on .NET8 trying to reach Windows Server 2016.
At the very least you would need to raise log levels and provide the HTTP server logs to see if there is any indication of what goes wrong.
I did it, there are no logs in IIS WebServer on the destination, there are logs only appearing that the authentication failed with no reason for the user who is using NTLM authentication, so auth process is not reaching the application I guess. Interesting fact, that successful authentication is indicated by one user (LiveUser) and failed by another (Administrator): Successful log event by LiveUser:
- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
<Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
<EventID>4672</EventID>
<Version>0</Version>
<Level>0</Level>
<Task>12548</Task>
<Opcode>0</Opcode>
<Keywords>0x8020000000000000</Keywords>
<TimeCreated SystemTime="2024-02-14T06:43:32.305625300Z" />
<EventRecordID>77353216</EventRecordID>
<Correlation />
<Execution ProcessID="652" ThreadID="2440" />
<Channel>Security</Channel>
<Computer>SOMECOMPUTER</Computer>
<Security />
</System>
- <EventData>
<Data Name="SubjectUserSid">S-1-5-21-4135345634-3273676857-328331361-2109</Data>
<Data Name="SubjectUserName">LiveUser</Data>
<Data Name="SubjectDomainName">SOMEDOMAIN</Data>
<Data Name="SubjectLogonId">0x435bb0</Data>
<Data Name="PrivilegeList">SeSecurityPrivilege SeBackupPrivilege SeRestorePrivilege SeTakeOwnershipPrivilege SeDebugPrivilege SeSystemEnvironmentPrivilege SeLoadDriverPrivilege SeImpersonatePrivilege SeDelegateSessionUserImpersonatePrivilege SeEnableDelegationPrivilege</Data>
</EventData>
</Event>Failed log event by Administrator:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-5478-4994-A5BA-3E3B0328C30D}" />
<EventID>4625</EventID>
<Version>0</Version>
<Level>0</Level>
<Task>12544</Task>
<Opcode>0</Opcode>
<Keywords>0x8010000000000000</Keywords>
<TimeCreated SystemTime="2024-02-14T06:44:14.185681100Z" />
<EventRecordID>77353246</EventRecordID>
<Correlation />
<Execution ProcessID="652" ThreadID="2440" />
<Channel>Security</Channel>
<Computer>SOMECOMPUTER</Computer>
<Security />
</System>
<EventData>
<Data Name="SubjectUserSid">S-1-0-0</Data>
<Data Name="SubjectUserName">-</Data>
<Data Name="SubjectDomainName">-</Data>
<Data Name="SubjectLogonId">0x0</Data>
<Data Name="TargetUserSid">S-1-0-0</Data>
<Data Name="TargetUserName">Administrateur</Data>
<Data Name="TargetDomainName">SOMEDOMAIN.COM</Data>
<Data Name="Status">0xc000006d</Data>
<Data Name="FailureReason">%%2313</Data>
<Data Name="SubStatus">0xc0000064</Data>
<Data Name="LogonType">3</Data>
<Data Name="LogonProcessName">NtLmSsp</Data>
<Data Name="AuthenticationPackageName">NTLM</Data>
<Data Name="WorkstationName">B_12</Data>
<Data Name="TransmittedServices">-</Data>
<Data Name="LmPackageName">-</Data>
<Data Name="KeyLength">0</Data>
<Data Name="ProcessId">0x0</Data>
<Data Name="ProcessName">-</Data>
<Data Name="IpAddress">80.66.88.212</Data>
<Data Name="IpPort">0</Data>
</EventData>
</Event>What is the peer? It is Windows Server 2016 with NTLM\Windows local user authentication, if I understood your question correctly.
If this is server code where are credentials stored? Is this using winbindd integration or are you providing a password file? If this is client code, how are you providing credentials?
The test Powershell script I provided you, is just using default cmdlet to store credentials, then NTLM\Windows authentication used, I guess. The code that is in .NET app is using default System.Net.Http.HttpClient. Credentials are managed by System.Net.Primitives by NetworkCredential class. It provides credentials for password-based authentication schemes such as basic, digest, NTLM, and Kerberos authentication. Credentials themselves are encrypted and stored in MSSQL database. The same code was working on Alpine 3.17 with .NET6. In this scope what is changed is we migrated to .NET8 and moved to Alpine 3.18 Docker image.
Is the server a standalone machine or a domain member?
The destination server is standalone one, not binded to the domain.
P.S. I don't know if it will help, but auth requests are failing after 58 consecutive requests.
simo5
commented
9 months ago 0xC0000064 User logon with misspelled or bad user account
Perhaps you should use Administrator instead of Administrateur ?
yevheniilavrenchuk
commented
9 months ago I beg a pardon, failed auth log isn't related to this. This one was about someone trying to login using wrong username, indeed.
What is stranger, I noticed that there are no failed login attempts recorded on the destination server made by the application. Maybe by some chance you found something else catchy?
yevheniilavrenchuk
commented
8 months ago Hi @simo5 I tried Alpine 3.19 and the problem persists as well. the latest version It is working is Alpine 3.16 for now. I also found that there is a possibility to switch on debug logs on gss-ntlmssp library itself.
Here is what debug logs looks like. 55 and 56 requests are ok, starting from 57 is stops working and throws 401. Another thing is that when I restart PowerShell session or restart an application then it starts working again until 57th request. Looks like some pool of connections becoming full or something...:
55
[1708401633] ALLOK: parse_user_name() @ src/gss_names.c:232 [0:0]
[1708401633] ALLOK: gssntlm_acquire_cred_from() @ src/gss_creds.c:501 [0:0]
[1708401633] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401633] ALLOK: gssntlm_import_name_by_mech() @ src/gss_names.c:355 [0:0]
[1708401633] ALLOK: gssntlm_import_name_by_mech() @ src/gss_names.c:355 [0:0]
[1708401633] ALLOK: gssntlm_init_sec_context() @ src/gss_sec_ctx.c:246 [1:0]
[1708401633] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401633] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401634] ALLOK: gssntlm_import_name_by_mech() @ src/gss_names.c:355 [0:0]
[1708401634] ALLOK: gssntlm_cli_auth() @ src/gss_auth.c:277 [0:0]
[1708401634] ALLOK: gssntlm_init_sec_context() @ src/gss_sec_ctx.c:416 [0:0]
[1708401634] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401634] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401634] ALLOK: gssntlm_delete_sec_context() @ src/gss_sec_ctx.c:483 [0:0]
56
[1708401635] ALLOK: parse_user_name() @ src/gss_names.c:232 [0:0]
[1708401635] ALLOK: gssntlm_acquire_cred_from() @ src/gss_creds.c:501 [0:0]
[1708401635] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401635] ALLOK: gssntlm_import_name_by_mech() @ src/gss_names.c:355 [0:0]
[1708401635] ALLOK: gssntlm_import_name_by_mech() @ src/gss_names.c:355 [0:0]
[1708401635] ALLOK: gssntlm_init_sec_context() @ src/gss_sec_ctx.c:246 [1:0]
[1708401635] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401635] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401635] ALLOK: gssntlm_import_name_by_mech() @ src/gss_names.c:355 [0:0]
[1708401635] ALLOK: gssntlm_cli_auth() @ src/gss_auth.c:277 [0:0]
[1708401635] ALLOK: gssntlm_init_sec_context() @ src/gss_sec_ctx.c:416 [0:0]
[1708401635] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401635] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401636] ALLOK: gssntlm_delete_sec_context() @ src/gss_sec_ctx.c:483 [0:0]
57
[1708401637] ALLOK: parse_user_name() @ src/gss_names.c:232 [0:0]
[1708401637] ERROR: gssntlm_acquire_cred_from() @ src/gss_creds.c:475 [851968:1314127875]
[1708401637] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401637] ALLOK: gssntlm_display_status() @ src/gss_err.c:139 [0:0]
58
[1708401638] ALLOK: parse_user_name() @ src/gss_names.c:232 [0:0]
[1708401638] ERROR: gssntlm_acquire_cred_from() @ src/gss_creds.c:475 [851968:1314127875]
[1708401638] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401638] ALLOK: gssntlm_display_status() @ src/gss_err.c:139 [0:0]
59
[1708401639] ALLOK: parse_user_name() @ src/gss_names.c:232 [0:0]
[1708401639] ERROR: gssntlm_acquire_cred_from() @ src/gss_creds.c:475 [851968:1314127875]
[1708401639] ALLOK: gssntlm_release_name() @ src/gss_names.c:633 [0:0]
[1708401639] ALLOK: gssntlm_display_status() @ src/gss_err.c:139 [0:0] Hi @simo5 I managed to get it work, but it is not a good solution, I think it needs to be fixed anyway..
Steps to get it running on Alpine 3.18:
.include = /etc/ssl/openssl.cnf[provider_sect] default = default_sect legacy = legacy_sect
[default_sect] activate = 1
[legacy_sect] activate = 1
This is already fixed in 1.1.0 with automatic loading of the legacy provider, what you are doing in the above is to regress to a version that requires manual activation of the legacy provider.
I am not aware of any issues automatically loading the legacy provider. The logs you show simply tell that gssntlmssp was not able to find the credentials you asked it to look up, but the log level is not sufficient to establish what caused the failure.
yevheniilavrenchuk
commented
8 months ago As you can see, for us the old method of loading legacy provider works, but the new one is not, so something definitely doesn't work correctly. By some chance, do you have some advises for us, how to fix that? Are you going to look into it, or we are alone with it?
Thanks for your replies so far.
simo5
commented
8 months ago If you can run the code in a debugger and find out what breaks loading the legacy provider I can then make a fix for it. But I do not have a system setup to build or run .NET stuff so until I have a way to reproduce the issue outside of your particular environment I am of little utility.
laszlojau
commented
8 months ago Hi @simo5, we are facing the exact same issue - auth fails after 58 tries when the legacy provider is loaded via the library. If I also load the legacy provider in advance by updating the openssl config file, it fails after 57 tries, which might suggest it's loaded for every run and hits some kind of limit? I'm not sure how to debug C libraries, however, I've built 2 container images for simple testing and created a repo with instructions. Link to repo: gssntlmssp-alpine-debug Link to images: container/package/gssntlmssp-alpine-debug
gssntlmssp-alpine-debug:1.2.0 is v1.2.0 of the library built without any changes - this starts failing auth after 58 tries until you restart the container or start another PowerShell processgssntlmssp-alpine-debug:1.2.0-patched is v1.2.0 of the library without the OpenSSL 3-related changes (I've created a patch file to remove those for testing) and with the additional OpenSSL config for loading the legacy provider - this one doesn't failIf you run the container following the provided instructions, it prompts you for a URL, a username and a password and tries to call the URL 100 times and breaks the loop and exits with an error in case it fails. I'm also happy to do my own debugging if you're able to provide some pointers on how to do that.
yevheniilavrenchuk
commented
8 months ago Hi @laszlojau, thanks for your message. Just to confirm, your patch is working for our solution as well.
simo5
commented
8 months ago @laszlojau @yevheniilavrenchuk can one of you tell me if #99 make this work on alpine without explicitly loading the legacy provider via config file?
I am confused by the "57" times thing, I am not aware of any limit to the number of times you load/unload a provider on a brand new context.
In any case the PR should load the legacy only once on a long lived context, and should address whatever underlying OprenSSL problem may have caused it.
laszlojau
commented
8 months ago Thank you @simo5, I tried it and it seems to work perfectly. Ran it 500 times and it worked every time. No errors in the debug logs either.
simo5
commented
8 months ago Thanks for confirming, merged the fix, release soon.
yevheniilavrenchuk
commented
8 months ago Thank you so much, @simo5. It is working for us as well, confirmed!
Hi. I faced the problem that if I'll build the package for Alpine 3.18, it won't work. Here is a Dockerfile I'm using for that:
The build is working, but when I'm using gssntlmssp.so library, I'm receiving 403 errors during authentication requests. Previously and on our older images it was working perfectly fine with the same authentication endpoint. I've got to know that when OpenSSL 3 became default package in Alpine 3.18, then it stopped working. Reference: https://stackoverflow.com/questions/76185769/gssapi-operation-failed-with-error-an-unsupported-mechanism-was-requested-for https://bugs.launchpad.net/ubuntu/+source/gss-ntlmssp/+bug/1995915
Maybe someone has a solution for that? Thanks in advance!