lithnet / access-manager

Access Manager provides web-based access to local admin (LAPS) passwords, BitLocker recovery keys, and just-in-time administrative access to Windows computers in a modern, secure, and user-friendly way.
Other
238 stars 20 forks source link

Requests to one of configured domains get stuck at 'Submitting request' #208

Closed ivoo13 closed 1 year ago

ivoo13 commented 1 year ago

Hello,

We've got two domains configured for Just-In-Time access. Configuration was working fine for a few months, but recently requests for servers in the second domain are getting stuck with 'Submitting request...' prompt. There is no errors/timeouts in browser or in access-manager-webapp.log. At the same time requests to the first domain (in which Access Manager is hosted) work fine.

Both domains' statuses in Access Manager seem fine: obraz obraz

Each of those problematic requests seems to end on 'Attempting to create S4U AuthorizationContext against server' and there are no entries after that part (nothing about any kind of success/failure/response etc.). See in the attached log: Lithnet-issue-anonymized.txt

The only recent configuration change I am aware of was configuring three roles in Authorization Rules, but they also worked as expected during our tests for a few days before the problem first appeared. Now, if I sign in as user from the problematic domain and click the 'Roles' tab, the page doesn't load (Similarly to picking server from the second domain, logs end on 'Attempting to create S4U AuthorizationContext against server' entry).

We tried to find any clues about the issue on domain controllers, but to no avail. Our network team also verified that there is no traffic blocked between the access manager server and domain controllers.

We are currently running Access Manager version 2.0.9422.0 (updated shortly after issue appeared).

Do you have any suggestions on what might have caused such issue or what else we can do to diagnose it?

ryannewington commented 1 year ago

May 2023 windows update broke this

We're waiting on a fix from Microsoft

https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-22H2#3114msgdesc

The only solution is to uninstall the may and June cumulative updates for now

ivoo13 commented 1 year ago

Thank you. That's probably it. Our Access Manager is running on Windows server 2022 and indeed it had KB5026370 installed some time before we noticed the problem for the first time. (It's odd that MS didn't add info about that issue on https://support.microsoft.com/en-us/topic/may-9-2023-kb5026370-os-build-20348-1726-8c5dc605-d613-46ea-9232-1425cfc91d62, but I probably wouldn't realize it's the cause anyway.)

That also clarifies why we've been seeing increased CPU utilization on that server. I suspected that it might be related, but I didn't find any proof for that before.

I don't think our client will agree to remove security updates, so we will probably just stick to AD group-based access to servers in the affected domain for now. Hopefully MS fixes it this month.

ryannewington commented 1 year ago

@ivoo13 the symptoms you describe are all part of the known issue, including the CPU usage. Microsoft just haven't described the issue very well.

There are many Access Manager customers experiencing this issue, and it is confirmed that this is the cause.

When logging issues with Microsoft, it's always easier to try and reproduce the issue with their own software. Otherwise its very difficult to convince them the problem lies in their software, and not that of a third party. The "effective access" screen uses the same S4U remote procedure call that AMS does. Hence they both hang and spin the CPU. So it's been reported with that as the reproduction steps, rather than asking MS to install and configure AMS to trigger the issue.

If you have a Microsoft premier/unified contract, its probably worth opening a case and letting them know you are suffering from this issue. This helps speed things up and also so they can let you know as soon as they have a fix available.

There seems to be a fix out for the windows 11 variant of this issue, hopefully that means the server variant is solved soon as well.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

ryannewington commented 1 year ago

This should be fixed in this month's updates

https://www.bleepingcomputer.com/news/microsoft/microsoft-fixes-windows-bug-causing-file-explorer-freezes/

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs.

ivoo13 commented 1 year ago

I can confirm that installing KB5028171 (Windows Server 2022) on the server resolved our issues with the second domain. https://learn.microsoft.com/en-us/windows/release-health/status-windows-server-2022#computing-effective-access-might-not-show-results

Thanks for help!