Some PIV users experiencing access issues on specific pages

[timcosgrove]

Describe the defect

Some PIV users are reporting that specific pages show them an access denied page. These are pages that they previously had access to.

To Reproduce

This unfortunately is unreproduceable by other people. The problem is specific to certain users and certain pages. See also notes under Additional Context.

Randi Hecht:

1. Goes to https://prod.cms.va.gov - gets 'Access Denied'

Sara Torres:

1. Goes to https://prod.cms.va.gov/resources/about-electronic-health-information-sharing-at-va, hits 'Edit' (she does have access to edit the page)
2. Gets 'Access Denied'

Beth Potts:

1. Help Desk has reached out to determine the steps.
2. What they found is that Beth Potts is a content admin, so she should have access to everything.

AC / Expected behavior

Users should be able to access pages they have role-based access to.

Screenshots

Additional context

The general pattern is:

1. User is logged in via PIV.
2. User goes to a page that they have role-based access to.
3. User is redirected to https://prod.cms.va.gov/saml_login
4. https://prod.cms.va.gov/saml_login is actually not meant to be accessible by an authenticated user, and they DO remain logged in, so the user gets 'Access Denied'.
5. The user continues to be logged in and can access other pages as normal without logging back in.

Additional information that has come back from working with users experiencing the problem:

• The problem began to be reported once the certificate renewal progress began and when there was an update to SOCKS. Not clear if there's a relation to these problems.
• For one user, logging into the CMS via PIV in a different browser eliminated the problem (problem existed in Chrome; did not experience it in MS Edge)
• For the same user, logging out in Chrome and then logging in with their 'developer' login (i.e. not using PIV, but user/pass instead) eliminated the problem.
• However, when the same user above logged back in with PIV, they experienced the problem again.

• A user that was not able to access /user/295 (their personal user profile page) was able to access /user/295/edit This indicates it is not a Drupal-level access issue; in other words, the user has legitimate access to the content they're trying to view.

  Troubleshooting Steps

• Identify the people we know who are experiencing the issue.
• Grab time with those team members who are experiencing issue to have them replicate the issue.
• Synthesize findings into this ticket and regroup. Specifically, the person's name, browser, and URL of page getting blocked.

Acceptance Criteria

• [ ] The root cause of why users are incorrectly getting access denied messages.

[timcosgrove]

One error type I see in the logs indicates a user is being returned from logon.iam.va.gov without an appropriate adUPN attribute set

Drupal\simplesamlphp_auth\Exception\SimplesamlphpAttributeException: Error in simplesamlphp_auth.module: no valid "adUPN" attribute set. in Drupal\simplesamlphp_auth\Service\SimplesamlphpAuthManager->getAttribute() (line 284 of /var/www/cms/docroot/modules/contrib/simplesamlphp_auth/src/Service/SimplesamlphpAuthManager.php). [dd.trace_id=9223372036854775807 dd.span_id=4039230175985604495]

https://prod.cms.va.gov/admin/reports/dblog/event/4608084 (may be removed; not sure how long these are retained).

This has happened multiple times; it is not unique to the above. However, this doesn't seem clearly associated with the reported problem.

[gracekretschmer-metrostar]

All users known to have this issue have had the issue resolved by using a different browser, so we are going to move this to the parking lot until we get more information about the problem and/or hear from more users experiencing the issue.

Full discussion here: https://dsva.slack.com/archives/CT4GZBM8F/p1723747595417829

[gracekretschmer-metrostar]

Erika had an idea that it's users who have the translations set up and, specifically, if that module is the underlying issue.

Beth Potts has also reported experiencing the same issue.

[ian-sears]

CMS HD report from an Editor: Name: Linda.Wondra@va.gov Browser: ??? will ask and update here URL being accessed: ??? will ask and update here Also asking how recently she cleared her broswer cahce and if trying the "other" browser resolved it. Also asking how recently she was able to use this URL normally.

[ian-sears]

Name: megan.zehnder@va.gov Meghan writes: As of today, I’m still experiencing the issue.

Oddly, yesterday I encountered the issue much less frequently. I was able to go back into Edit mode in a Drupal page multiple times without clearing my cache and re-entering my PIV each time. But when I worked on a different post today, I hit the “Access denied” screen every time I tried to go back into Edit mode.

Here are some observations that may or may not be helpful!

• Yesterday (low incidence of the issue), my work in Drupal was almost exclusively on this unpublished step-by-step page. I was mostly working on text and images. I wasn’t editing links.
• Today (high incidence of the issue), I triggered the “Access denied” screen right away after I edited link text in this published benefits detail page.
• Sometime last week, I changed a setting in my Microsoft Edge to clear my cookies and history every time I exit Microsoft Edge. Before last week, I had ["microsoft Edge setting "Clear chosen data for Internet Explorer and Internet Explorer mode every time you exit Microsoft Edge"] set to off.

[TroyCMSSupport]

@timcosgrove - @ian-sears @gracekretschmer-metrostar A new piece of information regarding Access Denied -- A common thread: Editors are clicking on 'Home' in the breadcrumb trail at the top of the page. This leads them to the Home page - forcing them to log in again - THIS is when the Access Denied message appears... Editors report that the functionality of this changed 3 or so months ago. The Home link should take them to the same page as clicking on their user name at the top of the page...

Additionally from @ian-sears - As an Editor, when I am editing, within the last two months when I click on the breadcrumb "Home". It does not take me to the home page for "me"... it instead asks me to log in again. Workarounds that the Editors have discovered... "Click the VA logo". (or) click your own email address in the menu bar and then "View Profile". This takes them back to what they consider to be their own "home" page (restricted to the sections they have access to edit in WorkBench...)

[ian-sears]

Although unable to shoot a video of this, the Editor Clifford.Coy@va.gov provides this (if verbose) clear text sequence of how he experiences this:

Sure, happy to help. I have the knowledge to do what you are asking, in terms of a video, but not the set up for it. I would have to do some odd video call on teams, and then record that, then compress it…..that is a little more work than I want to do. I can tell you that I have went a head and played around a bit more and I get this response after doing pretty much anything in the page. So, if I clear the browser cache…no issues, I can log in like normal. I use this link: https://prod.cms.va.gov/section/vha/vet-centers/boston-vet-center I select the “edit” button for the Main vet center page section. That takes me here: https://prod.cms.va.gov/node/3597/edit?destination=/section/vha/vet-centers/boston-vet-center which is a logon for my PIV. I click that and I am in the edit cms for my page.

After that, I can do what I want, but as soon as I navigate away or click anywhere else and then try to get back into the edit page, it takes me to the piv log in page and I get nothing but access denied, until I clear the cache again. I tested this by just closing the window, “x” ing out as the kids call it. Then using the above listed link https://prod.cms.va.gov/section/vha/vet-centers/boston-vet-center get back in. From my cms edit page: https://prod.cms.va.gov/node/3597/edit?destination=/section/vha/vet-centers/boston-vet-center&check_logged_in=1 I have navigated to the knowledge base: https://prod.cms.va.gov/help, my profile: https://prod.cms.va.gov/user/3136, after logging out. If I navigate anywhere or click anywhere and then try to get back in without first clearing the cache, it blocks me. I only tested this on edge, not any other browsers.

-Cliff

From VAHELP-7971 VA CMS Help Desk

[ian-sears]

UPDATE: I just followed up with an Editor that had been experiencing this issue (reported) a little over a month ago. ANSWER: Yes. Although it continues to say Access Denied and force a re-login, it is ONLY when she clicks on the breadscrumb for "Home".

I am asking her if it occurs in any other way, and will report here if that is thee case..

[gracekretschmer-metrostar]

Another report of this issue here: https://dsva.slack.com/archives/CDHBKAL9W/p1730129417455099

[TroyCMSSupport]

@timcosgrove @ian-sears @gracekretschmer-metrostar - Megan Zehnder has reported in Slack that she is still experiencing this issue.

I tried several ways to reproduce, but still have been unable to.

For my next trick, I will set my account to be just like hers and try again to reproduce... but she is a content admin, assigned to ALL sections -- not really different from

[gracekretschmer-metrostar]

Pull in for sprint 24. Edmund will pull in more logging around login and search the logs for errors.

[TroyCMSSupport]

@gracekretschmer-metrostar @timcosgrove @ian-sears Looks like we've had 11 specific users report the 'Access Denied' screen over the course of 14 Jira tickets (some users created multiple tickets).

The 'typical' workaround has included: clearing browser cache, using alternate browser (these have worked with some success, but often not a permanent solution).

Another finding from an Office Hours meeting is that using the 'Content' button to find content may trigger the 'Access Denied' message - although, I replicated Megan Z's settings on a test account last week, accessed the same content she was denied access to, and I could not replicate the issue. She is a Content Admin, but most reports are from VAMC Editors.

In that same office hours meeting, it was suggested by an editor that navigating to the Content page by clicking on the Editor's email address at the top of the page seems to bypass the 'Access Denied' scenario -- Again, I have not been able to prove/disprove, or otherwise replicate that scenario.

I am wondering if this may have to do with browser VERSION. - No evidence of that either.

[TroyCMSSupport]

@ian-sears @gracekretschmer-metrostar @timcosgrove I have documented as - many as I can identify - of the related Jira tickets:

https://hitssllc-my.sharepoint.com/:x:/p/troy_griffin/ERNitSMWjqZIslT786477m8BjYW1TvsG3JgvXisQ-ht3yA?e=3tuPyN&nav=MTVfezI4Njc0QURDLUNENUUtNDhDNC1BRUYyLUI2MjRCNjdBRDQyOH0

[edmund-dunn]

I found this for Linda.Wondra@va.gov

Oct 31 23:07:37 ip-10-247-33-27 drupal: https://prod.cms.va.gov|1730416057|access denied|10.247.32.79|https://prod.cms.va.gov/saml_login|https://prod.cms.va.gov/user/2147|4|2147||Path: /saml_login. Drupal\Core\Http\Exception\CacheableAccessDeniedHttpException: This route can only be accessed by anonymous users. in Drupal\Core\Routing\AccessAwareRouter->checkAccess() (line 115 of /var/www/cms/docroot/core/lib/Drupal/Core/Routing/AccessAwareRouter.php).