PagerDuty - System Annotations Missing from Access Requests

[goakley]

We are seeing an issue with the PagerDuty request plugin v14.3.3 working with Teleport v14.3.19.

We have configured our Teleport roles with the following annotation:

spec:
  allow:
    request:
      annotations:
        pagerduty_services: [...]

The exact contents of that annotation array depends on the role itself - we have a couple dozen roles associated with various PagerDuty services.

The PagerDuty plugin is configured to look at that annotation:

[pagerduty]
notify_service = "pagerduty_notify_service"
services = "pagerduty_services"

However, when making any request for any role on any resources in Teleport, the plugin reports the following "error" (logged at the debug level, the most verbose level available):

DEBU  Cannot proceed further. Request is missing any annotations request_id:XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX request_op:put request_state:PENDING pagerduty/app.go:242

This is difficult to debug, because we can't find a way to view the raw access request data itself (tsh requests show doesn't provide all the details of a request). There are no other logs from the plugin when a request is created. As far as we can tell, these requests are from users with roles containing the annotation mentioned above.

Looking at the source code, it looks like this check happens before any other processing. That means that the plugin isn't even looking for a specific annotation - it's reporting that there are absolutely no annotations in the access request sent to the plugin. Is this a bug? Is there a way we can get more information about these access requests, in order to verify that Teleport did in fact ignore the annotations we have on these roles?

[hugoShaka]

This means there is no annotation on the access request. You can validate this by running tctl get access_request (note, this command was added in a recent version, make sure to have the latest tctl for your major locally and to update teleport to the latest v14.

If teleport is not adding any annotation to the AR, this is very likely not a plugin issue but a role configuration issue.

Make sure that:

• You are running Telpeort 14.3.20 as we backported several fixes in the request annotations logic that could cause the behaviour you are observing.
• ensure that the role that allows the access request is also the one adding the annotation.

  [!IMPORTANT] You cannot use two different roles, one to allow requesting and one to add the annotations The role should look like.

  spec:
  allow:
  request:
  annotations:
  pagerduty_services: [...]
  roles: ['dba']

• the user has the role

[goakley]

That fix does look like it will resolve the issue. Thanks for the backport!