POST API call to https://app.pagerduty.com/integration-slack/workspaces/myorgID/connections failed 403 Forbidden

maheshevizio commented 2 months ago

Hi there,

Not able to create the PagerDuty-cslack connection using the PagerDuty Provider. I tried with multiple user and API keys and have admin access but it doesn't work

Terraform Version

Terraform v1.7.5 on darwin_amd64

provider registry.terraform.io/hashicorp/aws v5.47.0
provider registry.terraform.io/hashicorp/local v2.5.1
provider registry.terraform.io/hashicorp/null v3.2.2
provider registry.terraform.io/hashicorp/random v3.6.1
provider registry.terraform.io/pagerduty/pagerduty v3.11.4.

Affected Resource(s)

Please list the resources as a list, for example:

resource "pagerduty_slack_connection"

If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

resource "pagerduty_slack_connection" "slack_link" {
  count = var.slack_channel_id == null ? 0 : 1
  source_id = pagerduty_service.service.id
  source_type = "service_reference"
  workspace_id = var.slack_workspace_id
  channel_id = var.slack_channel_id
  notification_type = "responder"
  provider = pagerduty
  config {
    events = [
      "incident.triggered",
      "incident.acknowledged",
      "incident.escalated",
      "incident.resolved",
      "incident.reassigned",
      "incident.annotated",
      "incident.unacknowledged",
      "incident.delegated",
      "incident.priority_updated",
      "incident.responder.added",
      "incident.responder.replied",
      "incident.status_update_published",
      "incident.reopened"
    ]
    #priorities = [data.pagerduty_priority.p1.id] #disable priotrities for now
    priorities = ["*"]
  }
}
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

Debug Output

TF_DEBUG=DEBUG option when tf apply was run https://github.com/maheshevizio/debugLogPD/blob/main/debugLogs

Panic Output

NA

Expected Behavior

What should have happened? The Pagerduty_Slack_connection resource should have been created

Actual Behavior

What actually happened? Its throws the 403 Forbidden error

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

terraform apply

Important Factoids

Are there anything atypical about your accounts that we should know? There are other slack channel connections that exist for the same workspace. I also tried with multiple API Key and User tokens

References

This is a private repo and I can't share the code

ermirry commented 3 weeks ago

I'm running into this exact issue using v3.12.2 of the PagerDuty provider. I've rotated my user-level API keys, made sure things were pasted correctly. As far as I can tell, everything is in place for this to work but I can't get past this 403.

ermirry commented 2 weeks ago

I've been tinkering with this since my last comment as I go back and forth with PagerDuty support troubleshooting this. I've unlinked and relinked my PagerDuty account to my Slack user, confirmed my workspace ID was correct (and tried a potential second one that PD support says is my actual workspace ID), linked a workspace in my profile under the Contact Information tab (and clicked the test button next to it to ensure I actually get a notification).

What I've been finding really weird is that I can accomplish this task no problem if I simply do it through the PD UI. I can see the channels I have access to when making a connection and create said connection successfully. I see PD incidents coming into my slack channel that I just connected as well. But for reasons still unknown to me, my terraform and my curl still give me a forbidden error. For transparency, here's the curl i'm running (taken from here):

curl --request POST \
  --url https://app.pagerduty.com/integration-slack/workspaces/$redacted_workspace_ID/connections \
  --header 'Accept: application/json' \
  --header 'Authorization: Token token=$redacted_user_level_api_token' \
  --header 'Content-Type: application/json' \
  --data '{
  "slack_connection": {
    "source_id": "$redacted_source_ID",
    "source_type": "service_reference",
    "channel_id": "$redacted_channel_ID",
    "notification_type": "responder",
    "config": {
      "events": [
        "incident.triggered",
        "incident.acknowledged",
        "incident.escalated",
        "incident.resolved",
        "incident.reassigned",
        "incident.annotated",
        "incident.unacknowledged",
        "incident.delegated",
        "incident.priority_updated",
        "incident.responder.added",
        "incident.responder.replied",
        "incident.status_update_published"
      ]
    }
  }
}'

I don't see why this would fail to execute but performing the same actions and testing their functionality is successful in the UI

ermirry commented 2 weeks ago

This is fixed for me now. Pagerduty Support managed to figure out that the url being used needs to be app.eu.pagerduty.com since my account is in the EU. I also had to specify priorities and urgency set to null in my curl otherwise it would complain about invalid parameters

ermirry commented 1 week ago

As for the terraform provider itself, the provider accepts a service_region attribute to ensure connections go to the EU which should fix this if your account is also in the EU like mine is

PagerDuty / terraform-provider-pagerduty