pagerduty_team_membership can not be deleted when user is in use in pagerduty_schedule

[mambax]

Hi all, I discovered something rather un-smooth and wanted to share. Maybe the fix is straight-forward for you.

What I report makes totally sense of course. Yet the error handling of the provider or the API is very misleading.

Terraform Version

❯ terraform version
Terraform v0.14.4
+ provider registry.terraform.io/pagerduty/pagerduty v1.8.0

Affected Resource(s)

• pagerduty_team_membership
• pagerduty_schedule

Terraform Configuration Files

Before

resource "pagerduty_team_membership" "TMTMTM" {
  user_id = data.pagerduty_user.UUUU.id
  team_id = pagerduty_team.TTTT.id
  role    = "manager"
}

resource "pagerduty_schedule" "EEE" {
  name        = format("%s-%s_Schedule_Eng1", local.organisation, local.teamT)
  time_zone   = "Europe/Berlin"
  description = local.managedByTerraForm

  layer {
    name                         = "engineer1"
    start                        = "2021-01-18T17:30:00+01:00"
    rotation_virtual_start       = "2021-01-18T17:30:00+01:00"
    rotation_turn_length_seconds = 604800
    users = [
      data.pagerduty_user.IIIII.id,
      data.pagerduty_user.UUUU.id,
      data.pagerduty_user.LLLLL.id,
      data.pagerduty_user.KKKK.id
    ]
  }
}

After

Before

resource "pagerduty_team_membership" "TMTMTM" {
  user_id = data.pagerduty_user.A_NEW_USER_OOOOO.id        # <----------- Note, user changed
  team_id = pagerduty_team.TTTT.id
  role    = "manager"
}

resource "pagerduty_schedule" "EEE" {
  name        = format("%s-%s_Schedule_Eng1", local.organisation, local.teamT)
  time_zone   = "Europe/Berlin"
  description = local.managedByTerraForm

  layer {
    name                         = "engineer1"
    start                        = "2021-01-18T17:30:00+01:00"
    rotation_virtual_start       = "2021-01-18T17:30:00+01:00"
    rotation_turn_length_seconds = 604800
    users = [
      data.pagerduty_user.IIIII.id,
      data.pagerduty_user.UUUU.id,
      data.pagerduty_user.LLLLL.id,
      data.pagerduty_user.KKKK.id
    ]
  }
}

Panic Output

To reflect the change above TF must replace:

# pagerduty_team_membership.TMTMTM must be replaced
-/+ resource "pagerduty_team_membership" "TMTMTM" {
      ~ id      = "PYQV0AO:P1V3E98" -> (known after apply)
      ~ user_id = "PYQV0AO" -> "P261LLO" # forces replacement
        # (2 unchanged attributes hidden)
    }

.... yes ....

pagerduty_team_membership.TMTMTM: Destroying... [id=PYQV0AO:P1V3E98]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 10s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 20s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 30s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 40s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 50s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 1m0s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 1m10s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 1m20s elapsed]
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
Stopping operation...
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 1m30s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 1m40s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 1m50s elapsed]
pagerduty_team_membership.TMTMTM: Still destroying... [id=PYQV0AO:P1V3E98, 2m0s elapsed]

In words

I wanted to replace one user with another. For that I just changed the team membership (replace one member with another). But, in the schedule the old member was still used.

Expected Behavior

The API/Provider should output, that the object can not be destroyed because it is referenced in a schedule, or even in which schedule. Can not destroy pagerduty_team_membership.TMTMTM, entity is in use in schedule pagerduty_schedule.EEE

Actual Behavior

See output. Destroy ends in an endless position (or at least I did not wait long enough).

Steps to Reproduce

Described above.

Best regards, Dom

[stmcallister]

Thanks for reporting this, Dom! It looks like the retry logic is a little greedy. I'll put his on the list to address.

[thekbb]

This is still an issue in v1.9.6. @mambax, your assumption on actual behavior is correct. I can confirm that it will fail to destroy.

[deniojunior-hotmart]

Hi! I'm using provider version 2.11.0 and I'm getting a similar error when trying to delete an user from the schedule and team_membership.

I tries to delete the team_membership before deleting the user from the schedule, so it fails because the user belongs to a schedule.

I have tried to explicit the dependency in the schedule resource:

resource "pagerduty_schedule" "primary" {
[...]
depends_on = [pagerduty_user.users, pagerduty_team_membership.team_membership]

But it didn't work and I'm still getting the error below

pagerduty_team_membership.team_membership[0]: Destroying... [id=PO21JXJ:P2A6JL1]
pagerduty_team_membership.team_membership[0]: Still destroying... [id=PO21JXJ:P2A6JL1, 10s elapsed]
pagerduty_team_membership.team_membership[0]: Still destroying... [id=PO21JXJ:P2A6JL1, 20s elapsed]
[...]
pagerduty_team_membership.team_membership[0]: Still destroying... [id=PO21JXJ:P2A6JL1, 2m0s elapsed]

Error: DELETE API call to https://api.pagerduty.com/teams/P2A6JL1/users/PO21JXJ failed 400 Bad Request. Code: 2001, Errors: [User cannot be removed as they belong to an escalation policy on this team], Message: Invalid Input Provided

[twix14]

The same behaviour is still happening on version 3.0.1 of the provider. The membership is only deleted if the user is the active on-call, otherwise it will fail with 400.

https://github.com/PagerDuty/terraform-provider-pagerduty/blob/daed960297d0982f2070b9791467ce43804bc3e9/pagerduty/resource_pagerduty_team_membership.go#L248

The call above only returns users set explicitly on the escalation policy and users which are currently on-call according to the set schedule.

This can be fixed by setting:

agerduty.ListOnCallOptions{UserIds: []string{userID}, Since: "Today", Until: "90d from today"}