Organizations Are Missing Webhooks

[BenEmdon]

From running the following query on production, it is evident that there are still a significant amount of Organizations that are missing webhooks:

pry(main)> Organization.where(webhook_id: nil).count
DEBUG -- :    (7.0ms)  SELECT COUNT(*) FROM "organizations" WHERE "organizations"."deleted_at" IS NULL AND "organizations"."webhook_id" IS NULL
=> 3838

This number excludes Organizations that have a webhook_id but who's webhooks are deleted or inactive.

The source of this problem

The problem occurred when we introduced webhooks back in https://github.com/education/classroom/pull/797, we forgot to backfill organizations created before the change with web hooks.

Why this is important

All Organizations require webhooks to perform starter code imports as we rely on the repository_import event.

Steps to ensure all Organizations have webhooks that are active

• [x] OrganizationWebhook model to manage multiple classrooms (#1692 #1694)
  • [x] change is_webhook_active column on Organization to reflect the last webhook delivered as a date (move this to OrganizationWebhook) (#1696)
  • [x] Backfill all Organizations to belong to a OrganizationWebhook (#1695)
• [x] model GitHubOrganizationWebbhook as a GitHubResource (#1702)
• [x] method around ensuring a webhook is present and valid (#1713)
  • make sure we don't hit API cache
• [x] migration to ensure every OrganizationWebhook has a valid webhook (#1718)
• [x] move the logic for creating a webhook from the Organization::Creator to GitHubOrganizationWebhook (#1716)
• [x] make the last_webhook_recieved attribute visible from stafftools (#1725)

~Steps to ensure an org is in good health~

• [x] define a method to check if an Organization meets all minimum requirements (#1724)
  • if webhooks are present and enabled
  • ~if a user in the org has all the scopes needed~
• ~disable classrooms that are in bad health~
• ~migration to remove OrganizationWebhook github_ids that aren't active.~

~Switch all webhooks to app created webhooks~

• ~migration to switch all OrganizationWebhook from user created webhooks to OAuth app generated webhooks~
• ~use GitHubClassroom.github_client to create organization webhooks instead of user clients~
• [x] revert #1724

Do what we can with what we have

• [x] Run ensure_webhook_is_active! on all OrganizationWebhooks https://github.com/education/classroom/blob/d5d7a9a78c5b35e372abcff73936b890574c8973/app/models/organization_webhook.rb#L78-L99

[BenEmdon]

Just ran the organization_webhook_health_migration I had been working on and got the following results:

• Number of organizations that don’t webhooks because are missing a user with the admin:org_hook scope: 3634
• Number of organizations that no longer exists: 126
• Number of organizations that could be reconfigured to have organization webhooks: 67

Next

We should notify classrooms that don’t have webhooks to authorize the right token.

[BenEmdon]

Possible Solution:

Right now, webhooks are dependant on users OAuth tokens. This isn't great because users can revoke their tokens, or leave the organization, leaving the tokens on record as invalid.

It turns out we can use the classroom OAuth app token to create webhooks:

from the docs

If we use the classroom client OAuth token to create webhooks then we can ensure that every organization has webhooks enabled.

[srinjoym]

@BenEmdon What do you think about renaming this issue to include "Importing starter code taking a long time" or something like that so people know that it's related to this problem? That could help redirect people here as they run into issues

[BenEmdon]

I'd like to keep the development track and incident reports separate so we can filter out the noise 😕 Maybe I should start a master issue for all the incidents?

[srinjoym]

That totally makes sense! I think we can open up another pinned issue just for the incidents and include some instructions on what to do if people hit this issue. Hopefully that makes it a bit easier to sort through

[BenEmdon]

Pinned #1778

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.

[BenEmdon]

From the docs

Interesting; hooks created by the OAuth app cannot be seen by users. And hooks created by users cannot be seen by the Oauth app.

[BenEmdon]

Update

So after investigating the API, it turns out that OAuth apps can't use their own token to register web hooks. 😔 This means that we will have to continue to use user tokens to register webhooks.

Steps forward

• [x] close #1839
• Go back to the plan about warning users about their classroom missing webhooks

[BenEmdon]

Update

Ran the organization_webhook_health_migration and now the number of classrooms missing webhooks is:

PRODUCTION [1] pry(main)> OrganizationWebhook.where(github_id: nil).count
=> 3674

The 3674 Organizations without Webhooks

These organizations are missing webhooks for one of two reasons:

1. The organization no longer has a user token with the admin:org_hook scope (as in the user left the organization).
2. The organization was deleted.

We can't do anything about case 2, but we can let each classroom know that they are missing a user with the the admin:org_hook scope.

[BenEmdon]

This issue could be resolved if Classroom became a GitHub app #2049

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.