search

python / bugs.python.org

Meta-issue tracker for bugs.python.org
12 stars 5 forks source link

Timeout delivering webhook #23

Closed Mariatta closed 2 years ago

Mariatta commented 5 years ago

Webhook to bpo pull_request with this payload keep resulting in Service timeout error.

I think it was taking too long to process this particular delivery? I've tried re-delivering the same payload three times, all returning Service timeout.

delivery id 6efba906-09dd-11e9-9b45-e06201fc4ec9 timestamp: 2018-12-27 05:43:46 

{
  "ref": "refs/heads/master",
  "before": "34ae04f74dcf4ac97d07c3e82eaf8f619d80cedb",
  "after": "c6c7237272499b2c528acb5f62601421f329e92a",
  "created": false,
  "deleted": false,
  "forced": false,
  "base_ref": null,
  "compare": "https://github.com/python/cpython/compare/34ae04f74dcf...c6c723727249",
  "commits": [
    {
      "id": "c6c7237272499b2c528acb5f62601421f329e92a",
      "tree_id": "a1f00c8cec2833a64b45ce63d3c161ba2aec7441",
      "distinct": true,
      "message": "bpo-20182: AC convert remaining functions/methods in _hashopenssl.c (GH-9213)",
      "timestamp": "2018-12-27T15:43:43+02:00",
      "url": "https://github.com/python/cpython/commit/c6c7237272499b2c528acb5f62601421f329e92a",
      "author": {
        "name": "Tal Einat",
        "email": "taleinat+github@gmail.com",
        "username": "taleinat"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "username": "web-flow"
      },
      "added": [

      ],
      "removed": [

      ],
      "modified": [
        "Modules/_hashopenssl.c",
        "Modules/clinic/_hashopenssl.c.h"
      ]
    }
  ],
  "head_commit": {
    "id": "c6c7237272499b2c528acb5f62601421f329e92a",
    "tree_id": "a1f00c8cec2833a64b45ce63d3c161ba2aec7441",
    "distinct": true,
    "message": "bpo-20182: AC convert remaining functions/methods in _hashopenssl.c (GH-9213)",
    "timestamp": "2018-12-27T15:43:43+02:00",
    "url": "https://github.com/python/cpython/commit/c6c7237272499b2c528acb5f62601421f329e92a",
    "author": {
      "name": "Tal Einat",
      "email": "taleinat+github@gmail.com",
      "username": "taleinat"
    },
    "committer": {
      "name": "GitHub",
      "email": "noreply@github.com",
      "username": "web-flow"
    },
    "added": [

    ],
    "removed": [

    ],
    "modified": [
      "Modules/_hashopenssl.c",
      "Modules/clinic/_hashopenssl.c.h"
    ]
  },
  "repository": {
    "id": 81598961,
    "node_id": "MDEwOlJlcG9zaXRvcnk4MTU5ODk2MQ==",
    "name": "cpython",
    "full_name": "python/cpython",
    "private": false,
    "owner": {
      "name": "python",
      "email": "",
      "login": "python",
      "id": 1525981,
      "node_id": "MDEyOk9yZ2FuaXphdGlvbjE1MjU5ODE=",
      "avatar_url": "https://avatars3.githubusercontent.com/u/1525981?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/python",
      "html_url": "https://github.com/python",
      "followers_url": "https://api.github.com/users/python/followers",
      "following_url": "https://api.github.com/users/python/following{/other_user}",
      "gists_url": "https://api.github.com/users/python/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/python/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/python/subscriptions",
      "organizations_url": "https://api.github.com/users/python/orgs",
      "repos_url": "https://api.github.com/users/python/repos",
      "events_url": "https://api.github.com/users/python/events{/privacy}",
      "received_events_url": "https://api.github.com/users/python/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "html_url": "https://github.com/python/cpython",
    "description": "The Python programming language",
    "fork": false,
    "url": "https://github.com/python/cpython",
    "forks_url": "https://api.github.com/repos/python/cpython/forks",
    "keys_url": "https://api.github.com/repos/python/cpython/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/python/cpython/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/python/cpython/teams",
    "hooks_url": "https://api.github.com/repos/python/cpython/hooks",
    "issue_events_url": "https://api.github.com/repos/python/cpython/issues/events{/number}",
    "events_url": "https://api.github.com/repos/python/cpython/events",
    "assignees_url": "https://api.github.com/repos/python/cpython/assignees{/user}",
    "branches_url": "https://api.github.com/repos/python/cpython/branches{/branch}",
    "tags_url": "https://api.github.com/repos/python/cpython/tags",
    "blobs_url": "https://api.github.com/repos/python/cpython/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/python/cpython/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/python/cpython/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/python/cpython/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/python/cpython/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/python/cpython/languages",
    "stargazers_url": "https://api.github.com/repos/python/cpython/stargazers",
    "contributors_url": "https://api.github.com/repos/python/cpython/contributors",
    "subscribers_url": "https://api.github.com/repos/python/cpython/subscribers",
    "subscription_url": "https://api.github.com/repos/python/cpython/subscription",
    "commits_url": "https://api.github.com/repos/python/cpython/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/python/cpython/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/python/cpython/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/python/cpython/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/python/cpython/contents/{+path}",
    "compare_url": "https://api.github.com/repos/python/cpython/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/python/cpython/merges",
    "archive_url": "https://api.github.com/repos/python/cpython/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/python/cpython/downloads",
    "issues_url": "https://api.github.com/repos/python/cpython/issues{/number}",
    "pulls_url": "https://api.github.com/repos/python/cpython/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/python/cpython/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/python/cpython/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/python/cpython/labels{/name}",
    "releases_url": "https://api.github.com/repos/python/cpython/releases{/id}",
    "deployments_url": "https://api.github.com/repos/python/cpython/deployments",
    "created_at": 1486754631,
    "updated_at": "2018-12-27T09:13:20Z",
    "pushed_at": 1545918224,
    "git_url": "git://github.com/python/cpython.git",
    "ssh_url": "git@github.com:python/cpython.git",
    "clone_url": "https://github.com/python/cpython.git",
    "svn_url": "https://github.com/python/cpython",
    "homepage": "https://www.python.org/",
    "size": 262946,
    "stargazers_count": 21608,
    "watchers_count": 21608,
    "language": "Python",
    "has_issues": false,
    "has_projects": false,
    "has_downloads": true,
    "has_wiki": false,
    "has_pages": false,
    "forks_count": 8094,
    "mirror_url": null,
    "archived": false,
    "open_issues_count": 951,
    "license": {
      "key": "other",
      "name": "Other",
      "spdx_id": "NOASSERTION",
      "url": null,
      "node_id": "MDc6TGljZW5zZTA="
    },
    "forks": 8094,
    "open_issues": 951,
    "watchers": 21608,
    "default_branch": "master",
    "stargazers": 21608,
    "master_branch": "master",
    "organization": "python"
  },
  "pusher": {
    "name": "taleinat",
    "email": "taleinat+github@gmail.com"
  },
  "organization": {
    "login": "python",
    "id": 1525981,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjE1MjU5ODE=",
    "url": "https://api.github.com/orgs/python",
    "repos_url": "https://api.github.com/orgs/python/repos",
    "events_url": "https://api.github.com/orgs/python/events",
    "hooks_url": "https://api.github.com/orgs/python/hooks",
    "issues_url": "https://api.github.com/orgs/python/issues",
    "members_url": "https://api.github.com/orgs/python/members{/member}",
    "public_members_url": "https://api.github.com/orgs/python/public_members{/member}",
    "avatar_url": "https://avatars3.githubusercontent.com/u/1525981?v=4",
    "description": "Repositories related to the Python Programming language"
  },
  "sender": {
    "login": "taleinat",
    "id": 532281,
    "node_id": "MDQ6VXNlcjUzMjI4MQ==",
    "avatar_url": "https://avatars3.githubusercontent.com/u/532281?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/taleinat",
    "html_url": "https://github.com/taleinat",
    "followers_url": "https://api.github.com/users/taleinat/followers",
    "following_url": "https://api.github.com/users/taleinat/following{/other_user}",
    "gists_url": "https://api.github.com/users/taleinat/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/taleinat/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/taleinat/subscriptions",
    "organizations_url": "https://api.github.com/users/taleinat/orgs",
    "repos_url": "https://api.github.com/users/taleinat/repos",
    "events_url": "https://api.github.com/users/taleinat/events{/privacy}",
    "received_events_url": "https://api.github.com/users/taleinat/received_events",
    "type": "User",
    "site_admin": false
  },
  "distinct_commits": [
    {
      "id": "c6c7237272499b2c528acb5f62601421f329e92a",
      "tree_id": "a1f00c8cec2833a64b45ce63d3c161ba2aec7441",
      "distinct": true,
      "message": "bpo-20182: AC convert remaining functions/methods in _hashopenssl.c (GH-9213)",
      "timestamp": "2018-12-27T15:43:43+02:00",
      "url": "https://github.com/python/cpython/commit/c6c7237272499b2c528acb5f62601421f329e92a",
      "author": {
        "name": "Tal Einat",
        "email": "taleinat+github@gmail.com",
        "username": "taleinat"
      },
      "committer": {
        "name": "GitHub",
        "email": "noreply@github.com",
        "username": "web-flow"
      },
      "added": [

      ],
      "removed": [

      ],
      "modified": [
        "Modules/_hashopenssl.c",
        "Modules/clinic/_hashopenssl.c.h"
      ]
    }
  ],
  "ref_name": "master"
}
ewdurbin commented 5 years ago

GitHub has a timeout for web hook delivery, at 10s.

My best guess is that something on the new server is causing the handler to take longer than that. It's totally possible that the switch from the built in bespoke roundup-server to gunicorn is causing floods of requests to push GitHub past it's breaking point... or the handler is just slow as it's performed synchronously.

I've provisioned more worker processes for the cpython tracker's gunicorn stack, but we should probably investigate the performance of the handler itself to see if there's anything we can do to speed it up.

Aside from that... finding something to asynchronously process the hooks might be a better course of action, but roundup doesn't have any asynchronous task queue available.

ewdurbin commented 5 years ago

Also worth noting that each of the deliveries probably succeeded in doing whatever the handler would do with them... GitHub just gave up.

ezio-melotti commented 2 years ago

We now migrated to GitHub, so this can be closed.