Webhooks triggered by GitHub 'release' hang Kestra

[johnsturgeon]

Describe the issue

I have the following flow:

id: deploy_tgfp_web_to_prod
namespace: tgfp.prod
tasks:
  - id: publish_prod_site
    type: io.kestra.plugin.fs.ssh.Command
    host: tgfp-prod.lan
    authMethod: PASSWORD
    username: root
    password: "{{ secret('PROXMOX_ROOT_PW')}}"
    commands: ["/opt/tgfp/update.sh"]

triggers:
  - id: github
    type: io.kestra.plugin.core.trigger.Webhook
    key: "tgfp_prod_deploy_4b17c7d03897"
    conditions:
      - type: io.kestra.plugin.core.condition.ExpressionCondition
        expression: "{{ trigger.body.action == 'released' }}"

Which I trigger from Github which sends the following payload:

{
  "action": "released",
  "release": {
    "url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/releases/173241126",
    "assets_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/releases/173241126/assets",
    "upload_url": "https://uploads.github.com/repos/TheGreatFootballPool/tgfp-web/releases/173241126/assets{?name,label}",
    "html_url": "https://github.com/TheGreatFootballPool/tgfp-web/releases/tag/v1.1.1",
    "id": 173241126,
    "author": {
      "login": "johnsturgeon",
      "id": 9746310,
      "node_id": "MDQ6VXNlcjk3NDYzMTA=",
      "avatar_url": "https://avatars.githubusercontent.com/u/9746310?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/johnsturgeon",
      "html_url": "https://github.com/johnsturgeon",
      "followers_url": "https://api.github.com/users/johnsturgeon/followers",
      "following_url": "https://api.github.com/users/johnsturgeon/following{/other_user}",
      "gists_url": "https://api.github.com/users/johnsturgeon/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/johnsturgeon/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/johnsturgeon/subscriptions",
      "organizations_url": "https://api.github.com/users/johnsturgeon/orgs",
      "repos_url": "https://api.github.com/users/johnsturgeon/repos",
      "events_url": "https://api.github.com/users/johnsturgeon/events{/privacy}",
      "received_events_url": "https://api.github.com/users/johnsturgeon/received_events",
      "type": "User",
      "site_admin": false
    },
    "node_id": "RE_kwDOIG4C_c4KU3Mm",
    "tag_name": "v1.1.1",
    "target_commitish": "main",
    "name": "v1.1.1",
    "draft": false,
    "prerelease": false,
    "created_at": "2024-09-03T14:53:13Z",
    "published_at": "2024-09-03T14:54:09Z",
    "assets": [

    ],
    "tarball_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/tarball/v1.1.1",
    "zipball_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/zipball/v1.1.1",
    "body": "## What's Changed\r\n* Added the Sentry Javascript Loader Script by @johnsturgeon in https://github.com/TheGreatFootballPool/tgfp-web/pull/105\r\n\r\n\r\n**Full Changelog**: https://github.com/TheGreatFootballPool/tgfp-web/compare/v1.1.0...v1.1.1",
    "mentions_count": 1
  },
  "repository": {
    "id": 544080637,
    "node_id": "R_kgDOIG4C_Q",
    "name": "tgfp-web",
    "full_name": "TheGreatFootballPool/tgfp-web",
    "private": false,
    "owner": {
      "login": "TheGreatFootballPool",
      "id": 152522430,
      "node_id": "O_kgDOCRdOvg",
      "avatar_url": "https://avatars.githubusercontent.com/u/152522430?v=4",
      "gravatar_id": "",
      "url": "https://api.github.com/users/TheGreatFootballPool",
      "html_url": "https://github.com/TheGreatFootballPool",
      "followers_url": "https://api.github.com/users/TheGreatFootballPool/followers",
      "following_url": "https://api.github.com/users/TheGreatFootballPool/following{/other_user}",
      "gists_url": "https://api.github.com/users/TheGreatFootballPool/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/TheGreatFootballPool/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/TheGreatFootballPool/subscriptions",
      "organizations_url": "https://api.github.com/users/TheGreatFootballPool/orgs",
      "repos_url": "https://api.github.com/users/TheGreatFootballPool/repos",
      "events_url": "https://api.github.com/users/TheGreatFootballPool/events{/privacy}",
      "received_events_url": "https://api.github.com/users/TheGreatFootballPool/received_events",
      "type": "Organization",
      "site_admin": false
    },
    "html_url": "https://github.com/TheGreatFootballPool/tgfp-web",
    "description": "Web container for the Great Football Pool",
    "fork": false,
    "url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web",
    "forks_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/forks",
    "keys_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/teams",
    "hooks_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/hooks",
    "issue_events_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/issues/events{/number}",
    "events_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/events",
    "assignees_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/assignees{/user}",
    "branches_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/branches{/branch}",
    "tags_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/tags",
    "blobs_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/languages",
    "stargazers_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/stargazers",
    "contributors_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/contributors",
    "subscribers_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/subscribers",
    "subscription_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/subscription",
    "commits_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/contents/{+path}",
    "compare_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/merges",
    "archive_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/downloads",
    "issues_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/issues{/number}",
    "pulls_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/labels{/name}",
    "releases_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/releases{/id}",
    "deployments_url": "https://api.github.com/repos/TheGreatFootballPool/tgfp-web/deployments",
    "created_at": "2022-10-01T15:39:57Z",
    "updated_at": "2024-09-03T14:53:17Z",
    "pushed_at": "2024-09-03T14:54:09Z",
    "git_url": "git://github.com/TheGreatFootballPool/tgfp-web.git",
    "ssh_url": "git@github.com:TheGreatFootballPool/tgfp-web.git",
    "clone_url": "https://github.com/TheGreatFootballPool/tgfp-web.git",
    "svn_url": "https://github.com/TheGreatFootballPool/tgfp-web",
    "homepage": null,
    "size": 7534,
    "stargazers_count": 1,
    "watchers_count": 1,
    "language": "Python",
    "has_issues": true,
    "has_projects": true,
    "has_downloads": true,
    "has_wiki": true,
    "has_pages": false,
    "has_discussions": false,
    "forks_count": 0,
    "mirror_url": null,
    "archived": false,
    "disabled": false,
    "open_issues_count": 7,
    "license": {
      "key": "gpl-3.0",
      "name": "GNU General Public License v3.0",
      "spdx_id": "GPL-3.0",
      "url": "https://api.github.com/licenses/gpl-3.0",
      "node_id": "MDc6TGljZW5zZTk="
    },
    "allow_forking": true,
    "is_template": false,
    "web_commit_signoff_required": false,
    "topics": [

    ],
    "visibility": "public",
    "forks": 0,
    "open_issues": 7,
    "watchers": 1,
    "default_branch": "main",
    "custom_properties": {

    }
  },
  "organization": {
    "login": "TheGreatFootballPool",
    "id": 152522430,
    "node_id": "O_kgDOCRdOvg",
    "url": "https://api.github.com/orgs/TheGreatFootballPool",
    "repos_url": "https://api.github.com/orgs/TheGreatFootballPool/repos",
    "events_url": "https://api.github.com/orgs/TheGreatFootballPool/events",
    "hooks_url": "https://api.github.com/orgs/TheGreatFootballPool/hooks",
    "issues_url": "https://api.github.com/orgs/TheGreatFootballPool/issues",
    "members_url": "https://api.github.com/orgs/TheGreatFootballPool/members{/member}",
    "public_members_url": "https://api.github.com/orgs/TheGreatFootballPool/public_members{/member}",
    "avatar_url": "https://avatars.githubusercontent.com/u/152522430?v=4",
    "description": null
  },
  "sender": {
    "login": "johnsturgeon",
    "id": 9746310,
    "node_id": "MDQ6VXNlcjk3NDYzMTA=",
    "avatar_url": "https://avatars.githubusercontent.com/u/9746310?v=4",
    "gravatar_id": "",
    "url": "https://api.github.com/users/johnsturgeon",
    "html_url": "https://github.com/johnsturgeon",
    "followers_url": "https://api.github.com/users/johnsturgeon/followers",
    "following_url": "https://api.github.com/users/johnsturgeon/following{/other_user}",
    "gists_url": "https://api.github.com/users/johnsturgeon/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/johnsturgeon/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/johnsturgeon/subscriptions",
    "organizations_url": "https://api.github.com/users/johnsturgeon/orgs",
    "repos_url": "https://api.github.com/users/johnsturgeon/repos",
    "events_url": "https://api.github.com/users/johnsturgeon/events{/privacy}",
    "received_events_url": "https://api.github.com/users/johnsturgeon/received_events",
    "type": "User",
    "site_admin": false
  }
}

Additional Information

• I can run the flow manually and it takes about 50 seconds

• Github gives me the following error:

  

• I have to power cycle my LXC (in Proxmox) for it to recover. I see no execution in the execution list. So no indication that the flow even attempts to run.

• CPU and RAM (8G) spike to 100% when the webhook hits Kestra.

Environment

• Kestra Version:0.18.4
• Operating System (OS/Docker/Kubernetes): Debian 12 / Docker
• Java Version (if you don't run kestra in Docker):

[johnsturgeon]

Found some more information that I think will be very helpful:

I did figure out what I was doing wrong. I had moved the flow from one namespace to another but I did not update the "Github" webhook.

So, when the web hook was pointing to: https://kestra.mydomain.com/api/v1/executions/webhook/tgfp.dev/deploy_tgfp_web_to_prod/tgfp_prod_deploy_mykey it hung

But when I fixed github to point to the correct namespace, it worked: https://kestra.mydomain.com/api/v1/executions/webhook/tgfp.prod/deploy_tgfp_web_to_prod/tgfp_prod_deploy_mykey

No clue why pointing github at a webhook in the wrong namespace would cause it to hang instead of just returning back 'not found' or something...