500 internal error on status event

[LeoQuote]

Traceback:

AssertionError: null
  File "starlette/exceptions.py", line 73, in __call__
    raise exc from None
  File "starlette/exceptions.py", line 62, in __call__
    await self.app(scope, receive, sender)
  File "starlette/routing.py", line 590, in __call__
    await route(scope, receive, send)
  File "starlette/routing.py", line 208, in __call__
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 41, in app
    response = await func(request)
  File "fastapi/routing.py", line 133, in app
    raw_response = await dependant.call(**values)
  File "kodiak/main.py", line 96, in webhook_event
    queue=redis_webhook_queue, event_name=github_event, payload=event
  File "kodiak/queue.py", line 279, in handle_webhook_event
    await status_event(queue, StatusEvent.parse_obj(payload))
  File "kodiak/queue.py", line 144, in status_event
    pr_results = await asyncio.gather(*pr_requests)
  File "kodiak/queries/__init__.py", line 1020, in get_open_pull_requests
    session=self.session, installation_id=self.installation_id
  File "kodiak/queries/__init__.py", line 1288, in get_headers
    session=session, installation_id=installation_id
  File "kodiak/queries/__init__.py", line 1278, in get_token_for_install
    assert res.status_code < 300

I cannot see res as it is filterred, I checked permission and it's all match . This is a self-hosted instance for github enterprise.

related issue: https://github.com/chdsbd/kodiak/issues/562

[chdsbd]

If you configure Sentry with your installation you'll get better error reporting.

It's okay for some HTTP requests to fail. Do all outgoing requests fail like this?

[LeoQuote]

Installation event looks good, but status event looks bad, I dont think it should be like this.

[LeoQuote]

I did installed sentry in this project, but the res object is "filtered", I cannot see detailed response body...

[chdsbd]

It's hard for me to debug this without more information. You should be able to disable filtering on the Sentry website.

Also, did you configure the API roots for your GitHub installation? https://kodiakhq.com/docs/self-hosting

[LeoQuote]

yes, I've configured api root for kodiak, I tried to disable filtering on sentry, but no luck , it only shows res = <Response [401 Unauthorized]> , no detailed response body, maybe we need a log level debug to print all response or raise error for that?

[chdsbd]

What kind of traceback do you get if you run this command?

docker exec -it $your_kodiak_container_name_here .venv/bin/kodiak list-installs

[LeoQuote]

certainly

root@test-kodiak-78f9754f7b-sdf8n:/var/app# .venv/bin/kodiak list-installs 
Traceback (most recent call last):
  File ".venv/bin/kodiak", line 33, in <module>
    sys.exit(load_entry_point('kodiak', 'console_scripts', 'kodiak')())
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/var/app/kodiak/cli.py", line 44, in list_installs
    res.raise_for_status()
  File "/var/app/.venv/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://github.myghe.com/api/v3/app/installations

[chdsbd]

@LeoQuote

Try the following and see what the GitHub API is returning for the error API response.

JWT=$(.venv/bin/kodiak create-jwt)
curl  -H "Authorization: Bearer $JWT" https://api.github.com/app

[LeoQuote]

strange enough. got an 200

< HTTP/1.1 200 OK
< Server: GitHub.com

{
  "id": 12,
  "slug": "kodiak",
  "node_id": "MDM6QXBwMTI=",
  "owner": {
    "login": "sa",
    "id": 50,
    "node_id": "MDEyOk9yZ2FuaXphdGlvbjUw",
    "avatar_url": "https://avatars.github.mydomain.com/u/50?",
    "gravatar_id": "",
    "url": "https://github.mydomain.com/api/v3/users/sa",
    "html_url": "https://github.mydomain.com/sa",
    "followers_url": "https://github.mydomain.com/api/v3/users/sa/followers",
    "following_url": "https://github.mydomain.com/api/v3/users/sa/following{/other_user}",
    "gists_url": "https://github.mydomain.com/api/v3/users/sa/gists{/gist_id}",
    "starred_url": "https://github.mydomain.com/api/v3/users/sa/starred{/owner}{/repo}",
    "subscriptions_url": "https://github.mydomain.com/api/v3/users/sa/subscriptions",
    "organizations_url": "https://github.mydomain.com/api/v3/users/sa/orgs",
    "repos_url": "https://github.mydomain.com/api/v3/users/sa/repos",
    "events_url": "https://github.mydomain.com/api/v3/users/sa/events{/privacy}",
    "received_events_url": "https://github.mydomain.com/api/v3/users/sa/received_events",
    "type": "Organization",
    "site_admin": false
  },
  "name": "kodiak",
  "description": "A GitHub bot to automatically update and merge GitHub PRs",
  "external_url": "https://kodiak.mydomain.com",
  "html_url": "https://github.mydomain.com/github-apps/kodiak",
  "created_at": "2021-09-10T06:26:23Z",
  "updated_at": "2021-09-10T06:26:23Z",
  "permissions": {
    "administration": "read",
    "checks": "write",
    "contents": "write",
    "issues": "write",
    "members": "read",
    "metadata": "read",
    "pull_requests": "write",
    "statuses": "write",
    "workflows": "write"
  },
  "events": [
    "check_run",
    "check_suite",
    "pull_request",
    "pull_request_review",
    "pull_request_review_comment",
    "push",
    "status"
  ],
  "installations_count": 2
}

[chdsbd]

If you try the same request with https://api.github.com/app/installations, do you get an error?

[LeoQuote]

no error at all. the response is a bit big so I'd rather not to paste it.

[chdsbd]

@LeoQuote Can you provide the full URL you used in your request?

[LeoQuote]

certainly, but the ghe instance is only available with vpn access,

curl -v -H "Authorization: Bearer $JWT" https://github.mydomain.com/api/v3/app/installations

[chdsbd]

@LeoQuote Is that URL identical to the one in the error message you received?

https://github.myghe.com/api/v3/app/installations

If so I'm really not sure what's going on.

[LeoQuote]

yes, it is identical, wierd thing is , I tried to use python shell to execute code ,and it succeeded.

>>> from kodiak import app_config as conf   
>>> url = conf.v3_url("/app/installations")                                                                                                                                                                                                                                                                   
>>> url
'https://github.mydomain.com/api/v3/app/installations'
>>> from kodiak.queries import generate_jwt, get_token_for_install
>>> app_token = generate_jwt(
...         private_key=conf.PRIVATE_KEY, app_identifier=conf.GITHUB_APP_ID
...     )
>>> 
>>> app_token
'eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpYXQiOjE2MzE1OTM4MzYsImV4cCI6MTYzMTU5NDQzNiwiaXNzIjoiMTIifQ.CCQFuUptlq1aHFTtSslAv2T0gwEgcOIIjknqUWpQf3BeXPgPkkrMw3yp4iCIqCt9uK5aH59asnUfG7zd9GoxHTUwZ__TsOWUJqeOXqy-giYB0mR9Xo9EAQNF_BUIHOWO4DJa_32o3elhChn9OsoF2I9MDfA03jVPfSEMTvUfYVrIs7n7zVz7kmhOYzKZXXny9CbPQxHgyBH68wMCfp9n2ZEoxBkrcTTNaK-mbDPBnZkkJGkVB_8rqnzWIsk60CHG0yTXN6Z_Jqck2l-wUukMlrBjOlhA0f4047Bb2LTPXIOknIamSjkRnTKiy8CXWCeLknmptqAkGnok5FTGhDfnjQ'
>>> conf.GITHUB_APP_ID
'12'
>>> headers = dict(
...         Accept="application/vnd.github.machine-man-preview+json",
...         Authorization=f"Bearer {app_token}",
...     )
>>> 
>>> import requests
>>> res = requests.get(url, headers=headers)
>>> res.status_code
200

what could be the reason? the github configs are set by environment varibles. It looks fine.

[LeoQuote]

use list_installs function

>>> from kodiak.cli import list_installs
>>> list_installs
<click.core.Command object at 0x7facc2e9e610>
>>> list_installs()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/var/app/.venv/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/var/app/kodiak/cli.py", line 44, in list_installs
    res.raise_for_status()
  File "/var/app/.venv/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://github.intra.douban.com/api/v3/app/installations

use python shell to execute

>>> from pathlib import Path
>>> from typing import Any, Dict, List
>>>
>>> import click
>>> import requests
>>> from httpx import AsyncClient
>>> 
>>> from kodiak import app_config as conf
>>> from kodiak.config import V1
>>> from kodiak.queries import generate_jwt, get_token_for_install
>>> app_token = generate_jwt(
... private_key=conf.PRIVATE_KEY, app_identifier=conf.GITHUB_APP_ID
... )
>>> results: List[Dict[str, Any]] = []
>>> headers = dict(
... Accept="application/vnd.github.machine-man-preview+json",
... Authorization=f"Bearer {app_token}",
... )
>>> url = conf.v3_url("/app/installations")
>>> res = requests.get(url, headers=headers)
>>> res.status_code
200