Timeout errors - Githubissues

abemedia commented 3 years ago

I'm trying to set up a repo with v1.11.0 but keep getting the following error:

{
    "level": "error",
    "rid": "c112o158nids72mvqcn0",
    "github_event_type": "issue_comment",
    "github_delivery_id": "1c27cb00-7db3-11eb-9400-cea1ebb29542",
    "error": "failed to get pull request theappnest/baustelle#1: Get \"https://api.github.com/repos/theappnest/baustelle/pulls/1\": could not refresh installation id 14084299's token: could not get access_tokens from GitHub API for installation ID 14084299: dial tcp 140.82.121.5:443: i/o timeout\ngithub.com/palantir/bulldozer/server/handler.(*IssueComment).Handle\n\t/Users/Adam/go/pkg/mod/github.com/palantir/bulldozer@v1.10.1/server/handler/issue_comment.go:56\ngithub.com/palantir/go-githubapp/githubapp.Dispatch.Execute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:56\ngithub.com/palantir/go-githubapp/githubapp.(*scheduler).safeExecute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:183\ngithub.com/palantir/go-githubapp/githubapp.QueueAsyncScheduler.func1\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:257\nruntime.goexit\n\t/usr/local/Cellar/go/1.16/libexec/src/runtime/asm_amd64.s:1371",
    "time": "2021-03-05T13:08:02.586685627Z",
    "message": "Unexpected error handling webhook"
}

Any ideas what could be causing this?

asvoboda commented 3 years ago

Hey @abeMedia,

Can you share your config file (without the secrets of course)?

Are you running bulldozer behind any sort of proxy or MITM setup? The follow error suggests that bulldozer can't reach github.com

dial tcp 140.82.121.5:443: i/o timeout

abemedia commented 3 years ago

It's running on Lambda. The weird thing is occasionally it works but more often than not I get the above error or this one:

{
    "level": "error",
    "rid": "c1151nd47s8s72qhf5eg",
    "github_event_type": "pull_request",
    "github_delivery_id": "12e5ca40-7dc9-11eb-893c-7ce1c5dfe6d6",
    "error": "failed to get pull request theappnest/baustelle#11: Get \"https://api.github.com/repos/theappnest/baustelle/pulls/11\": could not refresh installation id 14084299's token: could not get access_tokens from GitHub API for installation ID 14084299: net/http: TLS handshake timeout\ngithub.com/palantir/bulldozer/server/handler.(*PullRequest).Handle\n\t/Users/Adam/go/pkg/mod/github.com/palantir/bulldozer@v1.11.0/server/handler/pull_request.go:61\ngithub.com/palantir/go-githubapp/githubapp.Dispatch.Execute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:56\ngithub.com/palantir/go-githubapp/githubapp.(*scheduler).safeExecute\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:183\ngithub.com/palantir/go-githubapp/githubapp.QueueAsyncScheduler.func1\n\t/Users/Adam/go/pkg/mod/github.com/palantir/go-githubapp@v0.5.0/githubapp/scheduler.go:257\nruntime.goexit\n\t/usr/local/Cellar/go/1.16/libexec/src/runtime/asm_amd64.s:1371",
    "time": "2021-03-05T15:41:31.164081196Z",
    "message": "Unexpected error handling webhook"
}

The config:

github:
  web_url: "https://github.com"
  v3_api_url: "https://api.github.com"
  app:
    integration_id: 123
    webhook_secret: <redacted>
    private_key: <redacted>
options:
  configuration_path: .github/bulldozer.yml

There's no server config as I'm proxying the requests from lambda directly to the underlying goji.Mux.

abemedia commented 3 years ago

Just for context here is how I'm running it on Lambda:


package main

import (
    "context"
    "encoding/json"
    "log"
    "os"
    "reflect"
    "strings"
    "unsafe"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/awslabs/aws-lambda-go-api-proxy/httpadapter"
    "github.com/kelseyhightower/envconfig"
    "github.com/palantir/bulldozer/server"
    "github.com/palantir/go-baseapp/baseapp"
)

var gojiLambda *httpadapter.HandlerAdapter

func init() {
    c, err := server.ParseConfig(nil)
    if err != nil {
        log.Fatal(err)
    }

    // set config from env vars
    if err = envconfig.Process("bulldozer", c); err != nil {
        log.Fatal(err)
    }

    s, err := server.New(c)
    if err != nil {
        log.Fatal(err)
    }

    // get unexported struct field "base" to access the underlying router
    v := reflect.ValueOf(s).Elem().FieldByName("base").Elem()
    base := (*baseapp.Server)(unsafe.Pointer(v.UnsafeAddr()))

    gojiLambda = httpadapter.New(base.Mux())
}

func handler(ctx context.Context, req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {
    return gojiLambda.ProxyWithContext(context.Background(), req)
}

func main() {
    lambda.Start(handler)
}

asvoboda commented 3 years ago

We would need to dig in a little more to double check whats failing here. It does seem like something is not correctly wired up for the http handler, however I don't have a good environment to test with lambda to be certain.

However, a word of caution: bulldozer makes good use of a request cache to avoid hitting rate limits and other issues when interacting with many events from the Github API. While it might be possible to run the app entirely on Lambda, you might find that the app hits these limits more frequently.

abemedia commented 3 years ago

It seems there's other issues running on Lambda as well. Sometimes you can see the hook coming in in the logs but nothing happens. My assumption was that there's some async stuff going on in the scheduler, rather than directly on the webhook request, and the Lambda gets frozen after the request completes.

shresthaujjwal commented 2 years ago

@abeMedia Where you able to get this to work with lambda ?

palantir / bulldozer

Timeout errors #224