error: invalid memory address or nil pointer

[sreejuGit]

I have configured bulldozer to use in our private git repo. which has api urls like https://github.org/api/v3

i have configured and setup github app and also able to receive successful response (200 response) when i manually close any pull request in my repo. But for all other subscribed events i'm receiving 500 error. For example, if i try to update pull request by adding a comment like " Please merge this pull request! "

I'm receiving 500 error as below. Any help please?

{"level":"error","rid":"bl8h1t6h2atu39dlula0","method":"POST","path":"/api/github/hook","error":"panic: runtime error: invalid memory address or nil pointer dereference\nruntime.gopanic
---
"message":"Unhandled error while serving route"}
{"level":"info","rid":"bl8h1t6h2atu39dlula0","method":"POST","path":"/api/github/hook","status":500,"size":33,"elapsed":54.2862,"time":"2019-08-12T07:10:44.447629100Z","message":"http_request"}

[bluekeyes]

Hey @sreejuGit, can you share the rest of the stack trace (the error parameter in the logs)? It looks like it was cut off in what you shared and it's hard to tell where the panic originated from without it.

[sreejuGit]

{"level":"info","rid":"bl90oknb548975k834mg","github_event_type":"issue_comment","github_delivery_id":"20e1e5e0-bd66-11e9-8ce2-f598b086bbe8","time":"2019-08-13T01:03:14.915424000Z","message":"Received webhook event"} {"level":"error","rid":"bl90oknb548975k834mg","method":"POST","path":"/api/github/hook","error":"panic: runtime error: invalid memory address or nil pointer dereference\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:522\nruntime.panicmem\n\t/usr/local/go/src/runtime/panic.go:82\nruntime.sigpanic\n\t/usr/local/go/src/runtime/signal_unix.go:390\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.cacheControl.func1.1\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/client_creator.go:337\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.roundTripperFunc.RoundTrip\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/middleware.go:151\ngithub.com/palantir/bulldozer/vendor/github.com/gregjones/httpcache.(Transport).RoundTrip\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/gregjones/httpcache/httpcache.go:214\ngithub.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation.(AppsTransport).RoundTrip\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation/appsTransport.go:78\ngithub.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation.(Transport).refreshToken\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation/transport.go:121\ngithub.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation.(Transport).Token\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation/transport.go:105\ngithub.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation.(Transport).RoundTrip\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/bradleyfalzon/ghinstallation/transport.go:87\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.ClientMetrics.func1.1\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/middleware.go:64\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.roundTripperFunc.RoundTrip\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/middleware.go:151\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.ClientLogging.func1.1\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/middleware.go:121\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.roundTripperFunc.RoundTrip\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/middleware.go:151\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.setInstallationID.func1.1\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/client_creator.go:362\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.roundTripperFunc.RoundTrip\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/middleware.go:151\nnet/http.send\n\t/usr/local/go/src/net/http/client.go:250\nnet/http.(Client).send\n\t/usr/local/go/src/net/http/client.go:174\nnet/http.(Client).do\n\t/usr/local/go/src/net/http/client.go:641\nnet/http.(Client).Do\n\t/usr/local/go/src/net/http/client.go:509\ngithub.com/palantir/bulldozer/vendor/github.com/google/go-github/github.(Client).Do\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/google/go-github/github/github.go:488\ngithub.com/palantir/bulldozer/vendor/github.com/google/go-github/github.(PullRequestsService).Get\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/google/go-github/github/pulls.go:177\ngithub.com/palantir/bulldozer/server/handler.(IssueComment).Handle\n\t/go/src/github.com/palantir/bulldozer/server/handler/issue_comment.go:54\ngithub.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp.(eventDispatcher).ServeHTTP\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/palantir/go-githubapp/githubapp/dispatcher.go:178\ngithub.com/palantir/bulldozer/vendor/goji%2eio.dispatch.ServeHTTP\n\t/go/src/github.com/palantir/bulldozer/vendor/goji.io/dispatch.go:17\ngithub.com/palantir/bulldozer/vendor/github.com/bluekeyes/hatpear.Recover.func1.1\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/bluekeyes/hatpear/hatpear.go:107\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:1995\ngithub.com/palantir/bulldozer/vendor/github.com/bluekeyes/hatpear.Catch.func1.1\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/bluekeyes/hatpear/hatpear.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:1995\ngithub.com/palantir/bulldozer/vendor/github.com/rs/zerolog/hlog.AccessHandler.func1.1\n\t/go/src/github.com/palantir/bulldozer/vendor/github.com/rs/zerolog/hlog/hlog.go:180\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:1995","time":"2019-08-13T01:03:14.939898900Z","message":"Unhandled error while serving route"} {"level":"info","rid":"bl90oknb548975k834mg","method":"POST","path":"/api/github/hook","status":500,"size":33,"elapsed":24.7403,"time":"2019-08-13T01:03:14.940041200Z","message":"http_request"}

[sreejuGit]

Also, I'm using the docker image from https://hub.docker.com/r/palantirtechnologies/bulldozer/ and hosted the service in a container in my local machine.

[bluekeyes]

Thanks for the extra information. This is an issue we've seen internally as well (it seems to happen randomly), but I haven't found the root cause yet.

[sreejuGit]

For my setup, this is the consistent behavior and was never able to get a successful response for all the valid pull requests, which meets all the checks.
The success(200) responses was only when bulldozer does not have to perform any action from its end. Is there any suggestions for solutions to try ? Is it any way related to platform ? i'm using MacOS Mojave Does building the code and using it, instead of the docker image works ?

[bluekeyes]

The error is coming from one of the dependency libraries, and is related to code for create GitHub clients with caching. I don't think it is platform related, but is more likely to be related to the request. If you have access to the GitHub hook delivery logs (in the admin section for the app), it may help me debug if you can share a (sanitized) copy of the payload sent by GitHub that causes the failure.

Building the code locally and in Docker should give the same results, but if you build locally, you can try commenting out this line:

https://github.com/palantir/bulldozer/blob/bd7776884a302c06d065a27203731c553fa4355f/server/server.go#L63

That will disable the caching functionality, which seems to be causing the problem.

[bluekeyes]

I identified a potential issue in the go-githubapp library that might cause this and submitted a fix: https://github.com/palantir/go-githubapp/pull/24

Once that merges, I'll update the library here and publish a new snapshot Docker image of Bulldozer that you can test with. Depending on your familiarity with Go, you could also use the branch from the linked PR in a local build of Bulldozer.

[sreejuGit]

Just now i saw your comments on fix, But before it, I had build the code locally from develop branch, hosted buldozer and that worked fine for me. without any errors .

I did not make any changes (cache changes) to the develop code, as you suggested.
also I have not taken the branch from the linked PR.

May be it is intermitten as you suggested or the existing docker image was having some issues. I'll any way wait for your fix and the updated docker image.

[bluekeyes]

palantirtechnologies/bulldozer:snapshot has been updated with the potential fix, if you'd like to try the Docker image again.

[sreejuGit]

I have pulled the latest snapshot image and run it, it is still providing me the same error .

[bluekeyes]

Huh, that's unfortunate. Can you share a stacktrace from the new snapshot image? I'd like to confirm that it's occurring on the same code.

I've also tried to reproduce this locally and so far have not been able to. Could you also share details about the specific GitHub events that cause this and how you have Bulldozer configured (e.g. your configuration file or environment variables)?

[sreejuGit]

# Options for the http server
server:
  # The listen address and port
  address: "0.0.0.0"
  port: 8080
  # Uncomment the "tls_config" block to enable HTTPS support in the server.
  # The cert and key files must be usable by net/http.ListenAndServeTLS().
  # tls_config:
  #   cert_file: /path/to/server.pem
  #   key_file: /path/to/server.key

# Options for logging output
logging:
  # If true, logs are printed in human-readable form. We recommend using
  # "false" to output JSON-formatted logs in production
  text: false

  # Set a minimum logging level threshold
  # Choose from: debug, info, warn, error
  level: debug

# Options for the HTTP Cache
cache:
  # The maximum size of the cache (specified in human readable units)
  max_size: 100 MB

# Options for connecting to GitHub
github:
  # The URL of the GitHub homepage. Can also be set by the GITHUB_WEB_URL
  # environment variable.
  #web_url: "https://github.service.org/orgname" 
  # The base URL for v3 (REST) API requests. Can also be set by the
  # GITHUB_V3_API_URL environment variable.
  #v3_api_url: "https://github.service.org/api/v3/orgs/orgname" 
  app:
    # The integration ID of the GitHub app. Can also be set by the
    # GITHUB_APP_INTEGRATION_ID environment variable.
    integration_id: 10
    # A random string used to validate webhooks. Can also be set by the
    # GITHUB_APP_WEBHOOK_SECRET environment variable.
    webhook_secret: "app_secret"
    # The private key of the GitHub app (.pem file download from GitHub). Can
    # also be set by the GITHUB_APP_PRIVATE_KEY environment variable.
    private_key: |
      -----BEGIN RSA PRIVATE KEY-----
      wj6E3YhUb5YFvxyasrByZK
      Trunccated private key=
      -----END RSA PRIVATE KEY----- 

# Options for application behavior
options:
  # The path within repositories to find the bulldozer.yml config file
  configuration_path: /config/pr-rule/.bulldozer.yml
  # The name of the application. This will affect the User-Agent header
  # when making requests to Github.
  app_name: bulldozer
  # An optional personal access token associated with a normal GitHub user that
  # is used to merge pull requests into protected branches with push
  # restrictions. Can also be set by the BULLDOZER_PUSH_RESTRICTION_USER_TOKEN
  #  environment variable.
 # push_restriction_user_token: token
  # Default repository config, the same as the config file described in README
  default_repository_config:
    merge:
      whitelist:
        comments: ["Please merge this pull request!"]
      blacklist:
        labels: ["do not merge"]
      delete_after_merge: false

# Optional configuration to emit metrics to datadog
datadog:
  # Database endpoint
  # address: "127.0.0.1:8125"
  # The metric flush frequency. Accepts any string
  # parseable by https://golang.org/pkg/time/#ParseDuration 
  interval: "10s"
  # Tags to include
  #tags:
  #  - "bulldozer"

[sreejuGit]

I could resolve the issue by following steps

1) building the code locally from develop branch after commenting out the caching, line of code as suggested. bulldozer/server/server.go Line 63 in bd77768 githubapp.WithClientCaching(true, func() httpcache.Cache { return lrucache.New(maxSize, 0) }),

2)Then, It started providing me with certificate error. Which i resolved by provided our organization certificate in DockerFile as we are connecting to our specific github instance.

This newly created image is working fine now.

[sreejuGit]

Hi bluekeyes,

This is an unrelated question to current thread.

If i set in config the delete_after_merge: true Is there any way I can stop specific branches from deleting?

The scenario is like.

I want to auto delete all feature branches(feature/) after merge into develop. But I don't want to auto delete a release/ or hotfix/* branch after their merge into develop. As we may need to merge it multiple times in between.Hence we want to delete it manually after all the release activities.

[clakech]

We have the same issue using the palantir/policy-bot that use the same technical stack. We suspect that the root cause is the performance of our bot. GitHub expect a response I less than 10 seconds and our policy-bot is too long to manage the hook event. So Github close the request with a client side time out and that may leads to a memory address / nil pointer error and then a 500 error. https://developer.github.com/v3/guides/best-practices-for-integrators/

May this help if anyone facing this issue again

[bluekeyes]

I believe the main issue here (the panic) is fixed as of 1.7.1 - at least, after upgrading the same dependency in policy-bot, we've stopped seeing this error. That said, the error happened when a GitHub request failed, so it's still likely that the request to Bulldozer will fail, but now you should see the actual cause instead of the panic.