search

argoproj / argo-events

Event-driven Automation Framework for Kubernetes
https://argoproj.github.io/argo-events/
Apache License 2.0
2.38k stars 740 forks source link

github app event source pod loops infinitely on startup #2764

Closed brizzbuzz closed 1 year ago

brizzbuzz commented 1 year ago

Describe the bug Hi, this could very well be an error in my setup, but, if it is, it would still be nice to get some better logging in place.

I have an event source set up to use a github app that I have configured w/ repository read access across my entire organization. 

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: github-event-source
  namespace: argo-events
spec:
  service:
    ports:
      - port: 12000
        targetPort: 12000
  github:
    example:
      repositories:
        - owner: testerino
          names:
            - cool-repo
      # Github application auth. Instead of using personal token `apiToken` use app PEM            
      githubApp:
        privateKey:
          name: my-secret-key
          key: private_key
        appID: app-id
        installationID: install-id
      webhook:
        # endpoint to listen to events on
        endpoint: /push
        port: "12000"
        method: POST
        url: events.example.com
      events:
        - "*"
      insecure: false
      active: true
      contentType: json

Everything looks ok, except that I can see in the debug of my github app, that any webhook request 503s. My networking config looks correct, and diving down to the pod backing the service that the event source creates, I can see that there are just a loop of the same few logs (despite pod status being healthy).

{"level":"info","ts":1692584231.4489276,"logger":"argo-events.eventsource","caller":"cmd/start.go:63","msg":"starting eventsource server","eventSourceName":"github","version":"v1.8.0"}
{"level":"info","ts":1692584231.4492145,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:432","msg":"Starting event source server...","eventSourceName":"github"}
{"level":"info","ts":1692584231.449381,"logger":"argo-events.eventsource","caller":"metrics/metrics.go:175","msg":"starting metrics server","eventSourceName":"github"}
{"level":"info","ts":1692584231.4500518,"logger":"argo-events.eventsource","caller":"eventbus/driver.go:36","msg":"eventBusConfig: {NATS:&NATSConfig{URL:nats://eventbus-default-stan-svc:4222,ClusterID:*eventbus-default,Auth:*token,AccessSecret:&v1.SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:eventbus-default-client,},Key:client-auth,Optional:nil,},} JetStream:nil Kafka:nil}","eventSourceName":"github"}
{"level":"info","ts":1692584231.450146,"logger":"argo-events.eventsource","caller":"base/stan.go:48","msg":"NATS auth strategy: Token","eventSourceName":"github","clientID":"client-github-eventsource-r8bw9-7f7c6c45bb-5jjmv-359"}
{"level":"info","ts":1692584231.4533868,"logger":"argo-events.eventsource","caller":"base/stan.go:60","msg":"Connected to NATS server.","eventSourceName":"github","clientID":"client-github-eventsource-r8bw9-7f7c6c45bb-5jjmv-359"}
{"level":"info","ts":1692584231.455363,"logger":"argo-events.eventsource","caller":"base/stan.go:73","msg":"Connected to NATS streaming server.","eventSourceName":"github","clientID":"client-github-eventsource-r8bw9-7f7c6c45bb-5jjmv-359"}
{"level":"info","ts":1692584231.4555738,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:581","msg":"Eventing server started.","eventSourceName":"github"}
{"level":"info","ts":1692584231.4556413,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:459","msg":"starting eventbus connection daemon...","eventSourceName":"github"}
{"level":"info","ts":1692584231.4562159,"logger":"argo-events.eventsource","caller":"github/start.go:223","msg":"started processing the Github event source...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584231.4563575,"logger":"argo-events.eventsource","caller":"github/start.go:231","msg":"retrieving webhook secret credentials if any ...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584231.456421,"logger":"argo-events.eventsource","caller":"github/start.go:260","msg":"choosing github auth strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584231.4565814,"logger":"argo-events.eventsource","caller":"github/start.go:266","msg":"setting up auth transport for http client with the chosen strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584240.3323395,"logger":"argo-events.eventsource","caller":"github/start.go:223","msg":"started processing the Github event source...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584240.3324273,"logger":"argo-events.eventsource","caller":"github/start.go:231","msg":"retrieving webhook secret credentials if any ...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584240.3324397,"logger":"argo-events.eventsource","caller":"github/start.go:260","msg":"choosing github auth strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584240.332583,"logger":"argo-events.eventsource","caller":"github/start.go:266","msg":"setting up auth transport for http client with the chosen strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584253.2297268,"logger":"argo-events.eventsource","caller":"github/start.go:223","msg":"started processing the Github event source...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584253.2298243,"logger":"argo-events.eventsource","caller":"github/start.go:231","msg":"retrieving webhook secret credentials if any ...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584253.2298355,"logger":"argo-events.eventsource","caller":"github/start.go:260","msg":"choosing github auth strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584253.2300208,"logger":"argo-events.eventsource","caller":"github/start.go:266","msg":"setting up auth transport for http client with the chosen strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584283.1141422,"logger":"argo-events.eventsource","caller":"github/start.go:223","msg":"started processing the Github event source...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584283.1142168,"logger":"argo-events.eventsource","caller":"github/start.go:231","msg":"retrieving webhook secret credentials if any ...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584283.1142292,"logger":"argo-events.eventsource","caller":"github/start.go:260","msg":"choosing github auth strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584283.1143453,"logger":"argo-events.eventsource","caller":"github/start.go:266","msg":"setting up auth transport for http client with the chosen strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584303.6863234,"logger":"argo-events.eventsource","caller":"github/start.go:223","msg":"started processing the Github event source...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584303.6864154,"logger":"argo-events.eventsource","caller":"github/start.go:231","msg":"retrieving webhook secret credentials if any ...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1692584303.686428,"logger":"argo-events.eventsource","caller":"github/start.go:260","msg":"choosing github auth strategy...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}

So, I am imagining that this is to blame for the service being unreachable, but, since there is not any actual error, I am unsure of how to proceed. Any advice here would be greatly appreciated 🙏

To Reproduce Steps to reproduce the behavior:

Unsure

Expected behavior Event source should not 503 when request is received.

Screenshots N/A

Environment (please complete the following information):

Additional context N/A

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

brizzbuzz commented 1 year ago

Alright, i made a bit of progress... it turns out, if you do not assign webhook permissions to the github app, this will cause this loop. IMO this should be an error, or at least a warning.

Now I'm failing due to my private key not being in the correct format... which... not sure how but will carry on 🫡

whynowy commented 1 year ago

Alright, i made a bit of progress... it turns out, if you do not assign webhook permissions to the github app, this will cause this loop. IMO this should be an error, or at least a warning.

Now I'm failing due to my private key not being in the correct format... which... not sure how but will carry on 🫡

hmmmm... if we didn't throw an error, we should fix it.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. It will be closed if no further activity occurs. Thank you for your contributions.