search

argoproj / argo-events

Event-driven Automation Framework for Kubernetes
https://argoproj.github.io/argo-events/
Apache License 2.0
2.35k stars 729 forks source link

Azure Events Hub examples don't work out of the box #710

Closed iMoses closed 4 years ago

iMoses commented 4 years ago

Describe the bug I've spent an entire day trying to get the gateway to function correctly without success. It seems that the gateway had connected to the service, because the following error only appears after a new event is sent (after the gateway starts), which seems like it receives it but...

ERRO[2020-06-21 13:26:44] failed to receive event from stream           error="rpc error: code = Unavailable desc = transport is closing" event-source=workflow-event
INFO[2020-06-21 13:26:44] received a event source state update notification 
INFO[2020-06-21 13:26:44] marking node phase                            node-name=workflow-event phase=Error
INFO[2020-06-21 13:26:44] phase updated                                 new-phase=Error node-name=workflow-event phase=Error
INFO[2020-06-21 13:26:44] detected gateway update. updating gateway watchers 
INFO[2020-06-21 13:26:44] received a gateway resource update notification 
INFO[2020-06-21 13:26:44] checking if any new subscribers are added

UPDATE I notice i missed part of the logs.. this could probably prove useful:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1779583]

goroutine 103 [running]:
github.com/argoproj/argo-events/gateways/server/azure-events-hub.(*EventListener).listenEvents.func1(0x26e7ee0, 0xc000882180, 0xc0007d4280, 0x1, 0x26a4fa0)
    /go/src/github.com/argoproj/argo-events/gateways/server/azure-events-hub/start.go:98 +0x83
github.com/Azure/azure-event-hubs-go/v3.(*receiver).handleMessage(0xc0007d8140, 0x26e7ee0, 0xc000882180, 0xc0004e8b40, 0xc000126e60)
    /go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.2.0/receiver.go:260 +0x1c7
github.com/Azure/azure-event-hubs-go/v3.(*receiver).handleMessages(0xc0007d8140, 0x26e7ee0, 0xc000882180, 0xc0009d6000, 0xc000126e60)
    /go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.2.0/receiver.go:237 +0x10b
created by github.com/Azure/azure-event-hubs-go/v3.(*receiver).Listen
    /go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.2.0/receiver.go:221 +0x1b8

To Reproduce

Follow the azure-events-hub examples.. View logs of newly created azure-events-hub gateway

INFO[2020-06-21 13:20:15] detected a new event-source...                event-source=azure-events-hub-event-source
INFO[2020-06-21 13:20:15] state of the connection to gateway server     name=workflow-event state=READY value="{atmz-staging-event-hub.servicebus.windows.net &SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-event-hub-shared-access-key,},Key:name,Optional:nil,} &SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:azure-event-hub-shared-access-key,},Key:key,Optional:nil,} argo-workflow }"
INFO[2020-06-21 13:20:15] deleted event sources                         event-source="[]"
INFO[2020-06-21 13:20:15] new event sources                             event-source="[2700525430]"
INFO[2020-06-21 13:20:15] activating new event source...                event-source=workflow-event
INFO[2020-06-21 13:20:15] event source is valid                         event-source=workflow-event
INFO[2020-06-21 13:20:15] listening to events from gateway server...    event-source=workflow-event
INFO[2020-06-21 13:20:15] received a event source state update notification 
INFO[2020-06-21 13:20:15] initializing the node                         node-name=workflow-event
INFO[2020-06-21 13:20:15] node is running                               node-message="event source is running" node-name=workflow-event
INFO[2020-06-21 13:20:16] detected gateway update. updating gateway watchers 
INFO[2020-06-21 13:20:16] received a gateway resource update notification 
INFO[2020-06-21 13:20:16] checking if any new subscribers are added    
ERRO[2020-06-21 13:26:44] failed to receive event from stream           error="rpc error: code = Unavailable desc = transport is closing" event-source=workflow-event
INFO[2020-06-21 13:26:44] received a event source state update notification 
INFO[2020-06-21 13:26:44] marking node phase                            node-name=workflow-event phase=Error
INFO[2020-06-21 13:26:44] phase updated                                 new-phase=Error node-name=workflow-event phase=Error
INFO[2020-06-21 13:26:44] detected gateway update. updating gateway watchers 
INFO[2020-06-21 13:26:44] received a gateway resource update notification 
INFO[2020-06-21 13:26:44] checking if any new subscribers are added

Expected behavior Not having an error, and having the sensor respond to the event

Environment (please complete the following information): K8s: 1.16.9 Version: 0.16

iMoses commented 4 years ago

These are the templates I'm using to create a gateway, sensor and event source:

apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: azure-events-hub-event-source
  namespace: {{ .Release.Namespace }}
spec:
  type: azureEventsHub
  azureEventsHub:
    workflow-event:
      fqdn: {{ .Values.azureEventsHub.fqdn }}
      sharedAccessKeyName:
        name: {{ .Values.azureEventsHub.sharedAccessKey.secretName }}
        key: name
      sharedAccessKey:
        name: {{ .Values.azureEventsHub.sharedAccessKey.secretName }}
        key: key
      hubName: argo-workflow
apiVersion: argoproj.io/v1alpha1
kind: Gateway
metadata:
  name: azure-events-hub
  namespace: {{ .Release.Namespace }}
  labels:
    gateways.argoproj.io/gateway-controller-instanceid: argo-events
spec:
  type: azureEventsHub
  eventSourceRef:
    name: azure-events-hub-event-source
  template:
    serviceAccountName: {{ .Values.serviceAccountName }}
  subscribers:
    http:
      - "http://azure-events-hub-sensor.{{ .Release.Namespace }}.svc:{{ .Values.sensors.port }}/"
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  namespace: {{ .Release.Namespace }}
  name: azure-events-hub
  labels:
    sensors.argoproj.io/sensor-controller-instanceid: argo-events
spec:
  template:
    serviceAccountName: {{ .Values.serviceAccountName }}
  subscription:
    http:
      port: {{ .Values.sensors.port }}
  dependencies:
    - name: workflow-event
      eventName: workflow-event
      gatewayName: azure-events-hub
  triggers:
    - template:
        name: workflow-trigger
        k8s:
          group: argoproj.io
          version: v1alpha1
          resource: workflows
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                namespace: {{ .Release.Namespace }}
                generateName: azure-event-hub-test-
              spec:
                entrypoint: whalesay
                arguments:
                  parameters:
                  - name: message
                templates:
                - name: whalesay
                  serviceAccountName: {{ .Values.serviceAccountName }}
                  inputs:
                    parameters:
                    - name: message
                  container:
                    image: docker/whalesay:latest
                    command: [cowsay]
                    args: [{{`"{{inputs.parameters.message}}"`}}]
          parameters:
            - src:
                dependencyName: workflow-event
              dest: spec.arguments.parameters.0.value
iMoses commented 4 years ago

I'm not an expert in go, far from it, but the error is coming from here:

        eventData := &events.AzureEventsHubEventData{
            Id:           event.ID,
            PartitionKey: *event.PartitionKey,
            Body:         event.Data,
        }

        eventBytes, err := json.Marshal(eventData)
        if err != nil {
            return errors.Wrapf(err, "failed to marshal the event data for event source %s and message id %s", eventSource.Name, event.ID)
        }

        channels.Data <- eventBytes
        return nil
    }

line 98 is this:

            PartitionKey: *event.PartitionKey,

and I was wondering whether it is right to access event.PartitionKey with what i assume is a pointer reference, or maybe that's the mistake...

VaibhavPage commented 4 years ago

Taking a look.

iMoses commented 4 years ago

@VaibhavPage upgraded to 0.17.0

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x170f7a8]

goroutine 135 [running]:
github.com/argoproj/argo-events/eventsources/sources/azureeventshub.(*EventListener).StartListening.func1(0x2666300, 0xc0004cd700, 0xc0005cae60, 0x1, 0x2626740)
    /home/runner/work/argo-events/argo-events/eventsources/sources/azureeventshub/start.go:91 +0x98
github.com/Azure/azure-event-hubs-go/v3.(*receiver).handleMessage(0xc0001de780, 0x2666300, 0xc0004cd700, 0xc00079a480, 0xc000384680)
    /home/runner/go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.3.0/receiver.go:260 +0x1a2
github.com/Azure/azure-event-hubs-go/v3.(*receiver).handleMessages(0xc0001de780, 0x2666300, 0xc0004cd700, 0xc000b29620, 0xc000384680)
    /home/runner/go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.3.0/receiver.go:237 +0x102
created by github.com/Azure/azure-event-hubs-go/v3.(*receiver).Listen
    /home/runner/go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.3.0/receiver.go:221 +0x18c
iMoses commented 4 years ago

I really think it's all about the asterisk in eventsources/sources/azureeventshub/start.go line 89

            PartitionKey: *event.PartitionKey,

should be:

            PartitionKey: event.PartitionKey,

unless i'm missing something... i can help test it if you can provide some sort of an rc version

VaibhavPage commented 4 years ago

@iMoses i'll test the eventsource asap and put a patch if required

AustinSmart commented 4 years ago

I was testing out a fresh install of Argo Events and Argo Workflows. I have encountered the same error when using an Azure event Hub as an eventsource

{"eventName":"example","eventSourceName":"azure-events-hub","eventSourceType":"azureEventsHub","level":"info","msg":"handling the partitions...","time":"2020-07-28 16:53:07"}                                                                                                                              
panic: runtime error: invalid memory address or nil pointer dereference                                                                                                                                                                                                                                     
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x170f7a8]                                                                                                                                                                                                                                     
goroutine 99 [running]:                                                                                                                                                                                                                                                                                     
github.com/argoproj/argo-events/eventsources/sources/azureeventshub.(*EventListener).StartListening.func1(0x2666300, 0xc000232a00, 0xc000822230, 0x1, 0x2626740)                                                                                                                                            
     /home/runner/work/argo-events/argo-events/eventsources/sources/azureeventshub/start.go:91 +0x98                                                                                                                                                                                                         
github.com/Azure/azure-event-hubs-go/v3.(*receiver).handleMessage(0xc000796140, 0x2666300, 0xc000232a00, 0xc000122090, 0xc0002aea00)                                                                                                                                                                        
     /home/runner/go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.3.0/receiver.go:260 +0x1a2                                                                                                                                                                                                          
github.com/Azure/azure-event-hubs-go/v3.(*receiver).handleMessages(0xc000796140, 0x2666300, 0xc000232a00, 0xc0006a50e0, 0xc0002aea00)                                                                                                                                                                       
     /home/runner/go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.3.0/receiver.go:237 +0x102                                                                                                                                                                                                          
created by github.com/Azure/azure-event-hubs-go/v3.(*receiver).Listen                                                                                                                                                                                                                                       
    /home/runner/go/pkg/mod/github.com/!azure/azure-event-hubs-go/v3@v3.3.0/receiver.go:221 +0x18c                                                                                                                                                                                                          
stream closed
VaibhavPage commented 4 years ago

@iMoses can you test with rc metalgearsolid/eventsource:v0.17.1 ?

iMoses commented 4 years ago

@iMoses can you test with rc metalgearsolid/eventsource:v0.17.1 ?

Will report soon 🙂

iMoses commented 4 years ago

@VaibhavPage i managed to deploy this image

{"level":"info","ts":1596674662.4063191,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:250","msg":"Starting event source server...","eventSourceName":"azure-events-hub"}
{"level":"info","ts":1596674662.406663,"logger":"argo-events.eventsource","caller":"driver/nats.go:92","msg":"NATS auth strategy: Token","eventSourceName":"azure-events-hub","clientID":"azure-events-hub-eventsource-mlr2k-bfbb9f4d7-hb8zv"}
{"level":"info","ts":1596674662.4090989,"logger":"argo-events.eventsource","caller":"driver/nats.go:104","msg":"Connected to NATS server.","eventSourceName":"azure-events-hub","clientID":"azure-events-hub-eventsource-mlr2k-bfbb9f4d7-hb8zv"}
{"level":"info","ts":1596674662.4108386,"logger":"argo-events.eventsource","caller":"driver/nats.go:117","msg":"Connected to NATS streaming server.","eventSourceName":"azure-events-hub","clientID":"azure-events-hub-eventsource-mlr2k-bfbb9f4d7-hb8zv"}
{"level":"info","ts":1596674662.4108675,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:327","msg":"Eventing server started.","eventSourceName":"azure-events-hub"}
{"level":"info","ts":1596674662.4113533,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:268","msg":"starting eventbus connection daemon...","eventSourceName":"azure-events-hub"}
{"level":"info","ts":1596674662.4113922,"logger":"argo-events.eventsource","caller":"azureeventshub/start.go:62","msg":"started processing the Azure Events Hub event source...","eventSourceName":"azure-events-hub","eventSourceType":"azureEventsHub","eventName":"argo-workflow"}
{"level":"info","ts":1596674662.411411,"logger":"argo-events.eventsource","caller":"azureeventshub/start.go:66","msg":"retrieving the shared access key name...","eventSourceName":"azure-events-hub","eventSourceType":"azureEventsHub","eventName":"argo-workflow"}
{"level":"info","ts":1596674662.4114232,"logger":"argo-events.eventsource","caller":"azureeventshub/start.go:72","msg":"retrieving the shared access key...","eventSourceName":"azure-events-hub","eventSourceType":"azureEventsHub","eventName":"argo-workflow"}
{"level":"info","ts":1596674662.4114337,"logger":"argo-events.eventsource","caller":"azureeventshub/start.go:80","msg":"connecting to the hub...","eventSourceName":"azure-events-hub","eventSourceType":"azureEventsHub","eventName":"argo-workflow"}
{"level":"info","ts":1596674662.411461,"logger":"argo-events.eventsource","caller":"azureeventshub/start.go:109","msg":"gathering the hub runtime information...","eventSourceName":"azure-events-hub","eventSourceType":"azureEventsHub","eventName":"argo-workflow"}
{"level":"info","ts":1596674662.5321205,"logger":"argo-events.eventsource","caller":"azureeventshub/start.go:117","msg":"handling the partitions...","eventSourceName":"azure-events-hub","eventSourceType":"azureEventsHub","eventName":"argo-workflow"}

this is all the logs i'm getting an then it stops.. before it would have received the event then crash on handling it, now it doesn't seem to receive it at all

iMoses commented 4 years ago

I think this issue needs to be reopened @VaibhavPage