azureQueueStorage event source cannot parse JSON event message

[yaroslav-ddt]

Describe the bug Storage Account sent blob creation and deletion events via the event subscription to the Storage Queue. azureQueueStorage listens to messages from Storage Queue. Message reseived by azure-queue-storage-eventsource pod but cannot be processed due the following error: "msg":"failed to marshal event data, will process next message...","eventSourceName":"azure-queue-storage-source","eventSourceType":"azureQueueStorage","eventName":"test","error":"json: error calling MarshalJSON for type *json.RawMessage: invalid character 'e' looking for beginning of value"

If the same message is entered manually into the Storage Queue, it is processed without errors.

To Reproduce Steps to reproduce the behavior:

1. azureQueueStorage event source is installed by following the example.
2. Create a Storage Account and add Queue "test" into Storage Queues
3. Go to Storage Events and add an Event Subscription that points to the "test" Queue
4. Go to Containers and upload the new Blob
5. Check the log of azure-queue-storage-eventsource-xxxx pod

Expected behavior The Queue messages from the Storage event are processed without errors

Screenshots

Environment (please complete the following information):

• Kubernetes: 1.26.6
• Argo Workflows: 0.26.5
• Argo Events: 1.8.1

Additional context Add any other context about the problem here.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had any activity in the last 60 days. It will be closed if no further activity occurs. Thank you for your contributions.

[OdkhuuB]

Hello,

With ver 1.9 I am also seeing the same error receiving messages from Azure Queue Storage.

{"level":"error","ts":1709149431.0724385,"logger":"argo-events.eventsource","caller":"azurequeuestorage/start.go:150","msg":"failed to marshal event data, will process next message...","eventSourceName":"azure-queue-storage","eventSourceType":"azureQueueStorage","eventName":"example","error":"json: error calling MarshalJSON for type *json.RawMessage: invalid character 'e' looking for beginning of value","stacktrace":"github.com/argoproj/argo-events/eventsources/sources/azurequeuestorage.(*EventListener).processMessage\n\t/home/runner/work/argo-events/argo-events/eventsources/sources/azurequeuestorage/start.go:150\ngithub.com/argoproj/argo-events/eventsources/sources/azurequeuestorage.(*EventListener).StartListening\n\t/home/runner/work/argo-events/argo-events/eventsources/sources/azurequeuestorage/start.go:111\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).run.func3.1\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:536\ngithub.com/argoproj/argo-events/common.DoWithRetry.func1\n\t/home/runner/work/argo-events/argo-events/common/retry.go:106\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:220\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:226\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:421\ngithub.com/argoproj/argo-events/common.DoWithRetry\n\t/home/runner/work/argo-events/argo-events/common/retry.go:105\ngithub.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).run.func3\n\t/home/runner/work/argo-events/argo-events/eventsources/eventing.go:535"}

[OdkhuuB]

With DEBUG_LOG:true

2024-02-28T20:22:22.329Z ERROR argo-events.eventsource azurequeuestorage/start.go:150 failed to marshal event data, will process next message... {"eventSourceName": "azure-queue-storage", "eventSourceType": "azureQueueStorage", "eventName": "example", "error": "json: error calling MarshalJSON for type *json.RawMessage: invalid character 'e' looking for beginning of value"} github.com/argoproj/argo-events/eventsources/sources/azurequeuestorage.(*EventListener).processMessage /home/runner/work/argo-events/argo-events/eventsources/sources/azurequeuestorage/start.go:150 github.com/argoproj/argo-events/eventsources/sources/azurequeuestorage.(*EventListener).StartListening /home/runner/work/argo-events/argo-events/eventsources/sources/azurequeuestorage/start.go:111 github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).run.func3.1 /home/runner/work/argo-events/argo-events/eventsources/eventing.go:536 github.com/argoproj/argo-events/common.DoWithRetry.func1 /home/runner/work/argo-events/argo-events/common/retry.go:106 k8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1 /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:220 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:233 k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:226 k8s.io/apimachinery/pkg/util/wait.ExponentialBackoff /home/runner/go/pkg/mod/k8s.io/apimachinery@v0.24.3/pkg/util/wait/wait.go:421 github.com/argoproj/argo-events/common.DoWithRetry /home/runner/work/argo-events/argo-events/common/retry.go:105 github.com/argoproj/argo-events/eventsources.(*EventSourceAdaptor).run.func3 /home/runner/work/argo-events/argo-events/eventsources/eventing.go:535

[bene-tleilax-werdna]

Yeah, we get the same issue. In our case it is because the queued JSON message is base64-encoded. You would think that you could use the decodeMessage option in the eventSource spec but that appears to fail as well, I submitted #3084 for that particular issue.