Webhooks are firing 4 times for each event

[joecryptotoo]

Bug Description

I'm using tarampampam/webhook-tester to test the webhook and I'm seeing 4 posts for each event. They all come a few milliseconds apart and the payload is the same on all of them except the eventDate advances by a few milliseconds each time.

[joecryptotoo]

twenty-worker-2   | Exception Captured
twenty-worker-2   |   undefined
twenty-worker-2   |   [
twenty-worker-2   |     ReferenceError: event is not defined
twenty-worker-2   |         at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
twenty-worker-2   |         at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
twenty-worker-2   |         at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
twenty-worker-2   |         at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
twenty-worker-2   |         at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
twenty-worker-2   |         at async _bullmq.Worker [as processFn] (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/bullmq.driver.js:33:13)
twenty-worker-2   |         at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
twenty-worker-2   |         at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
twenty-worker-2   |   ]
twenty-worker-2   | [Nest] 34  - 08/26/2024, 7:26:39 AM     LOG [CallWebhookJob] CallWebhookJob successfully called on targetUrl 'https://webhook-tester.climax.global/5526f745-811b-4823-8973-575f69b70cfb'
twenty-worker-2   | Exception Captured
twenty-worker-2   |   undefined
twenty-worker-2   |   [
twenty-worker-2   |     ReferenceError: event is not defined
twenty-worker-2   |         at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
twenty-worker-2   |         at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
twenty-worker-2   |         at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
twenty-worker-2   |         at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
twenty-worker-2   |         at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
twenty-worker-2   |         at async _bullmq.Worker [as processFn] (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/bullmq.driver.js:33:13)
twenty-worker-2   |         at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
twenty-worker-2   |         at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
twenty-worker-2   |   ]
twenty-worker-2   | [Nest] 34  - 08/26/2024, 7:26:39 AM     LOG [CallWebhookJob] CallWebhookJob successfully called on targetUrl 'https://webhook-tester.climax.global/5526f745-811b-4823-8973-575f69b70cfb'
twenty-worker-2   | Exception Captured
twenty-worker-2   |   undefined
twenty-worker-2   |   [
twenty-worker-2   |     ReferenceError: event is not defined
twenty-worker-2   |         at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
twenty-worker-2   |         at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
twenty-worker-2   |         at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
twenty-worker-2   |         at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
twenty-worker-2   |         at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
twenty-worker-2   |         at async _bullmq.Worker [as processFn] (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/bullmq.driver.js:33:13)
twenty-worker-2   |         at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
twenty-worker-2   |         at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
twenty-worker-2   |   ]
twenty-worker-2   | [Nest] 34  - 08/26/2024, 7:26:39 AM     LOG [CallWebhookJob] CallWebhookJob successfully called on targetUrl 'https://webhook-tester.climax.global/5526f745-811b-4823-8973-575f69b70cfb'
twenty-worker-2   | Exception Captured
twenty-worker-2   |   undefined
twenty-worker-2   |   [
twenty-worker-2   |     ReferenceError: event is not defined
twenty-worker-2   |         at CallWebhookJobsJob.handle (/app/packages/twenty-server/dist/src/engine/api/graphql/workspace-query-runner/jobs/call-webhook-jobs.job.js:87:65)
twenty-worker-2   |         at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
twenty-worker-2   |         at async MessageQueueExplorer.invokeProcessMethods (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:111:17)
twenty-worker-2   |         at async MessageQueueExplorer.handleProcessor (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:105:13)
twenty-worker-2   |         at async /app/packages/twenty-server/dist/src/engine/integrations/message-queue/message-queue.explorer.js:79:17
twenty-worker-2   |         at async _bullmq.Worker [as processFn] (/app/packages/twenty-server/dist/src/engine/integrations/message-queue/drivers/bullmq.driver.js:33:13)
twenty-worker-2   |         at async Worker.processJob (/app/node_modules/bullmq/dist/cjs/classes/worker.js:394:28)
twenty-worker-2   |         at async Worker.retryIfFailed (/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24)
twenty-worker-2   |   ]
twenty-worker-2   | [Nest] 34  - 08/26/2024, 7:26:39 AM     LOG [CallWebhookJob] CallWebhookJob successfully called on targetUrl 'https://webhook-tester.climax.global/5526f745-811b-4823-8973-575f69b70cfb'

[skyfox675]

Seeing the same error/behavior here as well

Version: v0.23.2 Environment: Self-Hosted MESSAGE_QUEUE_TYPE=pg_boss Seen on: Custom Data Models, haven't ruled out Standard Models though Webhook Filtering: Yes, filtered to Custom Model name and Update Webhook Responses are sending 200 OK's after external processing has finished on Webhook receiver (typically .5-1 seconds)

[joecryptotoo]

This is self hosted running 0.23.2 using redis message queues

[Bonapara]

Any idea @martmull?

[skyfox675]

FYI after switching to MESSAGE_QUEUE_TYPE=bull-mq with Redis, I'm now seeing the following exception behavior:

[charlesBochet]

We have recenlty fixed issue in 0.24, do you still have the issue on 0.24?

I wonder if this is because of a retry logic we have in place

[joecryptotoo]

I just tested it and the issue has been fixed. Thank you.