Queue Mode Fails - Githubissues

maipal-c commented 1 week ago

Bug Description

In Queue mode, some executions wont trigger

To Reproduce

make sure have access to external redis-cluster with (6 nodes - 3 shards with 1 replica each on k8s in my case)
make sure have access to external postgresql db (RDS in my case)
n8n official helm chart with official arm image (1.56.2) with 1 main pod , 2 workers pods & 1 webhook pods
put a kafka trigger on any kafka topic
receiving 3-5 messages per seconds on kafka topic trigger
1-5 out of 10K executions fails with error below

data object of execution (from network tab - http://localhost:8010/rest/executions/102813 then response.data.data)

[
   {
      "resultData": "1"
   },
   {
      "error": "2",
      "runData": "3"
   },
   {
      "message": "4",
      "stack": "5"
   },
   {},
   "Cannot read properties of null (reading 'data')",
   "Error: Cannot read properties of null (reading 'data')\n    at Queue.onFailed (/usr/local/lib/node_modules/n8n/node_modules/bull/lib/job.js:516:18)\n    at Queue.emit (node:events:519:28)\n    at Queue.emit (node:domain:488:12)\n    at Object.module.exports.emitSafe (/usr/local/lib/node_modules/n8n/node_modules/bull/lib/utils.js:50:20)\n    at EventEmitter.messageHandler (/usr/local/lib/node_modules/n8n/node_modules/bull/lib/queue.js:476:15)\n    at EventEmitter.emit (node:events:519:28)\n    at EventEmitter.<anonymous> (/usr/local/lib/node_modules/n8n/node_modules/ioredis/built/cluster/ClusterSubscriber.js:160:30)\n    at EventEmitter.emit (node:events:519:28)\n    at DataHandler.handleSubscriberReply (/usr/local/lib/node_modules/n8n/node_modules/ioredis/built/DataHandler.js:80:32)\n    at DataHandler.returnReply (/usr/local/lib/node_modules/n8n/node_modules/ioredis/built/DataHandler.js:47:18)"
]

kafka trigger node object

{
    "1": {
        "parameters": {
            "topic": "<topic_name>",
            "groupId": "<grp_id>",
            "useSchemaRegistry": false,
            "options": {
                "jsonParseMessage": true
            }
        },
        "id": "6c2db656-45e1-414f-adea-eeafe5cfe38f",
        "name": "Kafka Trigger",
        "type": "n8n-nodes-base.kafkaTrigger",
        "typeVersion": 1.1,
        "position": [
            60,
            240
        ],
        "credentials": {
            "kafka": {
                "id": "H738I7rANzImXhSW",
                "name": "Kafka account"
            }
        }
    }
}

Expected behavior

the execution should process atleast initial kafka trigger node. or even on without processing kafka trigger node it commits offset to kafka, which leads to skip the single message processing.

very critical issue for fintech payment related topics

Operating System

k3s on amazon linux 2023

n8n Version

1.56.2

Node.js Version

no idea

Database

PostgreSQL

Execution mode

queue

Joffcom commented 1 week ago

Hey @maipal-c,

We have created an internal ticket to look into this which we will be tracking as "GHC-293"

maipal-c commented 1 week ago

Hey @Joffcom, i am not sure how ciritcal this issue is for others, bt for our current flow we are processing some payment messages from kafka and this is a very critical bug as of now for us.

issue is like even on error, n8n commit offset to kafka which leads to process next message and skip current one

currently we switched to a simple js file with another nodejs instance.

waiting for n8n to fix it...

Joffcom commented 1 week ago

Hey @maipal-c

What error do you see in the n8n logs and have you tried a newer version to rule out any possible fixes?

Can you also provide a link to the official helm chart you have used?

maipal-c commented 1 week ago

There is nothing serious in logs, just one line "Cannot read properties of null (reading 'data')".

helm chart - (https://artifacthub.io/packages/helm/open-8gears/n8n/0.25.0) with image

repository: n8nio/n8n
tag: "1.56.2"
digest: sha256:144da5572a6dfac34bb1c079a347f055c56cc35943b0eb244c7225a18fe30dda

currently i have upgraded the same chart with image -

repository: n8nio/n8n
tag: "1.62.1"
digest: sha256:49082daed743802dcdbb01c9a70b87d42acee1ecbca75867739dc858dacdbc2c

i have one execution per day in same state (like same error) for a single workflow... i am currently checking other workflows...

i guess it could be an issue with ioredis(npm package) or maybe bull queue (package)... bt even in these cases the kafka offset should not be commited..

maipal-c commented 1 week ago

not all the workflows have same error bt i got 3 workflows out of 5 (active) which have these errors (total around 7 errors) out of 90K executions in last week

all of them have same errors, 1 workflow was webhook trigger and 2 was kafka trigger based

Again i am using redis-cluster ( image - docker.io/bitnami/redis-cluster:7.2.5-debian-12-r0) on arm (amazon linux 2023)

Joffcom commented 1 week ago

That isn't an official n8n helm chart it is community made but likely won't be the cause.

If you can share the logs from n8n it would be helpful as at the moment, the one line on its own doesn't really mean anything but maybe there is more if you set the log level to debug.

Are you also able to share a minimal workflow that reproduces this so we can work out the priority.

maipal-c commented 1 week ago

n8n main container logs -

"message": "[ScalingService] Added job 102601 (execution 102813)"
"source_type":  "kubernetes_logs"
"stream": "stdout"
"timestamp": "2024-10-07T03:30:51.376487604Z"

"message": "Problem with execution 102813: Cannot read properties of null (reading 'data'). Aborting."
"source_type": "kubernetes_logs"
"stream": "stdout"
"timestamp": "2024-10-07T03:30:51.382669558Z"

n8n worker logs (i guess its not same execution) -

"message": "[JobProcessor] Starting job 102813 (execution 103025)"
"source_type": "kubernetes_logs"
"stream": "stdout"
"timestamp": "2024-10-07T04:18:59.161575079Z"

i just got these logs from kibana (as those earliar pods got killed when updated to new version)

Joffcom commented 1 week ago

Do you have more logs? Just selected lines may be hiding something so we would need more of the output.

Out of interest do you have an enterprise license as well or are you using the community edition?

maipal-c commented 1 week ago

i guess the job worker is missing that particular execution... and job id as well from logs (by the way i am stroring all worker logs in same index , like for one k8s replica i have 2 pods and all logs form these pods are stored in same elastic search index)

just using community licence.

by the way i have checked all the logs in all redis-cluster nodes they are clean -

1:M 07 Oct 2024 03:50:02.054 * 10 changes in 300 seconds. Saving...
1:M 07 Oct 2024 03:50:02.055 * Background saving started by pid 3492341
3492341:C 07 Oct 2024 03:50:02.061 * DB saved on disk
3492341:C 07 Oct 2024 03:50:02.062 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 07 Oct 2024 03:50:02.156 * Background saving terminated with success
1:M 07 Oct 2024 03:55:03.031 * 10 changes in 300 seconds. Saving...
1:M 07 Oct 2024 03:55:03.032 * Background saving started by pid 3493779
3493779:C 07 Oct 2024 03:55:03.037 * DB saved on disk
3493779:C 07 Oct 2024 03:55:03.038 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 07 Oct 2024 03:55:03.133 * Background saving terminated with success
1:M 07 Oct 2024 04:00:04.080 * 10 changes in 300 seconds. Saving...
1:M 07 Oct 2024 04:00:04.080 * Background saving started by pid 3495218
3495218:C 07 Oct 2024 04:00:04.088 * DB saved on disk
3495218:C 07 Oct 2024 04:00:04.089 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 07 Oct 2024 04:00:04.181 * Background saving terminated with success
1:M 07 Oct 2024 04:05:05.084 * 10 changes in 300 seconds. Saving...
1:M 07 Oct 2024 04:05:05.084 * Background saving started by pid 3496680
3496680:C 07 Oct 2024 04:05:05.090 * DB saved on disk
3496680:C 07 Oct 2024 04:05:05.090 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 07 Oct 2024 04:05:05.185 * Background saving terminated with success
1:M 07 Oct 2024 04:10:06.055 * 10 changes in 300 seconds. Saving...
1:M 07 Oct 2024 04:10:06.056 * Background saving started by pid 3498121
3498121:C 07 Oct 2024 04:10:06.061 * DB saved on disk
3498121:C 07 Oct 2024 04:10:06.062 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 07 Oct 2024 04:10:06.156 * Background saving terminated with success
1:M 07 Oct 2024 04:15:07.071 * 10 changes in 300 seconds. Saving...
1:M 07 Oct 2024 04:15:07.072 * Background saving started by pid 3499555
3499555:C 07 Oct 2024 04:15:07.077 * DB saved on disk
3499555:C 07 Oct 2024 04:15:07.078 * Fork CoW for RDB: current 0 MB, peak 0 MB, average 0 MB
1:M 07 Oct 2024 04:15:07.172 * Background saving terminated with success

maipal-c commented 1 week ago

all n8n pod's logs at that time -

maipal-c commented 1 week ago

worflow on which i got 5 errors in last week -

{
   "name": "payment - All - Stores - Events - processing",
   "nodes": [
      {
         "parameters": {
            "authentication": "webhook",
            "content": "=```{{ JSON.stringify($json.message.body, null, 2) }}```",
            "options": {}
         },
         "id": "74deea2e-00d9-4989-ae6f-f10d6c3f890f",
         "name": "Discord",
         "type": "n8n-nodes-base.discord",
         "typeVersion": 2,
         "position": [1060, 1180],
         "alwaysOutputData": true,
         "executeOnce": true,
         "credentials": {
            "discordWebhookApi": {
               "id": "YpSTw57doPxUEvSf",
               "name": "Discord webhook payment-invoice"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "topic": "payment-app-store-events-v3",
            "groupId": "n8n-trigger-on-payment-app-store-event",
            "options": {
               "jsonParseMessage": true
            }
         },
         "id": "6c2db656-45e1-414f-adea-eeafe5cfe38f",
         "name": "Kafka Trigger",
         "type": "n8n-nodes-base.kafkaTrigger",
         "typeVersion": 1.1,
         "position": [60, 240],
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         }
      },
      {
         "parameters": {
            "method": "POST",
            "url": "http://postgrest-db.core.svc.cluster.local/invoices",
            "authentication": "genericCredentialType",
            "genericAuthType": "httpHeaderAuth",
            "sendHeaders": true,
            "headerParameters": {
               "parameters": [
                  {
                     "name": "Prefer",
                     "value": "resolution=merge-duplicates"
                  }
               ]
            },
            "sendBody": true,
            "bodyParameters": {
               "parameters": [
                  {
                     "name": "invoiceId",
                     "value": "={{ $json.message.body.invoiceId }}"
                  },
                  {
                     "name": "originalDeliveryId",
                     "value": "={{ $json.message.body.originalDeliveryId }}"
                  },
                  {
                     "name": "isRedelivery",
                     "value": "={{ $json.message.body.isRedelivery }}"
                  },
                  {
                     "name": "type",
                     "value": "={{ $json.message.body.type }}"
                  },
                  {
                     "name": "timestamp",
                     "value": "={{ $json.message.body.timestamp }}"
                  },
                  {
                     "name": "storeId",
                     "value": "={{ $json.message.body.storeId }}"
                  },
                  {
                     "name": "metadata_amount",
                     "value": "={{ $json.message.body.metadata.amount }}"
                  },
                  {
                     "name": "metadata_orderId",
                     "value": "={{ $json.message.body.metadata.orderId }}"
                  },
                  {
                     "name": "metadata_itemDesc",
                     "value": "={{ $json.message.body.metadata.itemDesc }}"
                  },
                  {
                     "name": "metadata_isLightning",
                     "value": "={{ $json.message.body.metadata.isLightning }}"
                  }
               ]
            },
            "options": {}
         },
         "id": "383faedc-c2ed-4f8c-af05-a587ebcaa881",
         "name": "Insert in db V3",
         "type": "n8n-nodes-base.httpRequest",
         "typeVersion": 4.2,
         "position": [1060, 740],
         "alwaysOutputData": true,
         "credentials": {
            "httpHeaderAuth": {
               "id": "0KS6kBFFRuPG5Qvo",
               "name": "PostgRest db JWT Auth"
            }
         }
      },
      {
         "parameters": {
            "method": "POST",
            "url": "http://postgrest-db.core.svc.cluster.local/invoices",
            "authentication": "genericCredentialType",
            "genericAuthType": "httpHeaderAuth",
            "sendHeaders": true,
            "headerParameters": {
               "parameters": [
                  {
                     "name": "Prefer",
                     "value": "resolution=merge-duplicates"
                  }
               ]
            },
            "sendBody": true,
            "bodyParameters": {
               "parameters": [
                  {
                     "name": "invoiceId",
                     "value": "={{ $json.message.body.invoiceId }}"
                  },
                  {
                     "name": "originalDeliveryId",
                     "value": "={{ $json.message.body.originalDeliveryId }}"
                  },
                  {
                     "name": "isRedelivery",
                     "value": "={{ $json.message.body.isRedelivery }}"
                  },
                  {
                     "name": "type",
                     "value": "={{ $json.message.body.type }}"
                  },
                  {
                     "name": "timestamp",
                     "value": "={{ $json.message.body.timestamp }}"
                  },
                  {
                     "name": "storeId",
                     "value": "={{ $json.message.body.storeId }}"
                  },
                  {
                     "name": "metadata_amount",
                     "value": "={{ $json.message.body.metadata.amount }}"
                  },
                  {
                     "name": "metadata_orderId",
                     "value": "={{ $json.message.body.metadata.orderId }}"
                  },
                  {
                     "name": "metadata_itemDesc",
                     "value": "={{ $json.message.body.metadata.itemDesc }}"
                  },
                  {
                     "name": "metadata_isLightning",
                     "value": "={{ $json.message.body.metadata.isLightning }}"
                  }
               ]
            },
            "options": {}
         },
         "id": "afa53d4a-6b68-4961-a037-af7bd16e1749",
         "name": "Insert in db V",
         "type": "n8n-nodes-base.httpRequest",
         "typeVersion": 4.2,
         "position": [1060, 960],
         "alwaysOutputData": true,
         "credentials": {
            "httpHeaderAuth": {
               "id": "PbNH2rQ6CxlcWmAX",
               "name": "Postgrest db rds jwt Auth"
            }
         }
      },
      {
         "parameters": {
            "conditions": {
               "options": {
                  "caseSensitive": true,
                  "leftValue": "",
                  "typeValidation": "strict"
               },
               "conditions": [
                  {
                     "id": "62df61dc-2434-488a-96de-9d099b9ccea8",
                     "leftValue": "={{ $json.message.body.type }}",
                     "rightValue": "",
                     "operator": {
                        "type": "string",
                        "operation": "exists",
                        "singleValue": true
                     }
                  },
                  {
                     "id": "95871c06-fb0b-48b7-88c9-2c84037c4a18",
                     "leftValue": "={{ $json.message.body.type }}",
                     "rightValue": "Invoice",
                     "operator": {
                        "type": "string",
                        "operation": "startsWith"
                     }
                  }
               ],
               "combinator": "and"
            },
            "options": {}
         },
         "id": "08c943ee-fc2a-4807-8286-bb1b403daa62",
         "name": "If",
         "type": "n8n-nodes-base.if",
         "typeVersion": 2.1,
         "position": [620, 980]
      },
      {
         "parameters": {
            "topic": "payment-all-invoice-created-events-v3",
            "sendInputData": false,
            "message": "={{ JSON.stringify($json.message) }}",
            "options": {}
         },
         "id": "d8d21906-3b15-4365-b0df-846368f44481",
         "name": "kafka-invoice-created",
         "type": "n8n-nodes-base.kafka",
         "typeVersion": 1,
         "position": [1060, -800],
         "alwaysOutputData": true,
         "retryOnFail": true,
         "maxTries": 5,
         "waitBetweenTries": 5000,
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "topic": "payment-all-invoice-setteled-events-v3",
            "sendInputData": false,
            "message": "={{ JSON.stringify($json.message) }}",
            "options": {}
         },
         "id": "2b585127-ee7e-4f72-8aba-49d46f28b3f2",
         "name": "Kafka-invoice-settled",
         "type": "n8n-nodes-base.kafka",
         "typeVersion": 1,
         "position": [1060, -440],
         "alwaysOutputData": true,
         "retryOnFail": true,
         "maxTries": 5,
         "waitBetweenTries": 5000,
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "topic": "payment-all-invoice-processing-events-v3",
            "sendInputData": false,
            "message": "={{ JSON.stringify($json.message) }}",
            "options": {}
         },
         "id": "39c7a322-8220-40cc-ad74-2385b9fe72c3",
         "name": "Kafka-invoice-processing",
         "type": "n8n-nodes-base.kafka",
         "typeVersion": 1,
         "position": [1060, -620],
         "alwaysOutputData": true,
         "retryOnFail": true,
         "maxTries": 5,
         "waitBetweenTries": 5000,
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "topic": "payment-all-invoice-expired-events-v3",
            "sendInputData": false,
            "message": "={{ JSON.stringify($json.message) }}",
            "options": {}
         },
         "id": "e5f2860a-dac0-4ecc-a5a2-0658b1134a06",
         "name": "Kafka-invoice-expired",
         "type": "n8n-nodes-base.kafka",
         "typeVersion": 1,
         "position": [1060, -260],
         "alwaysOutputData": true,
         "retryOnFail": true,
         "maxTries": 5,
         "waitBetweenTries": 5000,
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "topic": "payment-all-invoice-payment-settled-events-v3",
            "sendInputData": false,
            "message": "={{ JSON.stringify($json.message) }}",
            "options": {}
         },
         "id": "b7751283-bae2-4ed1-a39c-bc56f55db1e0",
         "name": "Kafka-invoice-payment-settled",
         "type": "n8n-nodes-base.kafka",
         "typeVersion": 1,
         "position": [1060, -80],
         "alwaysOutputData": true,
         "retryOnFail": true,
         "maxTries": 5,
         "waitBetweenTries": 5000,
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "topic": "payment-all-invoice-payment-received-events-v3",
            "sendInputData": false,
            "message": "={{ JSON.stringify($json.message) }}",
            "options": {}
         },
         "id": "c0cbb641-3ff2-4ba0-aceb-4d6c796bca7b",
         "name": "Kafka-invoice-payment-received",
         "type": "n8n-nodes-base.kafka",
         "typeVersion": 1,
         "position": [1060, 300],
         "alwaysOutputData": true,
         "retryOnFail": true,
         "maxTries": 5,
         "waitBetweenTries": 5000,
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "authentication": "webhook",
            "content": "<hidden>",
            "options": {}
         },
         "id": "c8f6dcc1-78fa-4d8f-9ea0-9652dfd71cc3",
         "name": "Discord-error-notify",
         "type": "n8n-nodes-base.discord",
         "typeVersion": 2,
         "position": [1660, -300],
         "alwaysOutputData": true,
         "executeOnce": true,
         "credentials": {
            "discordWebhookApi": {
               "id": "C9fvntosl6Oo3iil",
               "name": "discord webhook test-bot"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "topic": "payment-all-invoice-invalid-events-v3",
            "sendInputData": false,
            "message": "={{ JSON.stringify($json.message) }}",
            "options": {}
         },
         "id": "2cac4a7d-aef5-476a-8837-66b13e9401d4",
         "name": "Kafka-invoice-invalid",
         "type": "n8n-nodes-base.kafka",
         "typeVersion": 1,
         "position": [1060, 120],
         "alwaysOutputData": true,
         "retryOnFail": true,
         "maxTries": 5,
         "waitBetweenTries": 5000,
         "credentials": {
            "kafka": {
               "id": "H738I7rANzImXhSW",
               "name": "Kafka account"
            }
         },
         "onError": "continueErrorOutput"
      },
      {
         "parameters": {
            "rules": {
               "values": [
                  {
                     "conditions": {
                        "options": {
                           "caseSensitive": true,
                           "leftValue": "",
                           "typeValidation": "strict"
                        },
                        "conditions": [
                           {
                              "leftValue": "={{ $json.message.body.type }}",
                              "rightValue": "/invoicecreated/ig",
                              "operator": {
                                 "type": "string",
                                 "operation": "regex"
                              }
                           }
                        ],
                        "combinator": "and"
                     },
                     "renameOutput": true,
                     "outputKey": "Invoice Created events"
                  },
                  {
                     "conditions": {
                        "options": {
                           "caseSensitive": true,
                           "leftValue": "",
                           "typeValidation": "strict"
                        },
                        "conditions": [
                           {
                              "id": "f29c4457-2371-4568-adff-d5f0dbf523c5",
                              "leftValue": "={{ $json.message.body.type }}",
                              "rightValue": "/InvoiceProcessing/ig",
                              "operator": {
                                 "type": "string",
                                 "operation": "regex"
                              }
                           }
                        ],
                        "combinator": "and"
                     },
                     "renameOutput": true,
                     "outputKey": "Invoice Processing events"
                  },
                  {
                     "conditions": {
                        "options": {
                           "caseSensitive": true,
                           "leftValue": "",
                           "typeValidation": "strict"
                        },
                        "conditions": [
                           {
                              "id": "21c87349-be25-4403-92cb-914e6031c4df",
                              "leftValue": "={{ $json.message.body.type }}",
                              "rightValue": "/InvoiceSettled/ig",
                              "operator": {
                                 "type": "string",
                                 "operation": "regex"
                              }
                           }
                        ],
                        "combinator": "and"
                     },
                     "renameOutput": true,
                     "outputKey": "Invoice Settled events"
                  },
                  {
                     "conditions": {
                        "options": {
                           "caseSensitive": true,
                           "leftValue": "",
                           "typeValidation": "strict"
                        },
                        "conditions": [
                           {
                              "id": "7928a62c-9393-4490-bc0c-48056e3e6c73",
                              "leftValue": "={{ $json.message.body.type }}",
                              "rightValue": "/InvoiceExpired/ig",
                              "operator": {
                                 "type": "string",
                                 "operation": "regex"
                              }
                           }
                        ],
                        "combinator": "and"
                     },
                     "renameOutput": true,
                     "outputKey": "Invoice Expired events"
                  },
                  {
                     "conditions": {
                        "options": {
                           "caseSensitive": true,
                           "leftValue": "",
                           "typeValidation": "strict"
                        },
                        "conditions": [
                           {
                              "id": "853a9435-7c0f-4014-a1b7-fde0802a2682",
                              "leftValue": "={{ $json.message.body.type }}",
                              "rightValue": "/InvoicePaymentSettled/ig",
                              "operator": {
                                 "type": "string",
                                 "operation": "regex"
                              }
                           }
                        ],
                        "combinator": "and"
                     },
                     "renameOutput": true,
                     "outputKey": "Invoice Payment Settled events"
                  },
                  {
                     "conditions": {
                        "options": {
                           "caseSensitive": true,
                           "leftValue": "",
                           "typeValidation": "strict"
                        },
                        "conditions": [
                           {
                              "id": "f9121a9d-4d2d-453d-91be-810c682680ec",
                              "leftValue": "={{ $json.message.body.type }}",
                              "rightValue": "/InvoiceInvalid/ig",
                              "operator": {
                                 "type": "string",
                                 "operation": "regex"
                              }
                           }
                        ],
                        "combinator": "and"
                     },
                     "renameOutput": true,
                     "outputKey": "Invoice Invalid events"
                  },
                  {
                     "conditions": {
                        "options": {
                           "caseSensitive": true,
                           "leftValue": "",
                           "typeValidation": "strict"
                        },
                        "conditions": [
                           {
                              "id": "8d6fabec-aa44-44d0-be26-3a652db3b775",
                              "leftValue": "={{ $json.message.body.type }}",
                              "rightValue": "/InvoiceReceivedPayment/ig",
                              "operator": {
                                 "type": "string",
                                 "operation": "regex"
                              }
                           }
                        ],
                        "combinator": "and"
                     },
                     "renameOutput": true,
                     "outputKey": "Invoice Received Payment events"
                  }
               ]
            },
            "options": {}
         },
         "id": "c0401bec-ce31-476a-8ba9-73c7574ceb4f",
         "name": "Switch",
         "type": "n8n-nodes-base.switch",
         "typeVersion": 3.1,
         "position": [620, -280],
         "onError": "continueErrorOutput"
      }
   ],
   "pinData": {},
   "connections": {
      "Kafka Trigger": {
         "main": [
            [
               {
                  "node": "If",
                  "type": "main",
                  "index": 0
               },
               {
                  "node": "Switch",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "If": {
         "main": [
            [
               {
                  "node": "Insert in db V",
                  "type": "main",
                  "index": 0
               },
               {
                  "node": "Insert in db V3",
                  "type": "main",
                  "index": 0
               },
               {
                  "node": "Discord",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "Kafka-invoice-settled": {
         "main": [
            [],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "Kafka-invoice-expired": {
         "main": [
            [],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "Kafka-invoice-payment-settled": {
         "main": [
            [],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "Kafka-invoice-payment-received": {
         "main": [
            [],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "Kafka-invoice-processing": {
         "main": [
            [],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "kafka-invoice-created": {
         "main": [
            [],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "Kafka-invoice-invalid": {
         "main": [
            [],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      },
      "Switch": {
         "main": [
            [
               {
                  "node": "kafka-invoice-created",
                  "type": "main",
                  "index": 0
               }
            ],
            [
               {
                  "node": "Kafka-invoice-processing",
                  "type": "main",
                  "index": 0
               }
            ],
            [
               {
                  "node": "Kafka-invoice-settled",
                  "type": "main",
                  "index": 0
               }
            ],
            [
               {
                  "node": "Kafka-invoice-expired",
                  "type": "main",
                  "index": 0
               }
            ],
            [
               {
                  "node": "Kafka-invoice-payment-settled",
                  "type": "main",
                  "index": 0
               }
            ],
            [
               {
                  "node": "Kafka-invoice-invalid",
                  "type": "main",
                  "index": 0
               }
            ],
            [
               {
                  "node": "Kafka-invoice-payment-received",
                  "type": "main",
                  "index": 0
               }
            ],
            [
               {
                  "node": "Discord-error-notify",
                  "type": "main",
                  "index": 0
               }
            ]
         ]
      }
   },
   "active": true,
   "settings": {
      "executionOrder": "v1"
   },
   "versionId": "2534a228-47df-43f1-b81d-2dee1fcf14d6",
   "meta": {
      "templateCredsSetupCompleted": true,
      "instanceId": "5ba6e44e99d313e722e1b2367d1576829fc8a963c2ec5491d27adc8a963963de7"
   },
   "id": "5RnMdEe8zBOtYQWi",
   "tags": []
}

Joffcom commented 1 week ago

Hey @maipal-c,

I have assigned this to one of the internal teams to pick up and prioritise, It could be that they pick this up quickly or they may see it is as a low priority issue as it is only 5 failure out of 10k executions which is something like a 0.05% failure rate which is pretty good.

For now have you tried tweaking the auto commit options and parallel processing to see that changes anything?

n8n-io / n8n

Queue Mode Fails #11124

Bug Description

To Reproduce

Expected behavior

Operating System

n8n Version

Node.js Version

Database

Execution mode