Netflix / conductor

Conductor is a microservices orchestration engine.
Apache License 2.0
12.82k stars 2.34k forks source link

Inline Task Getting executed multiple times And giving output as null while it is actually non null #3849

Open josephjacobmorris opened 11 months ago

josephjacobmorris commented 11 months ago

Describe the bug

Screenshot 2023-11-16 at 3 53 14 PM

Inline Task gets executed multiple times and in such cases where the subsequent tasks which refer to output of this inline task sometimes (I have noticed One Attempt of the inline task is running during the same time) it is getting null and causing my Workflow execution is failing at random

Details Conductor version: 3.13.7 Persistence implementation: Postgres Queue implementation: Redis Lock: Redis Workflow definition: Task definition: Inline Task Definition :

{
  "name": "lambda_xx_migrate_0_b52",
  "taskReferenceName": "lambda_xx_migrate_0_b52",
  "inputParameters": {
    "parallelTasks": [
      "e152639b-cc9a-4f9d-83d6-c1d86744225d",
      "c6c119eb-6997-4807-a961-27e211055aac",
      "f35f4800-da23-4129-a8c5-16c9d4294718",
      "91ad9020-b74d-4db3-9c0c-dfe63bd8d441",
      "e9ace7c1-5c83-417e-bdee-458e390c1eb9",
      "5394119a-851e-4f3a-b05d-8fae956e74c9"
    ],
    "expression": "function scriptFun(){ return $.parallelTasks[$.index-1]} scriptFun();",
    "evaluatorType": "javascript",
    "index": "${loop_xx_migrate_0_b52.output.iteration}"
  },
  "type": "INLINE",
  "startDelay": 0,
  "optional": false,
  "asyncComplete": false
}

Subsequent Task Refering to Inline Task

{
  "name": "xx_migrate_0_b52",
  "taskReferenceName": "xx_migrate_0_b52",
  "inputParameters": {
    "parallelTasksList": "${workflow.input.parallelTasksList}",
    "taskId": "${lambda_xx_migrate_0_b52.output.result}"
  },
  "type": "SUB_WORKFLOW",
  "startDelay": 0,
  "subWorkflowParam": {
    "name": "xx_migrate",
    "version": 6
  },
  "optional": false,
  "asyncComplete": false
}

Event handler definition: NA

To Reproduce Steps to reproduce the behavior:

  1. Randomly occurring under high load (large number of workflows getting executed simutaneously

Expected behavior

  1. The Inline task should be executed once or the subsequent implemention should'nt overwrite the

Screenshots Inline Task Getting executed multiple times

Screenshot 2023-11-16 at 3 53 14 PM

Subsequent Task Getting output as null even though the actual result of inline task is non null

Screenshot 2023-11-16 at 4 18 35 PM

See the taskId is given as null , even though it is not

Screenshot 2023-11-16 at 4 17 24 PM

Additional context Add any other context about the problem here.

josephjacobmorris commented 11 months ago

Hi @v1r3n , Can you check this issue and in case you have any idea what might be the cause of this. I'm happy to investigate/fix with the proper guidance.