Task is marked as "TIME_OUT" after an interval much larger than the time specified in "responseTimeoutSeconds"

[caubechankiu]

Describe the bug Task is marked as "TIME_OUT" after an interval much larger than the time specified in "responseTimeoutSeconds"

Details Conductor version: 3.18.8 Persistence implementation: MySQL Workflow definition:

{
  "createTime": 1693020291627,
  "updateTime": 1693020540567,
  "accessPolicy": {},
  "name": "simple_workflow",
  "description": "A Simple Workflow",
  "version": 1,
  "tasks": [
    {
      "name": "simple_task",
      "taskReferenceName": "simple_task",
      "inputParameters": {
        "hellomsg": "${workflow.input.hellomsg}"
      },
      "type": "SIMPLE",
      "startDelay": 0,
      "optional": false,
      "asyncComplete": false
    }
  ],
  "inputParameters": [
    "hellomsg"
  ],
  "outputParameters": {},
  "schemaVersion": 2,
  "restartable": true,
  "workflowStatusListenerEnabled": false,
  "ownerEmail": "example@email.com",
  "timeoutPolicy": "ALERT_ONLY",
  "timeoutSeconds": 0,
  "variables": {},
  "inputTemplate": {}
}

Task definition:

{
  "name": "simple_task",
  "description": "A Simple Task",
  "retryCount": 3,
  "timeoutSeconds": 60,
  "inputKeys": [
    "hellomsg"
  ],
  "outputKeys": [],
  "timeoutPolicy": "TIME_OUT_WF",
  "retryLogic": "FIXED",
  "retryDelaySeconds": 10,
  "responseTimeoutSeconds": 10,
  "inputTemplate": {},
  "rateLimitPerFrequency": 0,
  "rateLimitFrequencyInSeconds": 1,
  "ownerEmail": "example@email.com",
  "backoffScaleFactor": 1
}

To Reproduce Steps to reproduce the behavior:

1. Call api to start new workflow

   curl -X 'POST' \
   'http://localhost:8080/api/workflow/simple_workflow?priority=0' \
   -H 'accept: text/plain' \
   -H 'Content-Type: application/json' \
   -d '{
   "hellomsg": "World"
   }'

   response

   56f869dc-617a-469c-bbe7-635e1fd823b6

2. Call api to poll task

   curl -X 'GET' \
   'http://localhost:8080/api/tasks/poll/simple_task' \
   -H 'accept: */*'

   response

   {
   "taskType": "simple_task",
   "status": "IN_PROGRESS",
   "inputData": {
   "hellomsg": "World"
   },
   "referenceTaskName": "simple_task",
   "retryCount": 0,
   "seq": 1,
   "pollCount": 1,
   "taskDefName": "simple_task",
   "scheduledTime": 1693025902375,
   "startTime": 1693026064953,
   "endTime": 0,
   "updateTime": 1693026064954,
   "startDelayInSeconds": 0,
   "retried": false,
   "executed": false,
   "callbackFromWorker": true,
   "responseTimeoutSeconds": 10,
   "workflowInstanceId": "56f869dc-617a-469c-bbe7-635e1fd823b6",
   "workflowType": "simple_workflow",
   "taskId": "91646b0c-1751-427a-bd7e-f17e87784a35",
   "callbackAfterSeconds": 0,
   "outputData": {},
   "workflowTask": {
   "name": "simple_task",
   "taskReferenceName": "simple_task",
   "inputParameters": {
     "hellomsg": "${workflow.input.hellomsg}"
   },
   "type": "SIMPLE",
   "startDelay": 0,
   "optional": false,
   "taskDefinition": {
     "createTime": 1693025612726,
     "createdBy": "",
     "accessPolicy": {},
     "name": "simple_task",
     "description": "A Simple Task",
     "retryCount": 3,
     "timeoutSeconds": 60,
     "inputKeys": [
       "hellomsg"
     ],
     "outputKeys": [],
     "timeoutPolicy": "TIME_OUT_WF",
     "retryLogic": "FIXED",
     "retryDelaySeconds": 10,
     "responseTimeoutSeconds": 10,
     "inputTemplate": {},
     "rateLimitPerFrequency": 0,
     "rateLimitFrequencyInSeconds": 1,
     "ownerEmail": "example@email.com",
     "backoffScaleFactor": 1
   },
   "asyncComplete": false
   },
   "rateLimitPerFrequency": 0,
   "rateLimitFrequencyInSeconds": 1,
   "workflowPriority": 0,
   "iteration": 0,
   "subworkflowChanged": false,
   "taskDefinition": {
   "createTime": 1693025612726,
   "createdBy": "",
   "accessPolicy": {},
   "name": "simple_task",
   "description": "A Simple Task",
   "retryCount": 3,
   "timeoutSeconds": 60,
   "inputKeys": [
     "hellomsg"
   ],
   "outputKeys": [],
   "timeoutPolicy": "TIME_OUT_WF",
   "retryLogic": "FIXED",
   "retryDelaySeconds": 10,
   "responseTimeoutSeconds": 10,
   "inputTemplate": {},
   "rateLimitPerFrequency": 0,
   "rateLimitFrequencyInSeconds": 1,
   "ownerEmail": "example@email.com",
   "backoffScaleFactor": 1
   },
   "queueWaitTime": 162578,
   "loopOverTask": false
   }

3. Go to http://localhost:5000/execution/56f869dc-617a-469c-bbe7-635e1fd823b6?task=(id%3A%2791646b0c-1751-427a-bd7e-f17e87784a35%27)
4. See result

Expected behavior The duration of the task will be around 10 seconds as specified in "responseTimeoutSeconds"

Actual Behavior The duration of the task is 24 seconds. I have tried several times but the result is still the same, the duration of the task is around 20-30 seconds

Screenshots

1. The task result
2. The task definition

[prajwalbandak]

Hey @caubechankiu , will love to work on this issue, Can you assign this one to me?