In yorc logs, we see sometimes, two task executions are registered for one step at the same time. (See two duplicated entries "All previous steps of step" in the log below). As a result, workers concurrently work on the registered task execution. They create and remove ansible files from each other (e.g., overlay files) and cause unexpected error.
2022/09/20 17:05:26 [DEBUG] All previous steps of step:"Secrule_inbound_RDSMySQL_database_endpoint_install" are done, so it can be registered to be executed
2022/09/20 17:05:26 [DEBUG] All previous steps of step:"Secrule_inbound_RDSMySQL_database_endpoint_install" are done, so it can be registered to be executed
2022/09/20 17:05:26 [DEBUG] Register task execution with ID:"445ecfcc-da88-418e-ac35-1493d3c7a057", taskID:"89f0adf7-4b64-47bc-b76e-7b74e5c58905" and step:"Secrule_inbound_RDSMySQL_database_endpoint_install"
2022/09/20 17:05:26 [DEBUG] Will store runningExecutions with id "445ecfcc-da88-418e-ac35-1493d3c7a057" in txn for task "89f0adf7-4b64-47bc-b76e-7b74e5c58905"
2022/09/20 17:05:26 [DEBUG] Register task execution with ID:"d88fc3f0-6a59-4be1-81b8-20c800ed5b81", taskID:"89f0adf7-4b64-47bc-b76e-7b74e5c58905" and step:"Secrule_inbound_RDSMySQL_database_endpoint_install"
2022/09/20 17:05:26 [DEBUG] Will store runningExecutions with id "d88fc3f0-6a59-4be1-81b8-20c800ed5b81" in txn for task "89f0adf7-4b64-47bc-b76e-7b74e5c58905"
Expected behavior
One task execution for a given step is registered.
Actual behavior
Two duplicated task executions are registered.
Additional information you deem important
This issue happens only occasionally when two or more steps join into one step. For example:
Step 1 ------> Step 3
Step 2 -----------^
Worker 1 completes Step 1.
Worker 2 completes Step 2 and registers next Step 3.
Worker 2 checks all previous steps of 3 are DONE, acquires lock for task, registers task execution A for step 3, and unlocks.
Worker 1 checks all previous steps of 3 are DONE, acquires lock for task, registers task execution B for step 3, and unlocks.
Bug Report
Description
In yorc logs, we see sometimes, two task executions are registered for one step at the same time. (See two duplicated entries "All previous steps of step" in the log below). As a result, workers concurrently work on the registered task execution. They create and remove ansible files from each other (e.g.,
overlayfiles) and cause unexpected error.Expected behavior
One task execution for a given step is registered.
Actual behavior
Two duplicated task executions are registered.
Additional information you deem important
This issue happens only occasionally when two or more steps join into one step. For example:
Step 1 ------> Step 3 Step 2 -----------^
The following lock:
https://github.com/ystia/yorc/blob/develop/tasks/workflow/step.go#L621
cannot prevent this issue.
Output of
yorc versiondevelop
Priority
Low