Open nkr0 opened 8 months ago
@nkr0, can you provide additional downloadable logs? Method CancelAndKillProcessTree should send signals and kill the process tree. Also, you can check the output of the variable value.
I'm not sure if it is safe to share downloadable logs. But I've it and if you ask for something I can share relevant info here.
grep -r "Force kill process on cancellation"
returned
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 07:14:45Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 07:18:06Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 07:53:42Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 08:55:15Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 09:07:59Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 09:11:22Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 09:15:55Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 09:18:55Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 09:23:01Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 09:30:00Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 10:54:51Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 11:09:39Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 11:17:47Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 11:39:35Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 13:03:10Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 13:27:23Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 13:29:31Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 13:31:38Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 13:57:33Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 13:59:02Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:15:54Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:17:59Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:20:37Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:27:38Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:29:34Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:31:08Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:41:38Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:53:42Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 14:56:03Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 15:03:26Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 15:06:59Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 15:14:08Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 15:19:29Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 15:27:06Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 16:01:35Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 16:40:38Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:45:59Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:46:54Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:48:33Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:50:01Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:50:47Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:55:52Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:56:29Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Worker_20231218-205629-utc.log:[2023-12-18 20:56:31Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Worker_20231218-205629-utc.log:[2023-12-18 20:56:43Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Worker_20231218-205629-utc.log:[2023-12-18 20:56:43Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
Quite weird that Build 20231218.52
initiated around 2023-12-18 20:56:43
has log entries at 2023-12-18 07:14:45
From the timestamps, I would say that the last 4 lines are from the same run. And it looks like the agent is sending force kill, before the worker gets to try SIGINT
and SIGTERM
.
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Agent_20231217-170531-utc.log:[2023-12-18 20:56:29Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'True'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Worker_20231218-205629-utc.log:[2023-12-18 20:56:31Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Worker_20231218-205629-utc.log:[2023-12-18 20:56:43Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
Agent Diagnostic Logs/Build 20231218.52-JenkinsBuild/Worker_20231218-205629-utc.log:[2023-12-18 20:56:43Z INFO ProcessInvokerWrapper] Force kill process on cancellation: 'False'
New issue checklist
Task name
PythonScript
Task version
0.231.0
Issue Description
This page says that
SIGINT
orSIGTERM
is sent on job cancellation. Below is a simplified task callingtestSigint.py
.But the logs show that
cancelJob
was not called, suggesting that the process was not ended with any of the signals I tried. Running the script locally and pressingCtrl+C
triggersSIGINT
and is caught as expected.Issue #16731 is similar in that
SIGINT
from the agent becomesSIGTERM
by the time it reached the child process. But here it's none of those.Environment type (Please select at least one enviroment where you face this issue)
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Ubuntu 22.04
Relevant log output
Full task logs with system.debug enabled
Repro steps
No response