Open LeaCCC opened 3 months ago
Hi @LeaCCC, Thanks for the feedback! Could you set env. variables VSTSAGENT_TRACE and VSTS_AGENT_HTTPTRACE for detailed logs?
Hi Dmitrii, Thank you for looking at this issue. Do I add these two variables to the DevOps pipeline and set them to 'true'? This is where I set system.debug = true? Thanks, lea
From: Dmitrii Bobreshev @.> Sent: Friday, May 24, 2024 10:37 PM To: microsoft/azure-pipelines-agent @.> Cc: Deng, Lea @.>; Mention @.> Subject: Re: [microsoft/azure-pipelines-agent] [BUG]: Self-hosted agents stuck/stop/freeze/time out at step "Try to append 1 batches web console lines for record..." (Issue #4812)
You don't often get email from @.**@.>. Learn why this is importanthttps://aka.ms/LearnAboutSenderIdentification
Hi @LeaCCChttps://github.com/LeaCCC, Thanks for the feedback! Could you set env. variables VSTSAGENT_TRACEhttps://github.com/microsoft/azure-pipelines-agent/blob/master/src/Agent.Sdk/Knob/AgentKnobs.cs#L206 and VSTS_AGENT_HTTPTRACEhttps://github.com/microsoft/azure-pipelines-agent/blob/master/src/Agent.Sdk/Knob/AgentKnobs.cs#L306 for detailed logs?
- Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-pipelines-agent/issues/4812#issuecomment-2129208962, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEF6JH5TF6QH77JKVN2VLLTZD4J3VAVCNFSM6AAAAABIGQNMQ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMRZGIYDQOJWGI. You are receiving this because you were mentioned.Message ID: @.**@.>>
This electronic email and any files transmitted with it are intended solely for the use of the individual or entity to whom they are addressed. The views expressed in this message are those of the individual sender and may not necessarily reflect the views of the Christchurch City Council. If you are not the correct recipient of this email please advise the sender and delete the email.
[Christchurch City Council logo]https://ccc.govt.nz/
Hi @LeaCCC, You need to set them as the system environment variables , please make sure that you re-run agent after you set them.
Hi Dmitrii, thank you for your reply. I've added VSTSAGENT_TRACE and VSTS_AGENT_HTTPTRACE as environment variables on the window server where the self-hosted agents are running. Please find agent log and worker log attached. Pipeline started around 2024-05-27T22:21:48, then froze, and time out after about 20 minutes (20 minutes is the time out defined for the pipeline)
Agent_20240527-221243-utc_froze.log Worker_20240527-222136-utc_froze.log
Very much interested in the outcome of this investigation as we're experiencing the exact same problem.
These are the latest log lines of our agent job execution, where you can observe there is almost a 1h gap
[2024-05-28 07:46:07Z INFO ProcessInvokerWrapper] Finished process 21272 with exit code 0, and elapsed time 00:08:01.6348290.
[2024-05-28 07:46:07Z INFO AsyncCommandContext] Start flush buffered output.
[2024-05-28 07:46:07Z INFO JobServerQueue] Try to append 1 batches web console lines for record '1f06056c-1ae7-54e3-0278-7970ab3d0383', success rate: 1/1.
[2024-05-28 08:37:33Z INFO Worker] Cancellation/Shutdown message received.
[2024-05-28 08:37:33Z INFO ExpressionManager] Evaluating: always()
[2024-05-28 08:37:33Z INFO ExpressionManager] Result: True`
running v3.236.1 btw
What happened?
We have 7 self-hosted agents running on the server for ReadyAPI tests. When there are 2 to 7 tests running in parallel, one of the parallel tests (sometimes 2) can get stuck at the ReadyAPI task, and eventually time out. This happens 1 out of 3 trial.
Versions
Azure DevOps Self Hosted Agent version: 3.225.0, 2.239.1
Environment type (Please select at least one enviroment where you face this issue)
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Windows Server 2019 Standard
Version controll system
Azure DevOps Repo (Git)
Relevant log output