search

microsoft / azure-pipelines-agent

Azure Pipelines Agent 🚀
MIT License
1.72k stars 865 forks source link

Azure Pipelines Agents fills %temp% and does not clean up after itself #2680

Closed miracle152004 closed 4 years ago

miracle152004 commented 4 years ago

Agent Version and Platform

2.136.1 - 2.153.1

OS of the machine running the agent? Windows

Azure DevOps Type and Version

AzureDevops on prem

If on-premises, which release? Version Dev17.M153.5

What's not working?

The BuildAgent no longer uses the _temp directory in its working folder but instead using the folder defined in the TEMP/TMP-Environment Variable. This would not be a problem if the build did not leave no longer needed files (i.e. .tmp, .pdb) I think the build agent should at least clean up the files it creates.

Could you please take a look.

Thanks.

Best regards miracle152004

jtpetty commented 4 years ago

Looking at the code, it does appear that the agent should clean up after itself in _temp. It is possible that a task is using TEMP/TMP environment variable.

Could you try setting the environment variable VSTS_OVERWRITE_TEMP=1 for the agent process? This will replace TEMP/TMP to point to the agent's _temp directory and should get cleaned up.

miracle152004 commented 4 years ago

@jtpetty: Thanks, that works. I do not know what changed. But we had one agent running on an older version with the same buildsteps and that one used the _temp directory. Only the newer agents used the TEMP/TMP directory of the user. One of the buildsteps which has written the pdb file was the normal MSBuild Build Step.

jtpetty commented 4 years ago

Where the versions of all the tasks the same?

miracle152004 commented 4 years ago

It should be, because it happend for the same Build on different agents. But I will take a look tomorrow to see if we still have logs of the builds on the agent with the older version.

miracle152004 commented 4 years ago

I verified it. Both Agents used the version 1.151.1 of the msbuild task. The older agent had the version 2.122.2

jtpetty commented 4 years ago

Version 2.122.2 is quite an old agent version, so it does not surprise me that there was some behavior differences. That predates my time with this project by a lot but I am sure that the change was made to address a particular bug or failure syndrome.

It sounds like with that environment variable you are unblocked. Is there anything further I can help with regarding this issue?

miracle152004 commented 4 years ago

@jtpetty : No thanks it work for us now. It would just be great if there would be a knowledge base or wiki article describing such environment variables. I could not find any before posting the question here.

jtpetty commented 4 years ago

@vtbassmatt - Do we have any docs where we list out these environment variable controlled options?

vtbassmatt commented 4 years ago

None that I know of. Let's huddle next week and see if we want to tackle better config management for the agent. If so, I'd make it first class in the agent binary (to list settings, show their values, and describe how to set them). If not, we can just do a doc.

jtpetty commented 4 years ago

@vtbassmatt - Since we are moving forward with a centralized config management system for the agent, I am going to go ahead and close this issue.