Log4j vulnerability

[Michael-Coetzee]

Agent Version and Platform

Version of your agent? 2.192.0

OS of the machine running the agent? Linux

What's not working?

/opt//externals/tee/lib/log4j-1.2.14.jar /opt//externals/tee/lib/slf4j-log4j12-1.7.19.jar

General Question

Are we affected with the Log4j vulnerability? Based upon the files the agent uses as seen above.

[anatolybolshakov]

Hi @Michael-Coetzee log4j is used pretty poorly by agent - this is a dependency for Tee plugin - for previous Log4j vulnerabilities use cases which agent is using were not affected - let us check it for the recent vulnerability as well.

[robertspurlin]

Does the previous Log4j vulnerabilities reviewed include this one as well? https://access.redhat.com/security/cve/CVE-2021-4104

The issue above seems to affect Log4j versions 1.x (I also see version 1.2.14 as a dependency for the agent/Tee), and the "fix" identified is to upgrade Log4j to 2.15.0. Apologies if this vulnerability was already reviewed.

[dude0001]

We identified our self-hosted agents as being potentially vulnerable as well. Can the agent be deployed without the TEE plugin, and if so what functionality do we lose? Are there any other recommendations to mitigate ahead of possible patches?

[madskristiansen]

Our Linux servers were flagged because of the log4j dependency and we had to shut down the pipeline agents.

[a1exstr]

so in order to address this vulnerability shall we just remove the jar files (replace them) or wait for an updated version of agent binaries to be released very soon ?

[anatolybolshakov]

Hi everyone, just as a quick update - we've investigated recent Log4j vulnerability (CVE-2021-44228) which affects Log4j >= 2.0 and 2.14.1. Agent uses this dependency in two locations:

• Log4j v1.2.17 - as dependency of one of the test projects (build time only) (source code line)
• Log4j v1.2.14 - as dependency of Tee plugin

So this vulnerability does not affect the agent.

Tee plugin is used for the case with TfVC repo checkout and Linux environment - for other cases it's not used at all, so possible vulnerabilities related only to this case.

Continue checking for https://access.redhat.com/security/cve/CVE-2021-4104, will let you know once we complete investigation.

[dude0001]

Thank you for the update. Is there any way to remove the TEE plugin without breaking the agent.This is a very severe issue and we'd rather play it safe and remove an unneeded component. We do not build TFVS repos from Azure Pipelines.

[alexander-smolyakov]

About CVE-2021-4104 vulnerability. Based on the vulnerability description this issue only affects the Log4j 1.2 when specifically configured to use JMSAppender, which is not the default behavior of the Log4j. As we can see from the source the Team Explorer Everywhere (abbreviated TEE) plugin doesn't use this type of appender at all. It means that this vulnerability is not relevant for the Azure Pipelines Agent.

As @anatolybolshakov mentioned earlier the azure pipelines agent uses the TEE plugin as the dependency. The agent uses only the "Command-line Client for TFS" from the TEE plugin to perform the checkout for the repositories that are based on Team Foundation Version Control (abbreviated TFCV) on non-Windows machines.

We are shipping the TEE plugin as a part of agent distribution for the following platforms:

• macOS
• Linux x64
• Linux ARM
• Linux ARM64
• RHEL 6 x64

[dude0001]

We have compliance concerns (HIPPA and HITRUST) that restrict us from using unsupported (EOL) operating systems, software or software libraries. If there are incidents involving unsupported software, large fines can be enforced. I understand the vulnerable Log4J library is coming from an external dependency. From our perspective, we don't seem to need this dependency. It greatly mitigates our risk to remove unnecessary software, especially when it is unsupported and has known vulnerabilities. The analysis done and provided above is greatly appreciated as it helps us evaluate our options to mitigate this internally, but we may need to take this a step further.

With that said, I'd like to repost my question from above. Is there any way to permanently remove from our deployment the TEE plugin and/or the Log4J library without otherwise breaking the Linux agent for non-TFS workflows? We can just give it a try in our build agent images, but it would be helpful to get some guidance on going this route for those that need it.

[alexander-smolyakov]

@dude0001 If you are not using TFVS based repositories and using self-hosted pipelines agents you actually can manually remove the TEE plugin from the agent layout. To do this, you just need to remove the tee folder from the externals folder.

Also, please note the fact that the tee folder will be restored during agent update to mitigate this, you can temporarily turn off agent pool update in the Azure DevOps to make sure that the tee folder will not be restored.

[a1exstr]

are there any plans to release updated agent zip's with non-vulnerable newer version of Log4J library for those who for some reason still use TFVC and tee

[a1exstr]

@dude0001 If you are not using TFVS based repositories and using self-hosted pipelines agents you actually can manually remove the TEE plugin from the agent layout. To do this, you just need to remove the tee folder from the externals folder.

Also, please note the fact that the tee folder will be restored during agent update to mitigate this, you can temporarily turn off agent pool update in the Azure DevOps to make sure that the tee folder will not be restored.

Removing the whole tee folder gives an error: `Error reported in diagnostic logs. Please examine the log for more details.

• /azp/agent/_diag/Agent_20211216-145427-utc.log File not found: '/azp/agent/externals/tee/license.html'`

[tinylabspace]

I haven't tried it but rather than delete the entire tee folder delete only the /lib/log4j-1.2.14.jar file. That would leave tf.cmd and license.html in /externals/tee.

[a1exstr]

yep, that is what I ended up with. And it seems to work

[sreejithvm]

@a1exstr @alexander-smolyakov Thanks, guys. How can we turn off the auto-update feature for azure agent pools once we delete the log4j.jar files from /externals/tee directory? Is it possible to do it via the azure DevOps web portal?

[MarRich1]

@a1exstr @alexander-smolyakov Thanks, guys. How can we turn off the auto-update feature for azure agent pools once we delete the log4j.jar files from /externals/tee directory? Is it possible to do it via the azure DevOps web portal?

2455

[MarRich1]

Even when Gloridel Morales mentioned that the older CVE-2019-17571 is not exploitable we/my company would appreciate when all unsecure versions will be updated

[alexander-smolyakov]

@a1exstr @alexander-smolyakov Thanks, guys. How can we turn off the auto-update feature for azure agent pools once we delete the log4j.jar files from /externals/tee directory? Is it possible to do it via the azure DevOps web portal?

You can turn off the agent update in the settings of the related agent pool. To do this, please go to Organization settings -> Pipelines -> Agent pools and open the pool for which you want to disable auto-update. On the agent pool page go to the Settings tab and turn off the Allow agents in this pool to automatically update option.

[alexander-smolyakov]

Even when Gloridel Morales mentioned that the older CVE-2019-17571 is not exploitable we/my company would appreciate when all unsecure versions will be updated

Hi @MarRich1, our team is currently investigating options for getting rid of log4j v1 from the pipelines agent distribution.

[bjtucker]

How is this going, and when can we expect more news? I do see what I hope is positive progress in an open pull request, but I am left wondering if this will remove the vulnerability entirely, and when it will be rolled out as a release.

I see this pull request that I'm hopeful will do the trick

• Download TEE plugin conditionally during checkout #3684

Does this remove the vulnerable log4j from the package? I think it does...

Looks like it's waiting on review from microsoft/akvelon-build-task-team. I know it's the holidays, but is anyone here in a position to give that team a nudge?

Turning off auto update and deleting parts of the installed package is not a great situation, especially when a good solution appears to be already in the works. Is there any status update or release time estimate you can give us @alexander-smolyakov? Is there someone different I should ask? :)

[alexander-smolyakov]

Hi all, let me share the current progress regarding this issue:

As a short-term solution, we have decided to remove the TEE plugin for agent distribution and provide the ability to download this plugin in the checkout runtime if the TEE plugin is required to perform a checkout. The related changes will be included in the next release of the pipeline agent. We will update the ticket once the agent will be available for all users.

At the same time, we are working on a long-term solution. As the main possible solution, we are investigating the possibility of rewriting the command-line client from the TEE plugin to C#. In this case, we will be able to completely remove the Java dependency from agent distribution. We will provide an update regarding the long-term solution as soon as possible.

[tinylabspace]

If your self hosted agents run behind firewalls/proxies they will need to be able to reach https://vstsagenttools.blob.core.windows.net/tools/tee/14_135_0/TEE-CLC-14.135.0.zip to download it. it might be worth checking your rules.

[DaniilShmelev]

@tinylabspace this change is supposed to be a temporary measure. We're planning to remove it once we resolve problems with TEE. Also, this logic will only affect users who use TFVC on Linux or MacOS agents. If your agent is having issues with downloading TEE, you can download it manually. We'll put up an instruction on how to do that.

[xinyi-joffre]

Is there an ETA on when the new agent version will be released? Unfortunately, this is causing a lot of vulnerability alerts, even if this is not a real vulnerability for Azure DevOps.

[DaniilShmelev]

@xinyi-joffre we're planning to finish rolling out the new version today if we don't encounter any regressions.

[alexander-smolyakov]

Hi all, since we have rolled out the agent with temporary fix let me close this ticket and open a new one for long term solution.

[MarRich1]

Can you provide a link to the new ticket?

[MarRich1]

Hi all, since we have rolled out the agent with temporary fix let me close this ticket and open a new one for long term solution.

Is the new issue already created? Can you provide the link?

[jonlorusso]

Hi all, since we have rolled out the agent with temporary fix let me close this ticket and open a new one for long term solution.

@alexander-smolyakov can you share a link to the ticket please 🙏

[llyons]

Is this fixed in agent version 2.204.0?

[g-campbell]

@llyons I have 2.204.0 installed on my agents and I am still seeing the issue.