Endpoint auth. parameter that is not confidential getting masked in agent logs

[GitHubSriramB]

2.122.0

Windows

VSTS

Attaching details about the custom endpoint type & logs that has this issue.

Endpoint details logged in agent:

{
  "data": {},
  "id": "cfa7e53f-5778-423c-af65-507a80224c79",
  "name": "cfa7e53f-5778-423c-af65-507a80224c79",
  "type": "habitatoriginendpoint",
  "url": "https://bldr.habitat.sh",
  "authorization": {
    "parameters": {
      "username": "********",
      "revision": "********",
      "publickey": "********",
      "password": "********",
      "githubauthtoken": "********",
      "useSudo": "********"
    },
    "scheme": "UsernamePassword"
  },
  "isReady": false
}

Endpoint contribution type:

{
  "id": "habitat-origin",
  "description": "Habitat Origin",
  "type": "ms.vss-endpoint.service-endpoint-type",
  "targets": ["ms.vss-endpoint.endpoint-types"],
  "properties": {
    "name": "habitatoriginendpoint",
    "displayName": "Habitat Origin",
    "url": {
      "displayName": "Habitat Depot URL",
      "helpText": "URL to the Habitat depot that will be used to deploy to"
    },
    "inputDescriptors": [],
    "authenticationSchemes": [
      {
        "type": "ms.vss-endpoint.endpoint-auth-scheme-basic",
        "inputDescriptors": [
          {
            "id": "username",
            "name": "Origin Name",
            "description": "Name of the Habitat origin",
            "inputMode": "textbox",
            "isConfidential": false,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "revision",
            "name": "Revision",
            "description": "Revision of the origin to use",
            "inputMode": "textbox",
            "isConfidential": false,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "publickey",
            "name": "Public Key",
            "description": "Public item of the origin key pair",
            "inputMode": "textarea",
            "isConfidential": false,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "password",
            "name": "Signing Key",
            "description": "Signing item of the origin key pair",
            "inputMode": "textarea",
            "isConfidential": true,
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "githubauthtoken",
            "name": "GitHub Auth Token",
            "description": "Authentication token for GitHub for publishing Habitat packages.",
            "inputMode": "textbox",
            "isConfidential": "true",
            "validation": {
              "isRequired": true,
              "dataType": "string"
            }
          },
          {
            "id": "useSudo",
            "name": "Use Sudo",
            "description": "Use sudo on habitat commands",
            "inputMode": "combo",
            "isConfidential": false,
            "validation": {
              "isRequired": false,
              "dataType": "string"
            },
            "values": {
              "inputId": "useSudoValues",
              "defaultValue": "1",
              "possibleValues": [
                {
                  "value": "1",
                  "displayValue": "True"
                },
                {
                  "value": "0",
                  "displayValue": "False"
                }
              ]
            }
          }
        ]
      }
    ],
    "helpMarkDown": "Please provide the requested information so that Habitat packages can be built and deployed. The GitHub authentication token requires `user:email` and `read:org` scopes"
  }
}

[bryanmacfarlane]

Are you sure this is an agent issue (and not lib/service)?

[GitHubSriramB]

@bryanmacfarlane - I was referring to this piece of code in agent that needs a fix: https://github.com/Microsoft/vsts-agent/blob/327cde0be616c9809168b160f9431702b7b46e1c/src/Agent.Worker/Worker.cs#L144

[GitHubSriramB]

Fix for this issue is blocked due to a service-side bug in Build. Waiting for the service-side bug to be fixed before we can take up this fix.

@ericsciple - pl. assign this back to us once the service-side bug is fixed.

[chperich]

This feedback ticket has been raised recently by a user. Character 1 gets masked in the user's logs because of masking all endpoint authorization parameters.

[peterox]

Is there any update to this issue? I am also having the problem where all '1' characters are being masked.

[chperich]

• Lova/Omesh, who will be able to address this better.

From: Peter Oxenham notifications@github.com Sent: Wednesday, December 12, 2018 9:50 AM To: Microsoft/azure-pipelines-agent azure-pipelines-agent@noreply.github.com Cc: Chandana Pericharla chperich@microsoft.com; Comment comment@noreply.github.com Subject: Re: [Microsoft/azure-pipelines-agent] Endpoint auth. parameter that is not confidential getting masked in agent logs (#1207)

Is there any update to this issue? I am also having the problem where all '1' characters are being masked.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FMicrosoft%2Fazure-pipelines-agent%2Fissues%2F1207%23issuecomment-446456779&data=02%7C01%7Cchperich%40microsoft.com%7C0ed0afe3b6d240e565a508d65fe916c2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636801852050381035&sdata=oxESjX0NhVoQGh%2BdLQ%2B0IQ0OABC1NNMvWpL%2F0TKRido%3D&reserved=0, or mute the threadhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAbtqD33iOswRcL7CPtn-j4v_p5bsYOZ7ks5u4IPzgaJpZM4PXFhR&data=02%7C01%7Cchperich%40microsoft.com%7C0ed0afe3b6d240e565a508d65fe916c2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636801852050381035&sdata=J2OiyI46epiJY96F4U1X1CoMgwXKhjjb3P1nFSS2GOA%3D&reserved=0.

[omeshp]

@ericsciple should be able to provide update on build side issue.

[ericsciple]

@peterox I believe this was fixed. Are you using the service "Azure Devops" and the latest agent? Or are you using on-premises TFS? Note you can right click on the pool to update all agents in a pool to the latest version.

[peterox]

@ericsciple I've just updated all the agents and I'm still getting all '1''s masked in the logs.

I disabled a Docker task that had an Azure subscription configured and ran the build and the issue was not there. Re-enabling the task and running the job again caused the issue to re-appear.

[ericsciple]

@TingluoHuang do you remember context on this? it looks like this code is still in master. I thought this was removed months ago?

[peterox]

Is there a work around available to prevent masking?

[peterox]

Bump

[willgarcia]

Just confirming that the latest Github released version (2.144.2) does not fix this issue after redeployment of my agents.

The HTTP link in the Web UI is also pointing to version 2.144.0 (instead of 2.144.2).

Current version of worker.cs on master is from Aug 17, 2018 (ab40f0b1..).

@ericsciple Would you be able to (un)confirm if the bug is still on master or a fixed on another branch?

[deadlydog]

We are using hosted Azure DevOps in the cloud with on-prem agents running version vsts-agent-win-x64-2.144.0 and we are also seeing this problem still.

[peterox]

Are we able to get an update on this? It seems there is a fix but it just hasn't been released?

This problem effectively makes the logs in our pipelines useless.

[willgarcia]

As spotted by @peterox, removing our Azure DevOps Docker tasks from pipelines seems to solve the problem.

Specific tasks types that require access to secrets may interfere and cause this issue.

[thewellington]

We experience this too - and our logs are useless because of it. How do we get this resolved?

[Ivers07]

We are also experiencing this issue. For us, *** replaces all spaces in the logs of any pipeline or release that uses a service connection for our service fabric instance.

It only seems to apply to tasks within the Job that uses the connection. If the release has more than one job then the other jobs are OK, but not ones that use the service connection.

Also, i found this issue through the feedback ticket mentioned by @chperich

This feedback ticket has been raised recently by a user. Character 1 gets masked in the user's logs because of masking all endpoint authorization parameters.

but it wasn't very clear from the title or original description that it was the Service Connection causing masking of incorrect characters

[frackham]

I'm getting this with strings being set as VSO variables within a powershell script task in Azure Devops (so also via service connection). The strings are not secret, but the value matches a secret elsewhere (in this case, a guid... and other guids are not masked).

I can get around it by masking the value: (blanking out some of the values for obvious reasons) None of these are secrets, all are set as release (VSO) variables.

1. The first is the value that matches an existing secret.
2. The second has x added at the start and end of the string.
3. The third has x inserted inside the string, unmasking it.

Looks to me like pattern matching is being applied to values against known secrets, to identify whether they should be obfuscated.

In this case, it's fine for me as I'm only using this for debugging, but seems like an odd way to identify secrets.

Agents are up to date, and no difference between agents.

[grahamauty]

For me, it's all lowercase 'b's and 'c's that are being masked with ***. These are obviously not secrets! This is happening in every log for every task (40 of them) in the release pipeline, which is running on Azure Devops with the latest agents.

Example:

This makes it very difficult to read most of the logs (the above example is mild compared to some of it). Can someone please update the status of this problem which was raised in 2017.

[Orthaanc]

In my case it is lower case 'a' being masked.

At the moment, I'm trying to investigate where the variables that are getting masked are collected. There are a number of Pre-defined Pipeline variables that correspond with characters that are being mentioned in this issue.

Build.BinariesDirectory lists the example c:\agent_work\1\b Agent.BuildDirectory (and Pipeline.Workspace) lists /home/vsts/work/1 Build.Repository.LocalPath c:\agent_work\1\s Build.ArtifactStagingDirectory c:\agent_work\1\a

[Paxilein]

For me it's the Azure Region's Name being secret... Which of course makes perfect sense....

[RamkiUIPath]

Is there any workaround for this? In our pipelines, test results will be published to a URL and runId gets masked 🤦‍♂️

[ajayvtiwari]

Any luck fixing masking of the characters issue in logs? I have tried the following 1) Used the latest Ubuntu build agent 2) SET the variable SYSTEM_DONOTMASKMULTILINESECRETS=TRUE

Did not work. Please help

[StingyJack]

This is happening to at least one of our pipelines, but its everything from DateTime stamps to task version, etc. There are no secrets defined in this particular pipeline.

From a powershell script task Task versions, timestamps, paths replaced partially with *

Starting: PowerShell Script
==============================================================================
Task         : PowerShell
Description  : Run a PowerShell script on Linux, macOS, or Windows
Version      : ***.***80.***
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/powershell
==============================================================================
Generating script.
========================== Starting Command Output ===========================
"C:\Windows\System3***\WindowsPowerShell\v***.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_temp\a60697d3-63d4-4fc5-b0***a-ce7***39aea8d9.ps***'"
[***0***/05/***4 06:***3:39] Transfer summary:
-----------------
Total files transferred: 9
Transfer successfully:   9
Transfer skipped:        0
Transfer failed:         0
Elapsed time:            00.00:00:00
Finishing: PowerShell Script

EDIT: @ericsciple - if you need a repro, this can caused by having the Azure Key Vault task in a pipeline, even when its not used by any of the rest of the pipeline tasks. Just add it to an existing pipeline as the first task and configure it to connect to a key vault.

[StingyJack]

@jtpetty - not sure if you are the correct person to tag but you have more recent activity than @ericsciple does in this repo. Can someone please address this? Obscured logs make troubleshooting and resolution take longer.

[itecompro]

This issue is open for years and no one has yet figured out what's going on!? 🙄

In our pipelines, we are using Variable Groups connected to a Key Vault, right after the Download Secret task, all 1s are replaced with ***.

Have checked all the secrets in the Key Vault. None of them is 1. Of course, it could be part of a secret like something_1_something_else, but not by itself as a whole.

[anatolybolshakov]

Hi everyone, prioritizing this item to check if there is something still need to be fixed after the PR.

[anatolybolshakov]

@StingyJack @itecompro I believe the issue with masked '1' should be already fixed and shipped with the next agent release - we are planning to start it this week.

[StingyJack]

@anatolybolshakov - thats great. I will direct the tech assigned to the support case we have open to this thread so they can follow up directly if necessary.

EDIT: This will things like I mentioned in the above post, right?

[anatolybolshakov]

@StingyJack yes correct, we observed the same issue with '1' being masked - when there are task which use Azure service connections.

[anatolybolshakov]

Hi @GitHubSriramB closing this one since changes has been rolled out - feel free to ask any other questions.

[StingyJack]

@anatolybolshakov - The engineer assigned to a support case we have open for this asked me to check our build logs now, and we are still having the problem (and he is on leave for the next week or so). Are you sure this is not still awaiting deployment? It is the presence of an Azure Key Vault task that triggers this for us.

Its actually kind of bad because some of the things its masking are things I can kind of easily guess, which means I now know the values of some of the secrets. For example

2022-03-12T23:40:40.2210221Z ##[section]Starting: Use Node 1***.18.***
2022-03-12T23:40:40.2244542Z ==============================================================================
2022-03-12T23:40:40.2245097Z Task         : Node.js tool installer
2022-03-12T23:40:40.2245714Z Description  : Finds or downloads and caches the specified version spec of Node.js and adds it to the PATH
2022-03-12T23:40:40.2246490Z Version      : 0.***00.0
2022-03-12T23:40:40.2246928Z Author       : Microsoft Corporation
2022-03-12T23:40:40.2247457Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/node-js
2022-03-12T23:40:40.2248059Z ==============================================================================
2022-03-12T23:40:40.3932609Z Downloading: https://nodejs.org/dist/v1***.18.***/node-v1***.18.***-linux-x64.tar.gz
2022-03-12T23:40:40.6562110Z Extracting archive
2022-03-12T23:40:40.6587983Z [command]/usr/bin/tar xC /home/vsts/work/_temp/14775***7***-0fb9-4***8***-bf57-***ae8199eab81 -f /home/vsts/work/_temp/090***8e***c-174b-4ac***-8a76-f95b8***8c88a5
2022-03-12T23:40:41.2737568Z Caching tool: node 1***.18.*** x64
2022-03-12T23:40:41.8463078Z Prepending PATH environment variable with directory: /opt/hostedtoolcache/node/1***.18.***/x64/bin
2022-03-12T23:40:41.9082799Z ##[section]Finishing: Use Node 1***.18.***

From this I know that a few of the secret values are 2 and 3 - kind of silly to put a value like this in secrets, but I think its sillier to mask it. Other tasks have similar trivial "secrets" being masked like "dev" or some filesystem paths, etc.. Some build tasks are more revealing, like ones that construct a document like an emailed summary or release notes. Those tend to have a bunch of *** in the logs.

[StingyJack]

@anatolybolshakov - Should this fix have made it to ring M201.1 by now?

[HofmeisterAn]

@anatolybolshakov We still have the issue with the agent 2.202.1. How does #3661 cover not confidential inputs? It looks like the change does only fix the problem for some EndpointAuthorizationParameters parameters, but what about custom extensions (inputDescriptors)?

[alexander-smolyakov]

Hey @HofmeisterAn, let us take another look at this issue.

[StingyJack]

@alexander-smolyakov also see this, it's affecting hundreds of pipelines my org has, and we have a support case open for this.

[anatolybolshakov]

@StingyJack @HofmeisterAn all service connection parameters are being masked by default for security reasons - there is no generic mechanism at the moment to distinguish between secret/non secret fields of service connection - hence it's not recommended to use fields with short values for custom service connections currently. Since service connection definitions is the part of ADO itself - I would suggest to open a ticket on https://developercommunity.visualstudio.com/search?space=21 for such possible enhancement

[HofmeisterAn]

there is no generic mechanism at the moment to distinguish between secret/non secret fields

The inputDescriptor has a specific field for that: isConfidential.

hence it's not recommended to use fields with short values for custom service connections currently

Where is that?

[anatolybolshakov]

The inputDescriptor has a specific field for that: isConfidential.

This is the schema of custom endpoint, but agent relies on ServiceEndpoint definition while working with service connections data (coming from external assembly - containing some generic interfaces and classes for interaction with Azure DevOps). This assembly is not the part of the agent, but is more related to Azure DevOps side - updating this would probably require other changes on ADO side as well I believe...

Where is that?

This is actual for inputs specified in custom endpoint schema (with input descriptors).

[StingyJack]

all service connection parameters are being masked by default for security reasons - there is no generic mechanism at the moment to distinguish between secret/non secret fields of service connection - hence it's not recommended to use fields with short values for custom service connections currently.

Sorry, @anatolybolshakov - I didnt say anything about service connections. We use service connections for pipelines because my org's repos are hosted on GitHub, and those dont cause this. The inappropriate masking of values in the logs is affecting pipelines that have an Azure Key Vault task that is enabled. If the task is disabled this problem goes away.

If you think that this is just a problem with endpoint authorization it means the actual problem is still not being recognized. The pipelines are masking values parts of the logs that will not contain or display secrets, and that behavior is allowing me to guess what some of the secret values are. The masking logic is actually exposing the secrets that it is supposed to be hiding.

One of the responses we got on our support case was that "this was by design". I've gently rejected that and am giving them the benefit of the doubt by choosing to believe that whomever said that also just doesnt quite understand what is being explained or reported as the problem. Also because there is no documentation that exists that warns users of the potential exposure, which I would expect of any potential leakage that could be caused by specific configurations or misuse.

[StingyJack]

I would also like to point out that all of the reports of this issue on the developer community site are getting closed as duplicates and are referring to this issue as the place where we should be tracking, so please do not ask us to "open another issue" or anything like that because we have reported the issue correctly already.

[HofmeisterAn]

The inappropriate masking of values in the logs is affecting pipelines that have an Azure Key Vault task that is enabled.

It also happens for custom extensions.

[anatolybolshakov]

@StingyJack we issue we observed happens with tasks which use specific service connection - Azure Key Vault uses Azure service connection which already has internal fields with values like '1' (if the task is disabled in pipeline - connection won't be used, hence this value won't be masked). We have confirmation that this issue was fixed by PR (we excluded some known non-secret parameters available) - this fix has been rolled out with v2.198.0 agent release. We also investigated task itself - it masks only one secret value, but we determined that it can't be '1'. It is strange that you are still seeing this issue with the latest agent... We could probably miss something during this fix, or there is some other reason for that. I checked the logs from devcom ticket - seems like the latest agent used there is 2.187.2 (which don't have this fix). Could you please confirm that it is reproducible for you with the latest version (or at least higher than 2.198.0)? Could you please also share the pipeline and agent logs for the latest agent version - with 'System.Debug' and 'Agent.Diagnostic' pipeline variables set to 'true'? We've made several improvements for masker logging recently, and this should help to analyze if there are some other inner service connection fields being masked we could miss, or if there is some other reason for this issue.

[StingyJack]

@anatolybolshakov - We only have a few builds that require us to host an agent. The rest are all AzDo hosted agents which are running the pipelines that have the inappropriate masking, so I'm only looking into these hosted agents. From a recent log...

2022-05-11T00:30:48.6258170Z Current agent version: '2.202.1'

I ran a pipeline with debug = true and System Diagnostics enabled and attached the logs to the support case we have open. I believe the support engineer has forwarded them to you at this point.

We also investigated task itself - it masks only one secret value, but we determined that it can't be '1'.

The logs attached appear to have values like "1", "2", "Dev", etc. being masked. Its masking them in places like the task version header, or in the timestamps, etc. See one of the examples I have already shared for a better idea.

[KonstantinTyukalov]

Hey everyone! We are introducing new agent knob: AGENT_IN_SECRET_LENGTH which will let you not to mask short secrets. You can specify values from 0 (all secrets will be masked as usual) to 4 (secrets shorten than 4 symbols will be ignored) Also, negative values will mean masking all the secrets as usual.

example:

  variables:
    AGENT_MIN_SECRET_LENGTH: 1

also it's available as environment variable.

This feature will be available in the next agent release, we will notify you once it will be rolled out. Thanks!

UPD: The new knob name is AZP_IGNORE_SECRETS_SHORTER_THAN

[StingyJack]

@KonstantinTyukalov is it going to stop the changing of dates and times on the logs or the header information for tasks?
There should be nothing masking these areas of the logs as secret values. This is a large part of the problem because that is the information that can be used to guess the secret values.

The request was for you to not mask trivial (single character) or values that are well known to not be secret, like "Dev" or "32" or "Q".

A length setting is not useful if the agent is still possibly going to mask easily discernable values like those. Is this an opt-in setting?

I think you guys missed the point still. It's not about not masking small values, the problem is you're masking values that are not secrets that help me reliably guess the secrets.

Also the name of the variable AGENT_IN_SECRET_LENGTH does not make sense. I would not be able to use that in a sentence explaining what it does.

[DenisRumyantsev]

@StingyJack I have tested that timestamps remain and are not affected by secret masking:

Also, please notice that the name of the variable is AGENT_MIN_SECRET_LENGTH , not AGENT_IN_SECRET_LENGTH

[StingyJack]

I have tested that timestamps remain and are not affected by secret masking

@DenisRumyantsev - Yes and I notice the log is now totally useless as well.

Also, please notice that the name of the variable is AGENT_MIN_SECRET_LENGTH , not AGENT_IN_SECRET_LENGTH

That makes more sense but its not what @KonstantinTyukalov said it was called.

We are introducing new agent knob: AGENT_IN_SECRET_LENGTH which will let you not to mask short secrets.

That name still doesnt refer to the masking which its affecting. Which is not as important as that you are still missing the point. We dont want you to avoid masking secrets. This solution makes the problem worse as it allows me to expose secrets directly instead of just inferring them indirectly.

The current design may be simple but it is far too aggressive, We want you to avoid masking data that is not a secret.

01 - 2022-03-12T23:40:40.2210221Z ##[section]Starting: Use Node 1***.18.***
02 - 2022-03-12T23:40:40.2244542Z ==============================================================================
03 - 2022-03-12T23:40:40.2245097Z Task         : Node.js tool installer
04 - 2022-03-12T23:40:40.2245714Z Description  : Finds or downloads and caches the specified version spec of Node.js and adds it to the PATH
05 - 2022-03-12T23:40:40.2246490Z Version      : 0.***00.0
06 - 2022-03-12T23:40:40.2246928Z Author       : Microsoft Corporation
07 - 2022-03-12T23:40:40.2247457Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/tool/node-js
08 - 2022-03-12T23:40:40.2248059Z ==============================================================================
09 - 2022-03-12T23:40:40.3932609Z Downloading: https://nodejs.org/dist/v1***.18.***/node-v1***.18.***-linux-x64.tar.gz
10 - 2022-03-12T23:40:40.6562110Z Extracting archive
11 - 2022-03-12T23:40:40.6587983Z [command]/usr/bin/tar xC /home/vsts/work/_temp/14775***7***-0fb9-4***8***-bf57-***ae8199eab81 -f /home/vsts/work/_temp/090***8e***c-174b-4ac***-8a76-f95b8***8c88a5
12 - 2022-03-12T23:40:41.2737568Z Caching tool: node 1***.18.*** x64
13 - 2022-03-12T23:40:41.8463078Z Prepending PATH environment variable with directory: /opt/hostedtoolcache/node/1***.18.***/x64/bin
14 - 2022-03-12T23:40:41.9082799Z ##[section]Finishing: Use Node 1***.18.***

Consider the following... If logging the Task header info (lines 2-8), dont apply masking because secrets cannot be exposed here by anyone authoring a pipeline If logging an agent-emitted section starting (line 1) or finishing (line 14), dont apply masking because secrets cannot be exposed here by anyone authoring a pipeline. If the task configuration does not reference any secrets, dont apply masking because no secrets are being used in the task. If the task configuration does reference secrets, mask only those secret values that are referenced, and only in the places where the reference would be emitted etc..

[StingyJack]

@DenisRumyantsev @KonstantinTyukalov @anatolybolshakov @alexander-smolyakov - I pointed out that your proposed solution creates an even larger problem than was being reported, and you have nothing to say in the last month?

I really hope that you are not still considering this AGENT_SECRET_MIN_LENGTH as viable, because nobody asked for you for a knob that let us expose any secrets. We asked you to stop making the secret values easy to guess.