search

microsoft / azure-pipelines-agent

Azure Pipelines Agent 🚀
MIT License
1.72k stars 866 forks source link

Some Pipeline variables not being resolved on Agent 2.165.x, works up through 2.164.8 #2842

Closed OranguTech closed 4 years ago

OranguTech commented 4 years ago

Have you tried troubleshooting?

Troubleshooting doc Yes, system.debug=true

Agent Version and Platform

Version of your agent? 2.165.0/2.165.1

OS of the machine running the agent? Windows, mostly Server 2016 but tested on Windows 10 1809 as well.

Azure DevOps Type and Version

dev.azure.com

If dev.azure.com, what is your organization name? https://dev.azure.com/{organization} or

'REDACTED' for now.

What's not working?

"A lot" of variables from our libraries aren't getting enumerated/expanded upon initialization, where they were before. I've narrowed down the issue to version of agent (on-prem), between 2.164.8 and 2.165.0.

Examples (potentially sensitive data obfuscated): Working:

2020-03-06T00:56:37.2355860Z ##[section]Starting: Initialize job
2020-03-06T00:56:37.2356368Z Agent name: 'ORG7IT11919 - 2.164.8'
2020-03-06T00:56:37.2356606Z Agent machine name: 'ORG7IT11919'
2020-03-06T00:56:37.2356759Z Current agent version: '2.164.8'
2020-03-06T00:56:37.2371109Z Agent running as: 'ORG7IT11919$'
2020-03-06T00:56:37.2394436Z Prepare release directory.
2020-03-06T00:56:37.2407267Z ReleaseId=7315, TeamProjectId=40854903-a7a9-4867-95a8-d5cd31de25d0, ReleaseDefinitionName=XKCD.Configuration
2020-03-06T00:56:37.2694988Z Release folder: C:\xfer\vsts-agent-win-x64-2.164.8\_work\r1\a
2020-03-06T00:56:37.2927325Z Environment variables available are below.  Note that these environment variables can be referred to in the task (in the ReleaseDefinition) by replacing "_" with "." e.g. AGENT_NAME environment variable can be referenced using Agent.Name in the ReleaseDefinition: 
                [AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN] --> [true]
                [AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN] --> [true]
                [AGENT_HOMEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8]
                [AGENT_ID] --> [226]
                [AGENT_JOBNAME] --> [Agent job]
                [AGENT_MACHINENAME] --> [ORG7IT11919]
                [AGENT_NAME] --> [ORG7IT11919 - 2.164.8]
                [AGENT_OS] --> [Windows_NT]
                [AGENT_OSARCHITECTURE] --> [X64]
                [AGENT_RELEASEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work\r1\a]
                [AGENT_RETAINDEFAULTENCODING] --> [false]
                [AGENT_ROOTDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work]
                [AGENT_SERVEROMDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\externals\vstsom]
                [AGENT_TEMPDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work\_temp]
                [AGENT_TOOLSDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work\_tool]
                [AGENT_VERSION] --> [2.164.8]
                [AGENT_WORKFOLDER] --> [C:\xfer\vsts-agent-win-x64-2.164.8\_work]
                [AUTH_AD_GROUP_DEVELOPERS] --> [ORG7\G-S-ORG7 XKCD Information Technology User]
                [AUTH_AD_GROUP_TESTERS] --> [ORG7\G-S-ORG7 XKCD Testing User]
                [AUTH_AM_ACTIVEDIRECTORYTENANTID] --> [https://login.microsoftonline.com/org7.onmicrosoft.com]
                [AUTH_AM_AD_ROOTURL] --> [https://login.windows.net]
                [AUTH_AM_ADFS_ROOTURL] --> [https://xkcd.sts.org7.gov]
                [AUTH_AM_ADTENANT] --> [https://login.windows.net/AGUID]
                [AUTH_AM_ADTENANTID_ADFS] --> [https://xkcd.sts.org7.gov/adfs]
                [AUTH_AM_ADTENANTID_AZUREAD] --> [https://login.windows.net/AGUID]
                [AUTH_AM_DATASYNC_USERNAME] --> [ORG7\svcAMDataSyncTORG]
                [AUTH_AM_TENANTID] --> [dadb1cce-67d9-457a-940f-b1a34e248fd6]
                [AUTH_BATCH_AMCREDENTIALSOURCE] --> [AzureActiveDirectory]
                [AUTH_BATCH_AOLCLIENTAPPID] --> [e0d70f83-c116-4bcb-ada1-7b0714928224]
                [AUTH_BATCH_DORAPI_USERNAME] --> [ORGtestaccount]
                [AUTH_BENEFITACCOUNT_HOST_ENCRYPTION_IDENTIFIER] --> [10-29-2019-17-40-16]
                [AUTH_BENEFITS_QUEUESERVICE_USERNAME] --> [ORG7\svcXKCDQSvcTORG]
                [AUTH_AOL_ADMIN_USERNAME] --> [ORG7\svcAOLOrgAdmTORG]
                [AUTH_AOL_AM_CLIENTAPPID] --> []
                [AUTH_AOL_AM_CLIENTSECRET] --> []
                [AUTH_AOL_AMTENANTID] --> [https://login.windows.net/c3dd5aab-8028-4b27-8234-5ef09229feca]
                [AUTH_AOL_DATASYNC_USERNAME] --> [ORG7\svcAOLDataSyncTORG]
                [AUTH_AOL_TESTING_ADMIN_USERNAME] --> [ORG7\svcXKCDDynTest0ORG]
                [AUTH_AOL_TESTING_INTEGRATIONS_USERNAME] --> [ORG7\svcXKCDDynTest6ORG]

Example not working/change in behavior:

2020-03-06T01:09:02.0484645Z ##[section]Starting: Initialize job
2020-03-06T01:09:02.0486050Z Agent name: 'ORG7IT11919 - 2.165.1'
2020-03-06T01:09:02.0486738Z Agent machine name: 'ORG7IT11919'
2020-03-06T01:09:02.0487199Z Current agent version: '2.165.1'
2020-03-06T01:09:02.0499850Z Agent running as: 'ORG7IT11919$'
2020-03-06T01:09:02.0509420Z Prepare release directory.
2020-03-06T01:09:02.0515576Z ReleaseId=7315, TeamProjectId=40854903-a7a9-4867-95a8-d5cd31de25d0, ReleaseDefinitionName=XKCD.Configuration
2020-03-06T01:09:02.0711345Z Release folder: C:\xfer\vsts-agent-win-x64-2.165.1\_work\r1\a
2020-03-06T01:09:02.1000680Z Environment variables available are below.  Note that these environment variables can be referred to in the task (in the ReleaseDefinition) by replacing "_" with "." e.g. AGENT_NAME environment variable can be referenced using Agent.Name in the ReleaseDefinition: 
                [AGENT_DISABLELOGPLUGIN_TESTFILEPUBLISHERPLUGIN] --> [true]
                [AGENT_DISABLELOGPLUGIN_TESTRESULTLOGPLUGIN] --> [true]
                [AGENT_HOMEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1]
                [AGENT_ID] --> [227]
                [AGENT_JOBNAME] --> [Agent job]
                [AGENT_MACHINENAME] --> [ORG7IT11919]
                [AGENT_NAME] --> [ORG7IT11919 - 2.165.1]
                [AGENT_OS] --> [Windows_NT]
                [AGENT_OSARCHITECTURE] --> [X64]
                [AGENT_RELEASEDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work\r1\a]
                [AGENT_RETAINDEFAULTENCODING] --> [false]
                [AGENT_ROOTDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work]
                [AGENT_SERVEROMDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\externals\vstsom]
                [AGENT_TEMPDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work\_temp]
                [AGENT_TOOLSDIRECTORY] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work\_tool]
                [AGENT_VERSION] --> [2.165.1]
                [AGENT_WORKFOLDER] --> [C:\xfer\vsts-agent-win-x64-2.165.1\_work]
                [AUTH_AD_GROUP_DEVELOPERS] --> [ORG7\G-S-ORG7 XKCD Information Technology User]
                [AUTH_AD_GROUP_TESTERS] --> [ORG7\G-S-ORG7 XKCD Testing User]
                [AUTH_AM_ACTIVEDIRECTORYTENANTID] --> [https://login.microsoftonline.com/org7.onmicrosoft.com]
                [AUTH_AM_AD_ROOTURL] --> [https://login.windows.net]
                [AUTH_AM_ADFS_ROOTURL] --> [https:***@ORG.xkcd.gov]
                [AOL_ILINX_URL] --> [https://testImaging/?mode=internalopen&appname=XKCD&docid=]
                [AOL_ORGANIZATION] --> [test2]
                [AOL_ORGANIZATION_ADMIN_ID] --> [ORG7\svcAOLOrgAdmTORG]
                [AOL_PORTAL_BASE_DEEPLINK] --> [https://test2.ORG7.ca.lcl/Staff/Routing?]

Agent and Worker's Diagnostic Logs

Local agent logs available on request, likely will have to sanitize them before posting publicly.

This looks likely related to issue #2839 . Possibly due to .NET core 3.1?

damccorm commented 4 years ago

Hey @OranguTech - would definitely be helpful if you're able to post logs. From your description, it looks like the issue you're seeing is that variables aren't getting propogated through - how are you setting those variables?

Would also be helpful to have your organization name if possible so that I can determine what bits you were getting from the service. If you're not comfortable sharing any of this publicly, feel free to email me at damccorm@microsoft.com

OranguTech commented 4 years ago

I replied in email earlier with more details and logs. (Seems to be only an issue in the “initialize job” section of the “Agent job” task, it is not outputting all the variables as it did before, but the values are getting evaluated in the pipeline correctly.)

Further testing and better looking at logs, I think it's worthwhile to look at PR #2791 "mask secrets in URLs" .

Example previously [ENV_ADFS_OAUTHTOKENSERVERURL] --> [https://abc.xyz.xkcd.gov/adfs/oauth2/token]

is now obfuscated to [ENV_ADFS_OAUTHTOKENSERVERURL] --> [https:***@xkcd.gov], and then a whole giant chunk of vars aren't enumerated.

Similary: [ENV_CRM_SOAP_URL] --> [https://test2.dumbcrm-test.org7.xkcd.lcl/XRMServices/2011/Organization.svc] --> [ENV_CRM_SOAP_URL] --> [https:***@org7.xkcd.gov] then a whole new chunk missing.

It does this 3 times in our set of variables. I don't know if the issue is the regex or what it does once the regex is tripped.

damccorm commented 4 years ago

That definitely looks like the issue - I think the regex is overly aggressive here. For example, if you have:

https://abc.def
https://ghi.jkl
https://mno.pqr
https://password@example.com

the following gets masked by our regex ((?<=//[^:/?#]+:)[^@]+(?=@))

//ghi.jkl
https://mno.pqr
https://password

I think the regex should be (?<=//[^:/?#\n]+:)[^@]+(?=@) (note the added \n which stops looking for a match if we reach the EOL)

I'll add a PR

OranguTech commented 4 years ago

Thanks for the quick response.

If I wanted to try compiling my own copy, would compiling from Master be likely to work, or should I just make your change on current release branch?

damccorm commented 4 years ago

Either should work.

OranguTech commented 4 years ago

For completeness, linking PR #2844

damccorm commented 4 years ago

@OranguTech Thanks! I should've linked them.

Were you able to compile an agent and verify this fixes the issue?

OranguTech commented 4 years ago

I have not - opening the solution file immediately have errors about nuget package versions that a simple restore or update doesn't fix. I'm pretty new to the VS world, and it's not a blocking issue, so left it alone.

On Tue, Mar 10, 2020 at 7:19 PM Danny McCormick notifications@github.com wrote:

@OranguTech https://github.com/OranguTech Thanks! I should've linked them.

Were you able to compile an agent and verify this fixes the issue?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/microsoft/azure-pipelines-agent/issues/2842?email_source=notifications&email_token=AAM44DNS4D3SCK5YRER3WYDRG3YJVA5CNFSM4LCWTLYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEON36AY#issuecomment-597409539, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM44DPX6DBC4OWSHYTDOLDRG3YJVANCNFSM4LCWTLYA .

OranguTech commented 4 years ago

(I'm assuming the fix will be in the next release)

OranguTech commented 4 years ago

@damccorm - Is there a general release schedule? I know this got merged into Master, I assume that will be branched into 2.165.3, but not sure on your dev/branch process. ( I'm just trying to get an ETA on this.)

damccorm commented 4 years ago

@OranguTech normally we deploy at least once every 3 weeks. Right now is an exception though, we are currently in a deployment freeze due to Cornavirus, so it may be a little bit before this fix gets out.

OranguTech commented 4 years ago

Newer agents finally rolled out to our instance, verified that as of 2.166.4, issue appears fixed. Thank you @damccorm !