Error downloading artifacts when behind a proxy

markallisongit commented 5 years ago

Environment

Server - Azure Pipelines
- If using Azure Pipelines, provide the account name, team project name, build definition name/build number: spektrix, spektrix, MigrateDB, MigrateDB-demo1-20181217.5
Agent - Hosted or Private:
- If using private agent, provide the OS of the machine running the agent and the agent version: Windows Server 2012 R2 Agent version vsts-agent-win-x64-2.142.1

Issue Description

When running a release, it fails at step 2 Download artifact with Failed in getBuild with error: Error: tunneling socket could not be established, statusCode=504

The proxy server we use does not require authentication. Agent was installed with the --proxy-url parameter. I tried setting the environment variables mentioned in #8909 but that didn't help. Release.zip

Task logs

Attached

Error logs

2018-12-17T14:02:23.6325042Z ##[debug]buildId=737 2018-12-17T14:02:24.6380306Z Error: in getBuild, so retrying => retries pending : 4 2018-12-17T14:02:27.8356169Z Error: in getBuild, so retrying => retries pending : 3 2018-12-17T14:02:31.8530052Z Error: in getBuild, so retrying => retries pending : 2 2018-12-17T14:02:35.8598831Z Error: in getBuild, so retrying => retries pending : 1 2018-12-17T14:02:39.8823314Z ##[error]Failed in getBuild with error: Error: tunneling socket could not be established, statusCode=504 2018-12-17T14:02:39.8831386Z ##[debug]Processed: ##vso[task.issue type=error;]Failed in getBuild with error: Error: tunneling socket could not be established, statusCode=504

omeshp commented 5 years ago

@markallisongit Which proxy server are you using? Is it expecting windows authentication? Does it support CONNECT requests?

markallisongit commented 5 years ago

we are using a feature of the Citrix Netscaler VPX as a simple forward proxy. No Windows Auth. https://www.citrix.com/blogs/2010/02/25/netscaler-feature-of-the-day-deploy-as-a-forward-proxy/

omeshp commented 5 years ago

@markallisongit Can you run the below command from agent box and provide the output: curl -x <proxyurl> https://www.google.co.in -v

<proxyurl> should be in the form like http://172.26.5.164:3128

ricohomewood commented 5 years ago

@omeshp That seems to work fine:

Proxy IP masked as x.x.x.x

C:\Program Files\Curl\bin>curl -x http://x.x.x.x:80 http://www.google.co.in -v
*   Trying x.x.x.x...
* TCP_NODELAY set
* Connected to x.x.x.x (x.x.x.x) port 80 (#0)
> GET http://www.google.co.in/ HTTP/1.1
> Host: www.google.co.in
> User-Agent: curl/7.63.0
> Accept: */*
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 200 OK
< Date: Fri, 04 Jan 2019 09:26:01 GMT
< Expires: -1
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< P3P: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Server: gws
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Set-Cookie: 1P_JAR=2019-01-04-09; expires=Sun, 03-Feb-2019 09:26:01 GMT; path=
/; domain=.google.co.in
< Set-Cookie: NID=154=gBbk0d_PiLv2DnmXUDYNWQw0J-YxRpAPsBSW0AnyG38HpA2XhxRiarimXe
EgWIfcxPQjBQWWvEAdWpNOvt_C98WOXEL42Sxuytr-t1SRUW8AmolctI0xMioHlFnDc8ekqfc-2x_EC7
M5MJWyPMh6u4ZSBSoDVYde0ORM0zm-Vfo; expires=Sat, 06-Jul-2019 09:26:01 GMT; path=/
; domain=.google.co.in; HttpOnly
< Accept-Ranges: none
< Vary: Accept-Encoding
< Transfer-Encoding: chunked

omeshp commented 5 years ago

@ricohomewood I assume https calls are also working fine with proxy? Is it possible for you to get on a call to debug this issue?

ricohomewood commented 5 years ago

@omeshp I can confirm that curl also works for https calls. I am free to debug this on Monday next week if that suits you?

omeshp commented 5 years ago

@ricohomewood Actually i will be on leave till Wednesday next week. Can we schedule it on Thursday or Friday. Also, if this is blocking your daily deployments we can temporary disable the new download flow for your account which should mostly not face this issue. Let me know if you want me to do till this is resolved.

ricohomewood commented 5 years ago

Hello @omeshp Sorry for the late reply. I am free from Thursday this week if you want to troubleshoot?

omeshp commented 5 years ago

Thanks @ricohomewood I will schedule sometime for tomorrow or day after. which timezone are you in?

ricohomewood commented 5 years ago

Thanks @omeshp I'm currently in UK/London so I can do 08:30 - 16:30 UK time if that suits tomorrow or day after?

omeshp commented 5 years ago

@ricohomewood Can you send a mail to RM_Customer_Queries@microsoft.com so that we can send you a meeting invite?

omeshp commented 5 years ago

Closing the issue since this has been resolved for the customer now. @ricohomewood Do let us know if you need additional help.

nikmikov commented 5 years ago

@omeshp We are facing exactly same issue - a bit more info on how it was resolved would be very helpful :)

omeshp commented 5 years ago

@nikmikov We had shifted the account to use agent based download flow for the customer at our end. If you share the failure details you are facing we can do the same for you.

nikmikov commented 5 years ago

@omeshp Exactly same error: self-hosted agent, behind proxy. Proxy doesn't require authentication, agent set up with "--proxy-url". Everything works except artifact download.

Failed in getBuild with error: Error: tunneling socket could not be established, statusCode=504
Error: tunneling socket could not be established, statusCode=504

Server - Azure Pipelines Account: accapplications Project: DataEngines, Release pipeline: Release-3 Agent - Self-hosted: OS: RHEL Agent version vsts-agent-win-x64-2.146

omeshp commented 5 years ago

@nikmikov We have shifted your account to use agent based download flow, let us know if the issue is resolved or not.

nikmikov commented 5 years ago

@omeshp thanks a lot, it's all good now!

dmeyer573 commented 5 years ago

We are having a very similar issue: self-hosted agent, behind proxy. Proxy doesn't require authentication, agent set up with "--proxy-url". Everything works except artifact download.

Log Snippet Showing Issue: 2019-05-29T16:07:54.3590229Z Download artifact to: E:\BuildReleaseAgent_work\r1\a/_commerce-sync.general-scheduler/Choco_Ipc.Scheduler.Service 2019-05-29T16:07:56.3867711Z Information, Minimatch patterns: [] 2019-05-29T16:08:18.7226387Z Warning, [https://masked.blob.core.windows.net/masked/masked?masked**] Try 1/5, retryable exception caught. Retrying in 00:00:01. Details: 2019-05-29T16:08:18.7227590Z Exception.Data didn't contain key LastRequestResponse or its value was null or not a string. Was HttpRequestExceptionExtensions.SetHttpMessagesForTracing called? System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.Http.WinHttpException: Error 12002 calling WINHTTP_CALLBACK_STATUS_REQUEST_ERROR, 'The operation timed out'. 2019-05-29T16:08:18.7228106Z at System.Threading.Tasks.RendezvousAwaitable1.GetResult() 2019-05-29T16:08:18.7228561Z at System.Net.Http.WinHttpHandler.StartRequest(WinHttpRequestState state) 2019-05-29T16:08:18.7229664Z --- End of inner exception stack trace --- 2019-05-29T16:08:18.7229866Z at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) 2019-05-29T16:08:18.7230401Z at Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task1 task, CancellationToken cancellationToken, Func1 makeMessage, String file, String member, Int32 line) 2019-05-29T16:08:18.7231102Z at Microsoft.VisualStudio.Services.BlobStore.WebApi.DedupStoreHttpClient.<>c__DisplayClass56_0.<b__0>d.MoveNext() 2019-05-29T16:08:18.7231270Z --- End of stack trace from previous location where exception was thrown --- 2019-05-29T16:08:18.7231391Z at Microsoft.VisualStudio.Services.Content.Common.AsyncHttpRetryHelper`1.InvokeAsync(CancellationToken cancellationToken)

I have masked some aspects of the download URL being used. This error occurs on all 5 attempts. We have confirmed that the agent is not sending this request through our proxy server.

Server - Azure Pipelines Account/Organization: CommerceSync Project: general-scheduler Agent: Self Hosted on Windows Server 2016 Agent Version: Not sure where to find this information.

Lovakumar commented 5 years ago

@dmeyer573 this is a known w.r.t DownloadArtifact being used behind proxy. We can turn off a feature flag to rollback your releases to use agent-plugin based download (instead of Download Build Artifact task based download).

Solution 1: Please note that turning off feature flag will impact selective artifacts download feature for all of your release definitions of your account. If you are ok with it, we can turn off feature flag.

Solution 2: If you are using proxy based agents for fewer release definitions then set the following variable on your release definitions to by pass this issue. "release.artifact.download.useagentplugin=true"

dmeyer573 commented 5 years ago

@Lovakumar - Thank you for the response. We elected to go the route of building an agent machine that is not required to communicate through the proxy. This approach is working fine for us at this time, no further action requested. Thank you!

microsoft / azure-pipelines-tasks