restore.cmd fails under a proxy environment

[SIkebe]

To Reproduce

Steps to reproduce the behavior:

1. Execute restore.cmd under a proxy environment.
2. Fail to download install.ps1 here. https://github.com/aspnet/AspNetCore/blob/28678ee430e9f9a8407e846441ab98692b9f9507/eng/common/tools.ps1#L162

Logs

C:\github\AspNetCore>build.cmd
WARNING: No default group of projects was specified, so building the 'managed' subsets of projects. Run `build.cmd
-help` for more details.
Detected JDK in C:\Program Files\AdoptOpenJDK\jdk-11.0.3.7-hotspot (via JAVA_HOME)
at GetDotNetInstallScript, C:\github\AspNetCore\eng\common\tools.ps1: line 162
at InstallDotNet, C:\github\AspNetCore\eng\common\tools.ps1: line 173
at InstallDotNetSdk, C:\github\AspNetCore\eng\common\tools.ps1: line 169
at InitializeDotNetCli, C:\github\AspNetCore\eng\common\tools.ps1: line 135
at InitializeBuildTool, C:\github\AspNetCore\eng\common\tools.ps1: line 354
at InitializeToolset, C:\github\AspNetCore\eng\common\tools.ps1: line 448
at <ScriptBlock>, C:\github\AspNetCore\build.ps1: line 342
at <ScriptBlock>, <No file>: line 1
authentication_failed

Are there any instructions as to build from source under a proxy environment? I cannot find one in here.

[dougbu]

@SIkebe I transferred this because I'm not familiar w/ using eng/common/tools.ps1 when a proxy is necessary. This repo is the owner of that file.

[alexperovich]

What do you mean by "under a proxy environment"? There are multiple ways to do that.

[SIkebe]

Authentication requirement against corporate proxy to access the Internet.

[alexperovich]

How did you set up the proxy settings? You have to tell powershell about them somehow. I think it might read the settings configured in windows. As of powershell 7.0 it will also read proxy settings from environment variables

[chcosta]

@SIkebe , how did you set up the proxy settings?

[chcosta]

Are there any instructions as to build from source under a proxy environment?

@dagood, do you know who could help out here? Is there anything particular about build from source that would contribute?

@slkebe, do you have access to other sites via your proxy environment?

[dagood]

"Build from source" here is ASP.NET Core's specific meaning, described in https://github.com/dotnet/aspnetcore/blob/master/docs/BuildFromSource.md. This looks to me like it's up to Arcade, PowerShell, and understanding proxies.

(This thread's "build from source" is from a developer/contributor's point of view. dotnet/source-build's "build from source" is from a Linux distro package maintainer's point of view: it has more baggage that doesn't apply here, such as not downloading prebuilt binaries. dotnet/source-build also doesn't currently support Windows. 😛)

[SIkebe]

@chcosta Sorry for late reply.

how did you set up the proxy settings?

Proxy script and http_proxy environmental variable. NuGet, Visual Studio, Visual Studio Code, Omnisharp and almost all other tools are working correctly.

do you have access to other sites via your proxy environment?

Yes, I can.

[SIkebe]

I have to modify GetDotNetInstallScript, LocateVisualStudio and InitializeXCopyMSBuild in tools.ps1 like this.

function GetDotNetInstallScript([string] $dotnetRoot) {
  $installScript = Join-Path $dotnetRoot 'dotnet-install.ps1'
  if (!(Test-Path $installScript)) {
    Create-Directory $dotnetRoot
    $ProgressPreference = 'SilentlyContinue' # Don't display the console progress UI - it's a huge perf hit

    $maxRetries = 5
    $retries = 1

    $uri = "https://dot.net/$dotnetInstallScriptVersion/dotnet-install.ps1"

    while($true) {
      try {
        Write-Host "GET $uri"

        $proxy_url = "proxy-server-address-and-port"
        $proxy_user = "user"
        $proxy_pass = "password"
        $s = ConvertTo-SecureString $proxy_pass -AsPlainText -Force
        $proxy_cred = New-Object System.Management.Automation.PSCredential $proxy_user, $s

        Invoke-WebRequest -Uri $uri -OutFile $installScript -Proxy $proxy_url -ProxyCredential $proxy_cred
        break
      }
      catch {
        Write-Host "Failed to download '$uri'"
        Write-Error $_.Exception.Message -ErrorAction Continue
      }

      if (++$retries -le $maxRetries) {
        $delayInSeconds = [math]::Pow(2, $retries) - 1 # Exponential backoff
        Write-Host "Retrying. Waiting for $delayInSeconds seconds before next attempt ($retries of $maxRetries)."
        Start-Sleep -Seconds $delayInSeconds
      }
      else {
        throw "Unable to download file in $maxRetries attempts."
      }

    }
  }

  return $installScript
}

[chcosta]

@slkebe, I'm not pushing back on your problem, or saying it's not valid; I am still trying to understand your scenario better.

How did you get here? ie, why are you running under a proxy environment in the first place. The docs I can find say that MSFTVPN is the supported method for connecting to MSFT resources, internal docs more specifically implicate that Direct Access and other methods are deprecated. I did find internal docs related to troubleshooting "can't access site" type issues, and perhaps they can help unblock your scenario.

Can you point me to the process you followed to create your proxy environment? I'd like to understand this better so that I can repro your problem and ensure we're addressing the correct fundamental issue.

[chcosta]

+ @JpratherMS who has more experience with proxy's and may have some thoughts here.

[SIkebe]

@chcosta

How did you get here? ie, why are you running under a proxy environment in the first place.

My company requires all dev machines to be behind the corporate proxy, so developers have to enter their credentials to access the Internet. It's very common practice here in Japan, thogh I'm not sure about other contries.

[chcosta]

@JpratherMS , do you know how I could setup a machine to repro this scenario? I don't feel comfortable making a change here without being able to repro the problem. @Slkebe, is this something that others on your team are experiencing?

A couple of alternatives:

• you're welcome to submit a PR. The code you provided above is a good start, we'd probably want to change things like the url and cred values to be defined / used if an env variable is present.

• If you install the pre-req's manually, then it shouldn't try to install them and fail). I recognize that this is not an ideal solution :(

  • dotnet sdk to env:DOTNET_INSTALL_DIR
  • msbuild to env:VSINSTALLDIR

[alexperovich]

If you haven't set either the IE proxy settings, or the https_proxy environment variable, I would never expect things to work out of the box, and I don't think we should change the code to support some other way to set these settings. These are supported ways to configure the machine to use a proxy, we shouldn't create yet another way that only works for our scripts.

[JpratherMS]

@chcosta it's been a long time since we had to use a proxy on main campus (in fact I think the last one that was around is long deprecated) so I'm not 100% sure on how to repro this issue without finding another supported proxy server. I can check around and ask some folks but this may be something very specific to the region he is in. It sound's like there is an issue passing the credentials here and I'm not sure the best way to resolve that. Let me do some digging regarding whether there are any proxy servers still around in Redmond

[chcosta]

@SIkebe , can your company update your proxy?

[chcosta]

@Slkebe, I'm going to close this issue.

In principal, I'm not opposed to a PR to Arcade to support a proxy. As you may have inferred, however, there may be some push back by others. My primary concern, and the reason I think we're unlikely to actually merge the PR, is that I don't think we're set up to support / maintain the functionality. It would be very easy for someone to unintentionally break the proxy authentication support and we wouldn't find out until we impacted you / others. That's a poor story and would likely cause a lot of bad experiences (both for you and for others).

Please feel free to push back if you feel that there is a strong case for further investment here.