search

dotnet / msbuild

The Microsoft Build Engine (MSBuild) is the build platform for .NET and Visual Studio.
https://docs.microsoft.com/visualstudio/msbuild/msbuild
MIT License
5.21k stars 1.35k forks source link

Running build.cmd on the Developer Command Prompt for VS 2017 fails #3937

Closed Sathyaish closed 4 years ago

Sathyaish commented 5 years ago

What I am trying to do

I just want to get the source code for MS Build on my machine so I can play with it and step through the code.

What I have

I had .NET Core version 2.1.500 on my machine. I also have Visual Studio 2017 Community.

Steps to reproduce

As per the instructions here:

  1. I cloned the MS Build repository from here.
  2. Opened the Developer Command Prompt for VS 2017.
  3. Browsed to the root of the git repository on my local machine.
  4. Typed build.cmd.

Expected behavior

I expected the build.cmd file to run successfully.

Actual behavior

I got the following error:

C:\Sathyaish\repos\public\msbuild>build.cmd
dotnet-install: Downloading link: https://dotnetcli.azureedge.net/dotnet/Sdk/2.1
.401/dotnet-sdk-2.1.401-win-x64.zip
dotnet-install: Cannot download: https://dotnetcli.azureedge.net/dotnet/Sdk/2.1.
401/dotnet-sdk-2.1.401-win-x64.zip
dotnet-install: Downloading legacy link: https://dotnetcli.azureedge.net/dotnet/
Sdk/2.1.401/dotnet-dev-win-x64.2.1.401.zip
Exception calling "Invoke" with "0" argument(s): "Failed to download https://dot
netcli.azureedge.net/dotnet/Sdk/2.1.401/dotnet-dev-win-x64.2.1.401.zip.  StatusC
ode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http.Stre
amContent, Headers:
{
  x-ms-request-id: 104c7d7f-f01e-00b5-284c-7ea534000000
  x-ms-version: 2009-09-19
  Date: Sat, 17 Nov 2018 08:09:36 GMT
  Server: Windows-Azure-Blob/1.0
  Server: Microsoft-HTTPAPI/2.0
  Content-Length: 215
  Content-Type: application/xml
}"
System.Management.Automation.MethodInvocationException: Exception calling "Invok
e" with "0" argument(s): "Failed to download https://dotnetcli.azureedge.net/dot
net/Sdk/2.1.401/dotnet-dev-win-x64.2.1.401.zip.  StatusCode: 404, ReasonPhrase:
'Not Found', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
  x-ms-request-id: 104c7d7f-f01e-00b5-284c-7ea534000000
  x-ms-version: 2009-09-19
  Date: Sat, 17 Nov 2018 08:09:36 GMT
  Server: Windows-Azure-Blob/1.0
  Server: Microsoft-HTTPAPI/2.0
  Content-Length: 215
  Content-Type: application/xml
}" ---> System.Management.Automation.RuntimeException: Failed to download https:
//dotnetcli.azureedge.net/dotnet/Sdk/2.1.401/dotnet-dev-win-x64.2.1.401.zip.  St
atusCode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http
.StreamContent, Headers:
{
  x-ms-request-id: 104c7d7f-f01e-00b5-284c-7ea534000000
  x-ms-version: 2009-09-19
  Date: Sat, 17 Nov 2018 08:09:36 GMT
  Server: Windows-Azure-Blob/1.0
  Server: Microsoft-HTTPAPI/2.0
  Content-Length: 215
  Content-Type: application/xml
} ---> System.Management.Automation.RuntimeException: Failed to download https:/
/dotnetcli.azureedge.net/dotnet/Sdk/2.1.401/dotnet-dev-win-x64.2.1.401.zip.  Sta
tusCode: 404, ReasonPhrase: 'Not Found', Version: 1.1, Content: System.Net.Http.
StreamContent, Headers:
{
  x-ms-request-id: 104c7d7f-f01e-00b5-284c-7ea534000000
  x-ms-version: 2009-09-19
  Date: Sat, 17 Nov 2018 08:09:36 GMT
  Server: Windows-Azure-Blob/1.0
  Server: Microsoft-HTTPAPI/2.0
  Content-Length: 215
  Content-Type: application/xml
}
   --- End of inner exception stack trace ---
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(Fu
nctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(Inter
pretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.R
un(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.R
un(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame
frame)
   at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0)

   at System.Management.Automation.ScriptBlock.InvokeWithPipeImpl(ScriptBlockCla
useToInvoke clauseToInvoke, Boolean createLocalScope, Dictionary`2 functionsToDe
fine, List`1 variablesToDefine, ErrorHandlingBehavior errorHandlingBehavior, Obj
ect dollarUnder, Object input, Object scriptThis, Pipe outputPipe, InvocationInf
o invocationInfo, Object[] args)
   at System.Management.Automation.ScriptBlock.<>c__DisplayClassa.<InvokeWithPip
e>b__8()
   at System.Management.Automation.Runspaces.RunspaceBase.RunActionIfNoRunningPi
pelinesWithThreadCheck(Action action)
   at System.Management.Automation.ScriptBlock.InvokeWithPipe(Boolean useLocalSc
ope, ErrorHandlingBehavior errorHandlingBehavior, Object dollarUnder, Object inp
ut, Object scriptThis, Pipe outputPipe, InvocationInfo invocationInfo, Boolean p
ropagateAllExceptionsToTop, List`1 variablesToDefine, Dictionary`2 functionsToDe
fine, Object[] args)
   at System.Management.Automation.ScriptBlock.DoInvoke(Object dollarUnder, Obje
ct input, Object[] args)
   at CallSite.Target(Closure , CallSite , ScriptBlock )
   --- End of inner exception stack trace ---
   at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(Fu
nctionContext funcContext, Exception exception)
   at System.Management.Automation.Interpreter.ActionCallInstruction`2.Run(Inter
pretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.R
un(InterpretedFrame frame)
   at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.R
un(InterpretedFrame frame)
at Invoke-With-Retry, C:\Sathyaish\repos\public\msbuild\artifacts\.dotnet\2.1.40
1\dotnet-install.ps1: line 137
at GetHTTPResponse, C:\Sathyaish\repos\public\msbuild\artifacts\.dotnet\2.1.401\
dotnet-install.ps1: line 193
at DownloadFile, C:\Sathyaish\repos\public\msbuild\artifacts\.dotnet\2.1.401\dot
net-install.ps1: line 480
at <ScriptBlock>, C:\Sathyaish\repos\public\msbuild\artifacts\.dotnet\2.1.401\do
tnet-install.ps1: line 568
at <ScriptBlock>, <No file>: line 1
at InstallDotNetCli, C:\Sathyaish\repos\public\msbuild\build\build.ps1: line 98
at Build, C:\Sathyaish\repos\public\msbuild\build\build.ps1: line 197
at <ScriptBlock>, C:\Sathyaish\repos\public\msbuild\build\build.ps1: line 419
at <ScriptBlock>, <No file>: line 1

Environment data

OS info: Windows 7 Home Premium 64-bit

michal-pawlowski commented 5 years ago

MSBuild is currently using DotNet Core SDK 2.1.401 which was not found on your machine. Currently used version is defined in .\build\Versions.props. When SDK is not found, the script attepmts to download and install it automatically, however in your case the download failed. Any idea why? On my machine the script downloads SDK correctly. The link from log is correct.

Please, install DotNet Core SDK 2.1.401 and rebuild the project.

rainersigwald commented 5 years ago

Does this happen consistently? Can you open https://dotnetcli.azureedge.net/dot net/Sdk/2.1.401/dotnet-dev-win-x64.2.1.401.zip in a browser? I don't think we've seen this step fail very often.

Sathyaish commented 5 years ago

Thank you, @michal-pawlowski and @rainersigwald .

I was able to download the zip file from https://dotnetcli.azureedge.net/dotnet/Sdk/2.1.401/dotnet-sdk-2.1.401-win-x64.zip and unzip it, but now I have a new problem. Its contents look like it is actually a non-installable X-copyable (like old win32 files) version of .NET Core. How do I merge it with the other versions of .NET Core that exist on my machine?

My problem is detailed on this Stack Overflow question.

Sathyaish commented 5 years ago

Currently used version is defined in .\build\Versions.props.

Thanks. Missed that. Edited Version.props to need the latest version 2.1.500 and trying it. Will report.

Sathyaish commented 5 years ago

Ok, this is strange. I edited Version.props to make it need version 2.1.500. And I do have that version. But running build.cmd attempts to now try and download version 2.1.500 of .NET Core.

It's not yet done so I'll know in some time what's going on, but in the meantime, I looked up build.ps1 (called by build.cmd on Windows), and here's probably the culprit code.

[CmdletBinding(PositionalBinding=$false)]
Param(
  ...

  # This is set to an empty string
  [string] $DotNetCoreSdkDir = "",
)

function Build {
  if (![string]::IsNullOrEmpty($DotNetCoreSdkDir) -and (Test-Path -Path $DotNetCoreSdkDir)) {
    $env:DOTNET_INSTALL_DIR = $DotNetCoreSdkDir
  }
  else {
   # The flow-of-control probably enters here because
   # the variable $(DotNetCoreSdkDir) is set to an empty
   # string in the parameter declaration section of this
   # script
    InstallDotNetCli
  }

  $env:DOTNET_HOST_PATH = Join-Path $env:DOTNET_INSTALL_DIR "dotnet.exe"

  ...
}
livarcocc commented 5 years ago

I believe this is by design. We write our build scripts to be completely independent. They will download all the pieces that the build needs to succeed and also binplace them within the repo itself, so that afterwards, something like a git clean can restore the machine to a clean state as far as the repo is concerned.

Do you know why the install script that build.cmd attempts to run is failing to download the SDK that it needs to build msbuild with? I just attempted straight up running build.cmd and everything worked for me.

Sathyaish commented 5 years ago

@livarcocc Thanks. I did set out to find out the reason and then got busy doing something else. I have it on my list and will get to it when I can (coming few days) and post here.

Sathyaish commented 5 years ago

Aaargh! It took me all of today to figure out what was wrong.

After a lot of putting in Set-Verbose or Write-Host "" -ForegroundColor Red statements, and stepping through the $(ProjectRoot)\artifacts\.dotnet\$(DotNetCliVersion)\dotnet-install.ps1, scanning through my event logs, reading a ton of documentation, and after many a wild goose chases down the wrong alley, many false leads, here is what was wrong:

  1. Fiddler is set up as a system wide proxy for me at port 8888 on the localhost. This was my default system proxy and I wasn't running the Fiddler process, so the attempts to download the first downloadable file, namely, https://dotnetcli.azureedge.net/dotnet/Sdk/2.1 .401/dotnet-sdk-2.1.401-win-x64.zip failed.

    I tried removing the system-wide proxy 127.0.0.1:8888 from the Win Inet options / Control Panel / Internet Options but I don't know why, Fiddler would re-instate itself as the proxy. I changed Fiddler -> Tools -> Telerik Fiddler Options -> Connections and unchecked Act as a system proxy on start-up and Monitor all connections but that didn't help either, so I just let Fiddler run while I ran the $(ProjectRoot)\build.cmd batch file.

    However, that was only half the problem.

  2. If the script fails to download the file https://dotnetcli.azureedge.net/dotnet/Sdk/2.1 .401/dotnet-sdk-2.1.401-win-x64.zip, it attempts to download a legacy file from the URL https://dotnetcli.azureedge.net/dotnet/ Sdk/2.1.401/dotnet-dev-win-x64.2.1.401.zip. It turns out this URL is invalid and hence any Web request to it yields a 404. So, this needs to be updated.

  3. Even though I have a 2 Mbps connection, it appears that the file https://dotnetcli.azureedge.net/dotnet/Sdk/2.1 .401/dotnet-sdk-2.1.401-win-x64.zip would take only slightly more than 10 minutes to download on my machine. However, the dotnet-install.ps1 script file had a time-out set to 10 minutes for every download. This is declared in the script block that's passed to the call to the Invoke-With-Retry method from within the GetHTTPResponse method.

    $HttpClient.Timeout = New-TimeSpan -Minutes 10

    I changed it to 30 minutes and it downloaded the file.

    Therefore, I advise that this limit be increased keeping in mind people like me. My connection is still far better than some of the connections in rural and sub-urban regions in South East Asia, Middle East and Africa.

    I understand this is within the purview of the .NET Core CLI team and so I have posted an issue about this on their repo.

  4. This is about the 20th time or so I am running the build process today. It appears to be going well till now except there's another timeout of 10000ms someplace, this time in one of the .csproj files and I can't afford to hunt it down anymore. It was trying to download nuget or something. I'll have to check the logs once this build process runs fully. But I suggest that all time-outs be increased to 30 minutes just to be safe.

False chase:

I had many a false chases in tracking this but one popular one was that the Add-Type commandlet was producing an Information entry in the event logs, so I thought something must be wrong there. Spent about 2 hours, retrying it, reading its documentation and putting in Set-Verbose around the code. I noticed that it read something like (from the Load-Assembly function in dotnet-install.ps1):

Add-Type -Assembly Assembly | Out-Null

I was certain this was failing because there is no switch named -Assembly for this commandlet. There was, however, an -AssemblyName switch. I suspected that since the commandlet usage above was trying to download all types from the System.Net.Http assembly, it was failing and hence all download calls (Invoke-WebRequest / HttpClient.GetAsync) could be failing because of this.

Then, after many hours of debugging, I remembered that Powershell allows you to type partial switch names so long as what you've typed can uniquely identify the switch. So, this was a false lead.

Finally, a request:

I am looking for work. If you know someone hiring remotely, please do let me know.

I am a programmer working in .NET. I've been programming since 1997.

My family situation presently only allows me to work from home remotely. I live in Noida, India.

wassimans commented 5 years ago

Hi all,

I have exactly the same issue as @Sathyaish! Seven attempts and counting without any success. I have no proxies. My Internet connection is quite good and stable. I'm getting the same errors after the SDK download attempt.