search

dotnet / roslyn

The Roslyn .NET compiler provides C# and Visual Basic languages with rich code analysis APIs.
https://docs.microsoft.com/dotnet/csharp/roslyn-sdk/
MIT License
19.04k stars 4.03k forks source link

Build system feedback #39098

Open tamlin-mike opened 5 years ago

tamlin-mike commented 5 years ago

As a first-time builder of any Roslyn code, I thought it could be interesting for you doing this on a daily basis to get some input from someone that tested it out for the first time.

Documentation:

Flexibility/options:

In my case I knew exactly what I needed to get VS stop crashing. I needed to fix one single bug, and build a replacement for one single file, Microsoft.VisualStudio.LanguageServices.dll. Instead of being able to build just that DLL (from what I have found at least), I'm now continually fighting with disk-space running out, running compact /c /f /s /exe:lzx artifacts\* like a madman from a separate shell, while periodically Suspending all the build processes from the Resource Monitor, trying to keep up with the torrent of other stuff it's filling up the volume with - stuff I really don't want produced. Yeah, it's partially a problem with me not allocating enough disk space, but I couldn't in my wildest imagination think I'd need over ten gigabyte to build a single DLL that's just a hair over 1.5 MB. That points back to the need for some documented info of how much disk space should be expected to be consumed.

Besides these issues, it seems the build system works pretty well. It could use some work to beat it into better shape, but overall it seems quite straightforward.

Thumbs up.

My 0.02.

huoyaoyuan commented 5 years ago

I'm missing an option to disable creation of any and all tests.

Ability to build a single DLL.

You can run Restore.cmd first so that VS can load the projects, then invoke Build inside VS only on your interests. I'm not sure whether this works for a clean clone, but I've only runned Restore.cmd but not Build.cmd in several months.

RikkiGibson commented 5 years ago

tagging @jaredpar and @jasonmalinowski

tamlin-mike commented 5 years ago

@huoyaoyuan Yes, I have now realized it is possible to build a single project from within VS, but loading Roslyn in VS is... not exactly speedy, nice to the CPU fan, or the responsiveness of the computer as a whole. :-)

I had hoped it would be possible to specify perhaps .csproj-level from command-line. Maybe it is, and I simply haven't found out how to do it yet.

jasonmalinowski commented 5 years ago

You can just do msbuild /m <project> like any other .NET project and that'll build just that and dependencies.

jaredpar commented 5 years ago

Please, consider using filesystem links for immutable files, where available.

This can be done by setting the environment variable %ROSLYNUSEHARDLINKS% to any value. That will enable all of the MSBuild options for using hard links when possible for files. This is not enabled by default for the following reasons:

  1. It doesn't have a meaningful impact on build performance so long as you're building on an SSD.
  2. It has no impact when the nuget cache directory and the build output directory are on different drives (which is fairly common in our infrastructure).
  3. Hard links in general can conflict with the Visual Studio experience due to circumstances out of our control.

At the same time though the option is there and available for anyone to use locally.

I'm missing an option to disable creation of any and all tests.

Can you be more specific here? Are you asking for the ability to avoid building a specific set of tests? If so why do you believe that would be valuable? Excluding certain tests from build wouldn't have a meaningful impact on build time. If you're interested in building only a specific exe then you can just build the projcet directly today and the tests won't come along with it.

I'm missing an option to disable creation of all the locale directories. To put it mildly, I am more than a little annoyed of all the thousands of locale directories VS itself spews all over the installation,

Can you be more specific here? VS install creates zero directories here. All of the directories created by the build are done so as this is the standard build layout that all repostitories in the .NET org use.

Ability to build a single DLL.

Can you be more specific here? What singel DLL are you trying to create? Any specific DLL in the project can be built by simply running msbuild in that directory. If you're looking for a single package that represents the whole of the repository then that doesn't exist. This repository produces a number of components for a variety of products so there is no one overarching item that encompasses all of them.

Instead of being able to build just that DLL

Why areyou unable to build that DLL? If you run msbuild on the project file for that DLL only that DLL and it's dependencies will be deleted.

It sounds like the root of many of your problems is you created a 10gig hard drive in order to build Roslyn. That is not going to be sufficient. Our build output is ~15Gig w/o hard links and our NuGet cache download is probably somewhere on the order of 1-5Gig (I haven't measured that in some time). A 25-30 Gig VM is probably the minimum that I would experiment with.

jasonmalinowski commented 5 years ago

Many, many warnings about SourceLink "Source control information is not available".

Paging @tmat on that one. Are you able to run git remote -v and share the output? When you do a build we embed URLs to GitHub in the build in the debugging information; this means that debugging can then fetch source files from GitHub. That needs to look at your remote URLs to know which GitHub repo it is; if you're using some of the "fancier" Git features it can sometimes break. (Or there's something else going on....)

tamlin-mike commented 5 years ago

You can just do msbuild /m <project>

Oh! Here I thought Build.cmd -> eng/*.ps1 did some black magic stuff.

Maybe I was skimming the wiki too fast, or the information that it is possible to build individual parts (even from command-line) isn't presented (or linked to) in close enough proximity to the starting build instructions?

%ROSLYNUSEHARDLINKS%

Excellent.

Are you asking for the ability to avoid building a specific set of tests? If so why do you believe that would be valuable?

I was looking for a way to not build any tests at all really. To reduce useless build time and disk consumption.

The scenario I had, I knew exactly what needed to be done (to fix a VS-crashing bug), and I needed to build a single dll. It ultimately failed due to some strong naming issue when trying to ngen it, but had it worked the build would have generated tens of gigabytes of space holding effectively useless junk, and a lot of CPU wasted, for a single DLL just a hair over 1.5 MB.

That was before I realized you don't actually need to run Build.cmd to "build world" so to speak.

VS install creates zero directories here.

I can see how that was confusing. I was referring to how VS installation drops all these locale directories in the many VS (and related) installation directories. I used that as an example for what I have previously experienced and consider wasteful and downright disrespectful waste of user resources. I didn't mean to imply VS dropped them into the roslyn build dir.

A 25-30 Gig VM is probably the minimum that I would experiment with.

It wasn't a VM, I was using a locally mapped VHD (to spare host volume from NTFS's insistence to generate fragmentation like it was in fashion).

Anyway, that's Good Info. Exactly the kind of conrete info I was hoping could be presented on the same page as the starting build instructions.

Are you able to run git remote -v

I used a two-stage repo. One bare clone of github roslyn, and then a working clone from that local bare. Perhaps that's an unanticipated scenario?

Thanks for the interest and feedback. I've learned a few new things about building roslyn today. I hope you in turn got some ideas about small but possibly important info that could be added to the early build instructions.

tmat commented 5 years ago

I used a two-stage repo. One bare clone of github roslyn, and then a working clone from that local bare.

Does the bare repo have a remote URL configured?

tamlin-mike commented 5 years ago

No remote URL. Only accessible locally via filesystem access.

jmarolf commented 5 years ago

Oh! Here I thought Build.cmd -> eng/*.ps1 did some black magic stuff.

restore.cmd does some black magic stuff. Once that has completed subsequent calls to msbuild should just work™

tmat commented 5 years ago

@jmarolf Restoring a single project should work as well. msbuild /restore foo.csproj, not need to call restore.cmd.

tmat commented 5 years ago

No remote URL. Only accessible locally via filesystem access.

I mean in the repo cloned from github.com/dotnet/roslyn. I assume you did

git clone http://github.com/dotnet/roslyn --bare

This adds a remote origin that points to http://github.com/dotnet/roslyn.

tamlin-mike commented 5 years ago

Yes, of course.

Bare "middle" repo:

[remote "origin"]
        url = https://github.com/dotnet/roslyn.git

Working repo:

[remote "origin"]
        url = LocalFilesystemPathWithDoubleBackslashes

Coming to think of it... could the backslashes be the unanticipated issue?

tmat commented 5 years ago

No, this scenario is currently unsupported, but as it happens I'm working on implementing it as we speak :) See https://github.com/dotnet/sourcelink/issues/418

tamlin-mike commented 5 years ago

Excellent, tmat. That information closes that concern.