Announcement: Engineering changes for corefx

weshaggard commented 7 years ago

Over the last couple months, we have been doing some overhauling of our corefx engineering system in the dev/eng branch. The time has finally come to get these changes into master so we are going to use this issue to give a heads up on the details.

Primary reasons for the changes

Build from source
- Value: Simplify on-boarding of new Linux distros
- As we have worked more in the open source space we finally made the realization that the source itself is also a product and we need to enable others to easily consume and build our full product from the source code itself. Our engineering system had numerous problems that made it difficult to bootstrap and build a full product from source with minimal external dependencies.
Package reduction
- Value: .NET Core is now 1 package instead of 100's of tiny packages
- We realized that we went overboard with the number of individual library packages we have. While it enables a very flexible system it also greatly increases the complexity of composing a fully functional framework.
Other changes
- Build produces flat product layout which is ready for execution
- Better build defaults (use your environment)

When are we merging

We are planning to begin merging changes on Friday 1/13 and to help with the merge conflicts we plan to disable commits/merges to corefx master branch on Friday 1/13, Sat, Sun, and Mon 1/16. We will open up the master branch again as soon as we have the merge in and the master branch functional. After the merge any active PRs will need to be rebased on top of master.

Fri 1/13 merge is in https://github.com/dotnet/corefx/pull/15165. The branch is unlocked now.

What does it mean to folks working in corefx

After pulling the merged changes you should do a full clean and build.

clean --all
build
build-tests

As part of this work we have split the build and build-tests into separate steps, and build will only build the product and build-tests will build and run the tests. We highly recommend reading over our developer guide to better understand the developer workflow https://github.com/dotnet/corefx/blob/dev/eng/Documentation/project-docs/developer-guide.md.

Given with the way we are building now you need to build from the root at least once before you can work on an individual project, from the command line or in VS.

What changed

Before we would multiplex our build configurations at every individual library level (i.e. .builds files) and in order to do that it required that we had a previous version of lots of the CoreFx libraries which we retrieved by restoring nuget packages of a previous corefx official build. That caused a lot of extra complexity in numerous ways such as restoring tons of nuget packages on every build as well as building for all configurations all the time. With our engineering work we have eliminated a lot of those complexities.

Now the only nuget packages are for things that don't build in corefx itself and the package restoring will be scoped to tool initialization and under the external directory. All dependencies that are part of corefx will be built live in corefx instead of being restored from older packages.

Now when you build from the root we will build all the reference assemblies and create a targeting pack in a flat directory and then build all the libraries referencing things from the targeting pack. By default this will produce a .NET Core build for the OS you are running on and it will create a flat runtime directory bin\runtime\netcoreapp-<OS>-Debug-x64 which will contain a fully runnable framework. That runtime directory can be used to run a .NET Core application via corerun or dotnet hosts. So no more groveling through the bin directory to try and cobble together a functional framework.

The tests will also build against this targeting pack and then run on the flat runtime directory that was produced. This also means no more copying or hard-linking the entire framework into every test output directory.

We believe these changes will make working in corefx a ton easier. However, it does add a little more work – you need to do root level build to setup the targeting pack and runtime before you can work on libraries. Moreover, if you want to work on more than one framework/configuration (.NET Core, desktop/netfx, and uap), you will need to do a root level build for each of the frameworks/configurations. Root level builds will only build the configuration that is specified so it will skip libraries that aren't applicable to that configuration.

We have replaced the .builds files at the library level with a configuration.props file. The configuration.props file will list out the set of supported configurations and when building we will pick and build the best configuration and skip it if there isn't a valid configuration for what you are building. For more details on the configurations see https://github.com/dotnet/corefx/blob/dev/eng/Documentation/coding-guidelines/project-guidelines.md.

Another part of the changes is in support of the package reduction goal. We now build a single package that contains all corefx libraries that are part of .NET Standard 2.0. This means that we will be deleting a lot of low level individual library packages and instead our root .NET Core package will contain all those libraries in the one package. This will greatly reduce the number of packages we produce and reduce the package graph depth and complexity for .NET Core applications. We hope this will remove the need for a degree in combinatorics and graph theory to be a .NET Core developer :).

Things left to be done

While we believe we got the basic .NET Core stack functional there are still a number issues that need to be addressed. We have tagged related issues with dev-eng and you can see the current backlog at https://github.com/dotnet/corefx/issues?q=is%3Aissue+is%3Aopen+label%3Adev-eng.

A few of the more notable ones are:

Desktop/Netfx build configuration and tests aren't functional https://github.com/dotnet/corefx/issues/14911 (ETA: 1-2 weeks)
UWP build configuration and tests aren't functional https://github.com/dotnet/corefx/issues/14592 (ETA: 2-3 weeks)

We hope that these changes will make working in corefx a lot better but please don't hesitate to let us know any issues you run into and we will do our best to address them.

karelz commented 7 years ago

Heads up to all CoreFX contributors, it will affect your dev workflow + you will need to rebase next week

cc'ing all recently active contributors (in random order), my apologies if I missed anyone (it was not intentional): @hughbe @justinvp @JonHanna @xoofx @AlexRadch @stephenmichaelf @nietras @benaadams @fujiy @jamesqo @SamuelEnglard @couven92 @tintoy @sparraguerra @vcsjones @bartdesmet @lmolkova @marek-safar @dnickless @bbowyersmyth @matekm @fitzchak @omajid @seanshpark @hseok-oh @ofirmakmal @hqueue @chunseoklee @Gregory-Bell @jcouv @Clockwork-Muse @jyoungyun @odyth @lemmaa @kant2002 @ahsonkhan @KrzysztofCwalina @lucenticus @bmeverett @ViIvanov @craigajohnson @tdupont750 @KirillOsenkov @svick @jirisykora83 @parjong @corivera @AntonLapounov @tralivali1234 @shrah

karelz commented 7 years ago

cc @khellang @cshung @AndyAyersMS @SedarG @nikitozz @varocarbas @andrewlock @cesarbs @roji @alaatm @vancem @cakine @TiagoSoczek @brfalcon @ChrisEelmaa @arc001 @thiagocamargos @mramosMS @anthonylangsworth @borgdylan @stevedesmond-ca

adamhathcock commented 7 years ago

So for NET Standard 2.0 there will be one large "mscorlib" instead of individual System.Linq etc packages?

Asking as a user, not a contributor here:)

weshaggard commented 7 years ago

@adamhathcock there will still be more than one assembly but one package Microsoft.NETCore.App. So yes there will still be a System.Linq.dll assembly but not a System.Linq nuget package.

shmuelie commented 7 years ago

@weshaggard So we won't be able to pair down the DLLs to just what we need?

weshaggard commented 7 years ago

For the shared framework that obviously isn't needed but for standalone apps we plan to work on adding some assembly level trimming to the publish. @terrajobst do we have any pointers for that feature?

shmuelie commented 7 years ago

So I just get the whole standard when writing libraries, no more saying what parts my library needs?

MattGal commented 7 years ago

Have we tested distributed Helix runs with this yet? I've seen PRs around this but have not seen a clean test build.

weshaggard commented 7 years ago

@SamuelEnglard Yes when you are writing a library you get all of .NET Standard APIs. For anything above .NET Standard you still need a reference, but with .NET Standard 2.0 it is much larger so hopefully that will not be necessary in most cases.

@MattGal please chat with @mellinoe about that work.

MaherJendoubi commented 7 years ago

What about compiler and runtime perfs?

mellinoe commented 7 years ago

@MattGal I've had some successful private builds and am currently working on adding the steps back into the official build definitions.

weshaggard commented 7 years ago

@MaherJendoubi I'm not sure I understand your question. What context are you referring to? Are you asking will the compiler perf be less if we always pass it .NET Standard? If that is what you are asking then I don't think it will be. .NET Standard 2.0 is still smaller then the .NET Framework and the compiler handles a lot more type references in that context just fine.

kevinchalet commented 7 years ago

For anything above .NET Standard you still need reference, but with .NET Standard 2.0 it is much larger so hopefully that will not be necessary in most cases.

And what about .NET Standard 1.x libraries? Do you plan to update the CoreFX NuGet packages so we can keep using them when targeting older .NET Standard TFMs?

weshaggard commented 7 years ago

@PinpointTownes We don't need to update the packages to support .NET Standard 1.x as the packages that are already released on NuGet already support those. If there are fixes needed for a servicing event then we will produce updates in our servicing branches. Also if you are writing a library targeting .NET Standard 1.x those will be supported in our .NET Core 2.0 package.

kevinchalet commented 7 years ago

We don't need to update the packages to support .NET Standard 1.x as the packages that are already released on NuGet already support those.

Actually, my question was not about using the bits that are already published, but future versions that will be released :smile:

Here's a concrete sample: let's say I have a .NET Standard 1.3 class library that depends on System.Security.Claims 4.3. If a bug is discovered and an updated library is released or if performance improvements are introduced in the next few months (e.g in System.Security.Claims 4.4), I suppose a .NET Core 1.x app won't be able to use the new bits, as they won't be published on NuGet.org, right?

weshaggard commented 7 years ago

The fixes will be rolled into our latest release, so .NET Core 2.0, in this case but they will not be in our older .NET Core 1.x releases. We will only be releasing servicing level fixes for those releases. To get the latest greatest changes people have to move to the latest .NET Core release. So you are correct a .NET Core 1.x will not be able to use them. That is true even with the small individual packages we produce today because when we release an updated version it generally bumps the versions of its dependencies as well and so you usually need to bump the full package graph which moves you to the latest .NET Core release.

Now I do want to add that this is only true for the things that ship as part of Microsoft.NETCore.App, which will not be everything that we build in corefx. There are things that will still ship as individual library packages on top of .NET Standard or .NET Core directly. However the majority of things in corefx will be part of Microsoft.NETCore.App.

kevinchalet commented 7 years ago

@weshaggard thanks for the clarification :+1:

MaherJendoubi commented 7 years ago

@weshaggard I mean one package still tons of references. Huge package has downsides.

weshaggard commented 7 years ago

@MaherJendoubi in the end it will be pulling down less files then what happens today. Today when you reference Microsoft.NETCore.App it has a pretty large package graph under it and causes it to pull a lot of packages and files in those packages that aren't even applicable to .NET Core. With the flat package it will only pull the files that are needed for .NET Core, so in the end I think it will have less impact on size on disk for the tools and building applications.

Now there are some potential concerns with producing a standalone .NET Core application because when you publish the application it will have all these binaries (which is true today as well) but we are planning work to do assembly level trimming for that scenario which should trim it down to just what is used by the application.

MaherJendoubi commented 7 years ago

@weshaggard thank you for enlightening me 👍

gulshan commented 7 years ago

Will the "assembly level trimming" be included within 2.0 release timeframe?

xoofx commented 7 years ago

This change is very much welcome, thanks!

About the problem I ran into recently in dotnet/runtime#19769 while trying to build a local coreclr with a local corefx, how this change will affect the cross building of this two inter-dependent repo?

Typically, are we going to have a repository like dotnet/versions where the commit id of coreclr and corefx working together will be stored? Currently the system is using a nuget version to match and almost force a myget server in between, so compiling the two repos in concert requires tweaking env variables and local files to fake a local build, which is very cumbersome... I would love to see these commit ids used somewhere...

Jetski5822 commented 7 years ago

Guy's, come on, this whole package reduction feels like a step back.

The whole idea behind this 'new world' was an 'opt in to what you need' approach. I remember watching Taylor Mullins talk at Orchard Harvest about the whole opt in approach... yes it was K Runtime etc at the time but the philosophy remained the same till now. I remember thinking, finally!!.

The part that I found most awesome, was, you just upgrade a small part of your framework, test, deploy. Rinse and repeat. You don't have to upgrade everything in one big shot.

The rest of the changes I think im cool with, except this one.

SteveDesmond-ca commented 7 years ago

Would it make sense to simplify the main package by including (instead of referencing) all its libraries, while still keeping the individual library packages available (and up-to-date)? That way, we have the choice: quick and easy vs specific.

Jetski5822 commented 7 years ago

Yeah, that would make much more sense... Then the people who want everything opt in to everything, and the people who want to keep their applications nimble and small, can do so too.

shmuelie commented 7 years ago

@Jetski5822 based on my understand of what @weshaggard has said, you will keep applications small. The difference is mainly two things:

1) Library developers don't specify the core assemblies they need. 2) Assembly trimming isn't done by the dev but by the tooling.

So in the end we get the small, self contained still but we don't have to manage the dependencies ourselves

weshaggard commented 7 years ago

Will the "assembly level trimming" be included within 2.0 release timeframe?

@gulshan I'm not sure if it is fully committed in this timeframe or not. I will try to find out.

About the problem I ran into recently in dotnet/runtime#19769 while trying to build a local coreclr with a local corefx, how this change will affect the cross building of this two inter-dependent repo?

@xoofx that scenario will become a lot easier now that we will have a flat runtime folder as part of the output. We are also working on other work that will make building the full .NET Core stack from source easier, that means orchestrating all the various different repo builds from one place. That is not quite ready yet but it will make your scenario much easier.

Would it make sense to simplify the main package by including (instead of referencing) all its libraries, while still keeping the individual library packages available (and up-to-date)? That way, we have the choice: quick and easy vs specific.

While that sounds nice in theory it is very difficult to accomplish in practice. Even today where we have the individual library packages whenever we update one we generally update its dependencies as well so you have to update not only it but its closure which can very quickly turn into updating all of .NET Core. Being able to maintain a library package such that it can run on any version of its past dependencies is quite difficult, you essentially have to turn every library into its own product (including lots of release and servicing branches) and test it in a huge amount of combinations. For that reason we always roll our library packages forward to the latest dependencies when we ship them, which means you don't have the characteristic of being able to update just one library package in most cases (except for maybe simple servicing level bug fixes).

As @SamuelEnglard points out the idea with .NET Core changing to one package is to ease the developer experience but that doesn't mean we loose the ability to trim the application nor does it mean we lose the ability to service or update a single library if we need to.

SteveDesmond-ca commented 7 years ago

Thanks, that makes sense -- I didn't really have an idea of how coupled the various CoreFX libraries were, so if the best atomic unit size is all of CoreFX, then I'm all for it!

xoofx commented 7 years ago

@stevedesmond-ca just computed the graph dependency of corefx libs: dotnet-corefx-dependencies

craigajohnson commented 7 years ago

I should know the answer to this but does Roslyn take a dependency on .NET Core, such as Roslyn using .NET Core to build itself which is then used to build future versions of .NET Core?

ellismg commented 7 years ago

I should know the answer to this but does Roslyn take a dependency on .NET Core, such as Roslyn using .NET Core to build itself which is then used to build future versions of .NET Core?

Yes. At least when building on Unix, Roslyn uses a toolset based on .NET Core plus a previously built version of the compiler to bootstrap themselves, and then they rebuild using that just built compiler.

karelz commented 7 years ago

FYI: dev/eng branch RI'd yesterday (#15165) and we restarted merging PRs again.

BTW: Mon is MS holiday, so if there is any fall out, it may need to wait for Tue to get the right attention from folks who understand all the new infra. Kudos to the folks who made it happen - @weshaggard @ericstj @mellinoe @joperezr @chcosta @karajas.

MaherJendoubi commented 7 years ago

@weshaggard the new size is impressive :-) 100 times!

ericstj commented 7 years ago

@MaherJendoubi you are looking at the Microsoft.NETCore.App package. Previously this only contained dependencies so the size of the single package was not a representation of the closure of all packages. Now this contains all reference assemblies for the entire framework.

A better test would be to restore NETCore.App with an empty packages folder before and after the change. I think you'll see a substantial decrease the other direction.

MaherJendoubi commented 7 years ago

@ericstj Thanks!

saurabh500 commented 7 years ago

How are the assembly dependencies being managed after this merge? They were in project.json which are not around after this merge.

weshaggard commented 7 years ago

See https://github.com/dotnet/corefx/blob/master/Documentation/coding-guidelines/project-guidelines.md#src. You should now just put a Reference in your src projects.

Mike-E-angelo commented 7 years ago

Great work everyone. Glad to hear the details and all the work. I was surprised to see that:

You attempted to do everything on a Friday the 13th.
I actually went through this past Friday not realizing that it was as such. 😮

Anyways, glad to see the efforts here and how everything is coalescing into the next generation of .NET tech. FWIW, this thread made The Week in .NET: https://blogs.msdn.microsoft.com/dotnet/2017/01/18/the-week-in-net-on-net-with-david-pine-pwdless-terraria/

omajid commented 7 years ago

Is there any documentation on "Simplify on-boarding of new Linux distros"? I am trying to get corefx to build on RHEL 7.3 and running into https://github.com/dotnet/corefx/issues/15274. Even a rough sketch on how to on-board new distros will be very useful.

jchannon commented 7 years ago

I'd like a more official announcement and what it means to current apps and what a new app now looks like

On 18 January 2017 at 18:58, Omair Majid notifications@github.com wrote:

Is there any documentation on "Simplify on-boarding of new Linux distros"? I am trying to get corefx to build on RHEL 7.3 and running into dotnet/corefx#15274 https://github.com/dotnet/corefx/issues/15274. Even a rough sketch on how to on-board new distros will be very useful.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dotnet/corefx/issues/15135#issuecomment-273566921, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGapvIVoNada4RKPYjGdycaFbKarAgdks5rTmDFgaJpZM4LiRq6 .

weshaggard commented 7 years ago

@omajid lets continue that conversation on the issue you filed.

@jchannon there will be more information on the build from source effort once we get closer to shipping. cc @bleroy @ellismg

lmolkova commented 7 years ago

What about netfx? Will there be another single package targeting it?

weshaggard commented 7 years ago

What about netfx? Will there be another single package targeting it?

No. The large single netfx package is the .NET Framework itself. Anything that doesn't ship inbox that we support on .NET Framework will still be shipped as an individual nuget package as it is today. The large single package is mostly about the core set of libraries that make up the lower layers, at a high level you can think about that be things as part of .NET Standard, although it isn't exactly that.

weshaggard commented 7 years ago

We still have a small tail of issues we are working through but we have mostly finished this work so I'm closing this issue now.

dotnet / runtime