[Proposal] Consider building and distributing dotnet using official distribution package sources

For now, Microsoft is using it's own 3rdparty repositories, located at packages.microsoft.com. While dotnet is an open source project under MIT license, dotnet distributed like a proprietary 3rdparty blob without source code available (from OSS point of view).

And even more, I can't find .src.rpm packages on packages.microsoft.com repositories, that means the build steps required to produce this rpm package is hidden from us, users. As the source code the package is built from. This raises questions: are those packages actually built from source or they're just a repack of a portable .tar.gz? This is not transparent.

The lack of official/semiofficial packages in distribution repositories (in my case, Fedora. But is valid for Debian/Ubuntu too) slows down the spread of .NET on Linux. There's some of pros for integration, I could think of:

Distribution packages built with target distribution in mind. 3rdparty packages doesn't integrate into the target distribution well. They built to be widely compatible, as I can tell without .spec file available.
While the package integrated into distribution repositories can be installed with one command, as the SDK for development. 3rdparty packages require additional, unnecessary steps to install/update. There's a need to install 3party repo just to use a runtime and to run .Net application (not self-contained). And for RedHat-based distributions, you can even use rpms to produce containers, this allows dotnet to be used easier in prod (bonus points).
3rdparty packages has a slowdown in supporting a new distribution version. Fedora suffers from that mostly, because of update cycle. Packages integrated in Fedora can be built against rawhide as future target and issues with the new distribution version can be corrected even before it becomes branched.
Official distribution packages are transparent to users and allow contribution to the build process on the same level, as GitHub PR's. Just git pull, edit spec/dist, push, pull request.
Distribution packages are always build and run against latest shared library versions available in distribution. This means security, but in rare cases - a bit of a headache.
Updates easily handled by distribution package manager. If there's fixes, just rebuild the package. Some distributions allow diff updates for packages, using projects like DRPM.
Using 3rdparty repos to install OSS without proper reason not to integrate it to the distribution (like copyright or weird licensing) is discouraged by community.
RPM Specs and DEB dists can be used as a perfect reference for build steps required to build package correctly from scratch for Linux. Which is good for other, not directly supported distributions. If you decide to not support ArchLinux/Gentoo, for example, using available spec/dist files they can recreate build process and integrate dotnet on their own.
Distribution integrated runtime will allow packaging .Net software in that distribution natively, without the need to use self-contained deployment. There's a change what crossplatform dotnet gui/cli applications will appear in the future, native dotnet support in distributions is a huge welcoming gift for developers, even in the containers era. Perfect example: PowerShell, I'm looking at you. For now, even if dotnet supports Linux, it's still considered as a foreign/alien technology, because it lacks integration with Linux on the level of Mono. This slows down it's usage in Linux. (Please, don't be like Java, I'm begging you :pleading_face:!)

TL;DR (main ones): It requires some steps to install, it can be distributed without support for my distribution features (only standard linux features included), the build process is not transparent (what is discouraged by OSS community + contribution is limited), updates coming from single point (hello SPOF) and unable to package an app using dotnet-runtime into the distribution.

As of cons:

Packages in distributions must be maintainted by someone. By a community member or by developer itself. Yes, it requires some time to be invested, of course.

Or at least, package must be published for review to the distribution and this can be decided later on.
The RPM spec/DEB dists file(s) published should comply with the target distribution guidelines and should build software from the source code. This includes some degree of pain on the first steps, because Linux is fragmented. But it will pay out with time, because spec file/dists are open source. Everyone capable of building rpms/debs could contribute later.

For RPMs, it doesn't necessary to make a new spec file for every distribution in family. There could be one for CentOS/Fedora and one for OpenSUSE/SLES family. With some RPM macro [s]crutches[/s] magic, this could be done for sure.
In most distributions, bundling 3rdparty libraries is discouraged or forbidden. This must be considered. If the library is OSS, this should not be a problem, but maybe, a little more of a headache/time investment.

Integrating dotnet into distributions is a next big step in Linux support which should be considered.

You'll be pleased to hear that we've been working on this for a while, with the source-build project: https://github.com/dotnet/source-build. In fact, .NET Core is shipping as a distro package in Fedora 32: https://docs.fedoraproject.org/en-US/fedora/f32/release-notes/developers/Development_Dotnet/. 😄 We have been shipping a source-built .NET Core in RHEL for a while now, as well, and it's in CentOS 8 AppStream.

There is an official doc about guidelines for distro package maintainers here: https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging.

Yes, packages.microsoft.com contains the "Microsoft distribution" of .NET Core, which does not build from source (as far as Linux distro expectations go), rather it's assembled from many repos' builds and involves prebuilts every step of the way, using varying pools of build machines and some optional tooling that isn't available publicly. It doesn't use spec files or anything of the sort, rather tools like fpm and a custom thing we made a long time ago for debs (dotnet-deb-tool). We do try to make the Microsoft RPMs/Debs match distro guidelines in terms of layout and metadata, though.

@fixinggunsinair If you want to get more involved in .NET Core on Fedora, please feel free to join the .NET SIG: https://fedoraproject.org/wiki/SIGs/DotNet

I had a couple of thoughts, though:

(Please, don't be like Java, I'm begging you!)

Could you clarify what this means? I have been looking at OpenJDK as a sort of role model of what .NET could be. What sort of issues do you see in Java on Linux that we should try and stay away from?

unable to package an app using dotnet-runtime into the distribution.

Unfortunately, even with .NET Core being part of a distro, this problem doesn't immediately get fixed. The way .NET Core works, everything still tries to get nuget packages from nuget.org, making this quite tricky. This is something we are still working on: https://github.com/dotnet/docs/issues/17757

@dagood

Glad to hear you thought about it even before I found out what it's already packaged as RPM. Now I feel kind of... uninformed %). You should definitely update the documentation on the microsoft.com to reflect that.

You can close the issue or keep it until you consider it's done and ready, if you want. So it would be visible for people like me.

Yes, packages.microsoft.com contains the "Microsoft distribution" of .NET Core, which does not build from source (as far as Linux distro expectations go)

Well, nothing stops you from using an internal repo made your way, but in Linux it feels like downloading a proprietary thing from publisher site, which will install itself THEIR WAY somewhere into /opt and you can't do anything with that.

@omajid

That's my subjective feeling of things got out accidentally.

The Java way, is to stay compatible with Linux but abstract from it as much as possible. Java things would run on Linux and work, but the internals would be so abstracted from the current state of things, it would feel like it's an alien or archaic technology. While on Windows, you sometimes can never say what this particular window is from Java application, because it's integrated as best as it can. But on Linux, you do. They didn't released SWT/JavaFX gtk3 wrapper for a long time, until gtk2 got deprecated, for example. So there was issues with GUI. But I hadn't used Java really for a long time and I'm glad I do.

Or maybe, TL;DR: Java way is to treat Linux only as POSIX-compatible Unix-way operating system, which Java must work on, but without delving to how Linux-way things work at all.

But that's subjective feeling, that doesn't compare to implementation. Of course it's a hard task to make one framework for everything without some level of abstraction in mind, considering Linux is a fragmentation hell and has freedom to fragmentate the way you freaking want.

So...

Unfortunately, even with .NET Core being part of a distro, this problem doesn't immediately get fixed.

I understand. Because by Fedora packaging guidelines, you should package every freaking bundled library into rpm before that happen. Or you CAN bundle, but only if licensing allows it and you promise to package every lib in new packages %).

But if someday it can be used in place of Mono, I'll be glad to see that.

Still a huge step forward.

Now I need to find out do PowerShell team follows you in that way or should I create another issue for them. Because I care about them too to become integrated. %)

You should definitely update the documentation on the microsoft.com to reflect that.

We're looking at ways to do this and have some ideas, but I'm curious what you saw to get the specific conclusions you arrived at. Source-build does have some docs, the most straightforward being https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging, but https://docs.microsoft.com/en-us/dotnet/core/install/linux-package-manager-rhel8 does point at RHEL 8 having .NET Core in the distro. Did you maybe start from https://docs.microsoft.com/en-us/dotnet/core/install/linux-package-manager-fedora31 and hop over to packages.microsoft.com to look in more detail, or are there more prominent places you have in mind that you would have liked to see more info about this?

(I have filed https://github.com/dotnet/docs/issues/18019 to put up a similar page for CentOS 8 as the one we have for RHEL 8, not that that's enough.)

Well, nothing stops you from using an internal repo made your way

This is too far--the Microsoft build does run using public sources. For example, you can see what our dotnet/runtime build does in Azure DevOps by starting at https://github.com/dotnet/runtime/blob/master/eng/pipelines/runtime.yml. That does involve non-public toolsets downloaded as prebuilts, but mostly that's limited to Microsoft signing tech. I'm not suggesting trying to follow it to do a local build, I do think that would be too difficult and I'm definitely not disputing your argument about how proper source-build helps devs new to the project figure out what's going on.

which will install itself THEIR WAY somewhere into /opt and you can't do anything with that.

If you encounter something with the .NET Core packages that seems to violate package guidelines for the target distro, I encourage you to open an issue and we can discuss it. We definitely don't install into /opt. 😛

PowerShell team

I'm not sure, maybe someone else in the thread knows more. I do see Make Powershell Core suitable for inclusion in Debian #5869 (from 2018) in their issues but no others that are obviously about distro packaging.

@dagood

https://docs.microsoft.com/en-us/dotnet/core/distribution-packaging

Never saw it until this issue. It's somelike deep in documentation and Google never showed me this page. I'm a simple man and when I need install instructions, I just google dotnet fedora install. Not the dotnet fedora distribution package (but it shows same results) or build instructions. And the first result I got, was:

https://docs.microsoft.com/en-us/dotnet/core/install/linux-package-manager-fedora31

Or even Fedora 30 link. But it's all the same instructions.

Or the link to the snapcraft.io, which mustdie.

https://docs.microsoft.com/en-us/dotnet/core/install/linux-package-manager-rhel8

Why do I need RHEL 8? I don't have that. So why should I read this page? %)

So, to perfectly reflect that, you may just add Fedora 32 page and link to the Fedora package first and alternatively you can use the ones we provide if you want it thing.

Google is changing right now. Because now I see links to the .NET SIG on first results:

And this screenshot made in the incognito tab. So it should be a global result.

I never saw dotnet package listed in the fedora repositories. But this changed magically on yesterday, because dotnet package from updates-testing just replaced the one from the packages.microsoft.com... I just did a dnf update on my toolbox. Availability in distribution also helps spread the world.

I'm not suggesting trying to follow it to do a local build, I do think that would be too difficult and I'm definitely not disputing your argument about how proper source-build helps devs new to the project figure out what's going on.

RPM Spec is better in this task, because you don't have to describe the build process of whole toolchain in one file. Even better, if you know what to look at. That one is relatively simple.

We definitely don't install into /opt. :stuck_out_tongue:

Got your word. I just have some Vietnam flashbacks about 3rdparty RPMs with closed source software.

So, to perfectly reflect that, you may just add Fedora 32 page and link to the Fedora package first and alternatively you can use the ones we provide if you want it thing.

I filed https://github.com/dotnet/docs/issues/18038 to track that.

dotnet / core

[Proposal] Consider building and distributing dotnet using official distribution package sources #4601