search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.09k stars 4.7k forks source link

Linux Packaging #2622

Closed RheaAyase closed 4 years ago

RheaAyase commented 7 years ago

A few things to make clear regarding packaging.

  1. What are we packaging? A single package that contains everything, or two or even three separate packages? (host, framework...)

  2. How do we call it? There are debian based packages with naming as dotnet-host, dotnet-hostfxr, dotnet-sharedframework. Now imagine dnf install or apt-get install - what would the user install when, which takes me to my next question...

  3. What are the dependencies of these packages, which would depend on which one?

  4. How do we version it? What would our packages be when it comes to versions? Using the above dotnet-host as example: dotnet-host-1.0

  5. And how can user combine versions/whatever at the same time? Say Bob has sdk 1.1 some time in the future, and wishes to be able to target framework 1.0 and 1.1. Our understanding is that Bob would have to install a package for said sdk, which as such would have only the latest framework as dependency (or not even that) and Bob would have to optionally add whatever other framework he would like to publish for.

  6. Bonus question. Where does our (packagers) responsibility start? Say we will have to fix something our-distro-specific and pushing it upstream is not good enough. Say we need the fix ASAP. We can of course build it ourselves with the fix, and we would, but what about nuget packages that do not contain this fix, if someone would be targeting their application for our distro, from some other platform? Can we be responsible for this somehow, can we get the fix in faster than with next whatever update in a few months?

Last but not least, I would like to suggest that these could be compiled into nice packaging guidelines for all the various Linux distributions out there.

cc @Fedora-dotnet @omajid @tmds @nmilosev (I hope that I got everything covered and didn't mix my notes up, please correct me if so...)

ellismg commented 7 years ago

/cc @Petermarcu @piotrpMSFT @leecow @bleroy @gkhanna79

What are we packaging? A single package that contains everything, or two or even three separate packages? (host, framework...)

Let's take a moment and step back and look at the different pieces of the product. There are three major parts:

  1. The host (also known as the muxer or dotnet). This is the native goup built from dotnet/core-setup. It contains both the host itself (dotnet) and the supporting policy libraries (installed in host/fxr). dotnet's job is to either active a run-time and launch an application against it or to active an SDK and let the SDK handle a command. The reason there are two components is due to how we service the component on Windows. On Windows trying to update a file that's in use by another program requires a reboot, so we want to put as much logic into something that we can version easily (in case there are bugs). dotnet itself is very small, it tries to find the newest version of the framework resolver (libhostfxr.so) and then call functions there. That library then needs to find the correct run-time and activate it. While you can have multiple copies of libhostfxr.so installed, the newest version wins.
  2. One or more shared frameworks. The shared frameworks live in shared/Microsoft.NETCore.App/<version>. The host will roll forward across patch versions, so if an application targets Microsoft.NETCore.App 1.0.0 and only 1.0.4 is present, it will launch against the 1.0.4 version. The SDK, as a managed application, has a dependency on a version of the shared framework, but you can have additional shared framework's installed (e.g. maybe you want to have the FTS and LTS versions installed)
  3. One or more versions of the tooling. While you can have multiple versions of the tools installed (and the muxer can load the right one based on project settings) the recommendation is to always use the newest tools (you can always target older versions of the shared framework).

How do we call it? There are debian based packages with naming as dotnet-host, dotnet-hostfxr, dotnet-sharedframework. Now imagine dnf install or apt-get install - what would the user install when, which takes me to my next question...

What are the dependencies of these packages, which would depend on which one?

How do we version it? What would our packages be when it comes to versions? Using the above dotnet-host as example: dotnet-host-1.0

We haven't really gone back and done a look across all the packages to understand how we want to name them, but given the above, this would be my straw-man (note I haven't actually talked to anyone about this, so this isn't some official guidance, it's just what I've been thinking recently.)

As far as dependencies go, I imagine that dotnet-sdk would depend on some version of dotnet-host and some dotnet-sharedframework-X.Y package (whatever package represents the shared framework the SDK was compiled for. I can image that dotnet-sharedframework-X.Y depends on the dotnet-host package as well, since if you are installing a shared framework you probabbly want the host so you can actually activate it. It's also possible at some point that you might need version > X of the host to activate some version of the framework.

And how can user combine versions/whatever at the same time? Say Bob has sdk 1.1 some time in the future, and wishes to be able to target framework 1.0 and 1.1. Our understanding is that Bob would have to install a package for said sdk, which as such would have only the latest framework as dependency (or not even that) and Bob would have to optionally add whatever other framework he would like to publish for.

In the above world, the host and tooling version in place. Shared frameworks can be installed independently at the major.minor level, but there's no packages for specific patch versions.

Bonus question. Where does our (packagers) responsibility start? Say we will have to fix something our-distro-specific and pushing it upstream is not good enough. Say we need the fix ASAP. We can of course build it ourselves with the fix, and we would, but what about nuget packages that do not contain this fix, if someone would be targeting their application for our distro, from some other platform? Can we be responsible for this somehow, can we get the fix in faster than with next whatever update in a few months?

A distro maintainer could always update whatever code ships in the shared framework, build a new copy of the shared framework and release an update. The issue above is only an issue for two cases:

There aren't great solutions for either cases. Since these dependencies are pulled down from nuget.org, there's no easy way for folks to do patches without having a revision shipped. The closest workaround would be to have a custom nuget server (or local source) and tell folks to reference that (and then publish packages to that feed which have you fixes). I believe things could be ordered in a way such that your version of the package would "win".

For the first case, it's possible that the new dotnet-cache feature could help. In this case, you could publish updated versions of the files to the cache and then when publishing applications folks could say: "I know these assemblies are cached, please don't include them in my application".

Regardless, there's real engineering work to be done here. I'm interested in how distributions handle other package managers. E.g. if I'm using npm, when do I get distro maintained versions of packages and when do I get things from npm.org?

Last but not least, I would like to suggest that these could be compiled into nice packaging guidelines for all the various Linux distributions out there.

Bertrand plans to produce something, but as I said above, we don't have great guidance today, so if you can help us figure out what makes sense that would be great.

bleroy commented 7 years ago

Seen from the user's perspective, I would expect most developers will want to just install the latest SDK. Users who want to setup a server environment where the tooling is not required will prefer dotnet-sharedframework-*. I don't see a scenario where you'd get only the muxer, this is mostly to be a dependency for the others. This makes me think we might want to tweak those package names a bit so the most common usage is made easiest. Maybe shorten the shared frameworks to just dotnet-1.0, dotnet-1.1, etc. dotnet-sdk is nice and clear, but we might want to give it dibs to the even simpler dotnet package name. This would be nice because someone just guessing the package name would mostly get exactly what they would expect.

I was thinking the same thing about giving distro package maintainers possibilities to update through a private NuGet feed that wins over nuget.org. Looks like the right thing to do.

About npm, as far as I can tell, things are a little different. The packages include npm, node, or both. Npm is just the tooling. Node does contain the standard library, but that pretty much comes as a monolithic thing, not packages. Everything else comes from the public npm repo, or from private npm repos. Those are relatively recent, and I haven't seen signs that they are used for distro-specific patching. I'm not even sure the scenario makes much sense for Node because of the monolithic nature of the SL. If you want, I could look into other platforms that would be closer in nature to .NET. Maybe Java or Go?

Of course this will have to be settled more precisely and written down in documentation form.

Petermarcu commented 7 years ago

One thing I've liked is a package convention I've seen which has "platformname" for the execution runtime, and "platformname-dev" for the development assets for that package. Is that a convention we should follow? It's very intuitive in that people ask the right questions. If someone sees a dockerfile with a -dev package in it they know its for development and they also then look for one that doesn't have -dev in it for production. This is similar to the first proposal @bleroy gave.

Java: sudo apt-get install default-jre gets the runtime sudo apt-get install default-jdk gets the sdk.

Java is the most like .NET in that it has a "shared framework". Things like go are different because you get the runtime as an output of your build. Node is somewhat similar in that you can run on a shared node install but it is different in that its not a compiled language and its completely natural to expect a compiler and most of the SDK to be available at runtime.

There is a lot of other content here that I didn't get to yet but I wanted to mention that. I'll try to read more of this tonight.

bleroy commented 7 years ago

Maybe:

Petermarcu commented 7 years ago

I personally like that pattern

Petermarcu commented 7 years ago

A couple comments but I think @ellismg has really well articulated much of it.

I could see there being a challenge when installing the dotnet-sdk package (latest) brings in dotnet-1.1, then we update the sdk to have 2.0 in it and updating dotnet-sdk would remove dotnet-1.1 and install dotnet-2.0

Am I correct on that being the behavior of the package manager? Is there a way to express that a dependency should be brought in on install but not removed when the referring package changes? Based on this, there may be other ways we should think about this.

One of the other options is to couple the sdk with a specific runtime and host to deliver only whole stacks.

dotnet-sdk -> dotnet-1.1-sdk -> dotnet-1.1 ->dotnet-host-1.1 -> dotnet-versionselector

update after 2.0 ships:

dotnet-sdk -> dotnet-2.0-sdk -> dotnet-2.0 ->dotnet-host-2.0 -> dotnet-versionselector

Hopefully this would leave dotnet-1.1-sdk alone if it were there and just install dotnet-2.0-sdk and its dependencies as well.

This is where my package manager knowledge falls short so I'm definitely looking for insight from others :)

My goal is that updating the SDK wouldn't remove a runtime that apps on the machine may be depending on.

omajid commented 7 years ago

Hey everyone. Thanks for the great input and thoughts.

@ellismg said:

It's also possible at some point that you might need version > X of the host to activate some version of the framework.

Would it be fair to say that users will want:

  1. Latest version of dotnet-host (or equivalent)
  2. Latest version of dotnet-sdk (or equivalent)
  3. Multiple versions of dotnet-sharedfamework (or equivalent)

Is this combination is working for 1.0 and 1.1? Is it expected to continue working for 2.0 and foreseeable future?

If multiple installations of host are supported, is the dotnet binary shared and only hostfxr.so provided by different packages?

I'm interested in how distributions handle other package managers. E.g. if I'm using npm, when do I get distro maintained versions of packages and when do I get things from npm.org?

I can describe how some distributions handle what Maven does. Maven is a build tool for Java that does a bunch of things, but I am only going to talk about the pieces that are analogous to dotnet restore.

Fedora ships with a maven version that pretty much behaves like upstream. So if a user uses mvn (think dotnet restore), maven will go out to Maven Central (think nuget.org) and download the packages exactly like how any other upstream build of maven will do it. That means a user using mvn will not run into any surprises if they have used mvn anywhere else.

Fedora also ships with a variant of maven called mvn-local that ignores packages from Maven Central. Fedora uses that for creating distribution packages for maven. mvn-local obeys distribution policies: it does not check Maven Central for packages, it only looks for packages currently installed on disk and uses significantly relaxed version constraints to find matching packages.

We might want to do something similar: the default mode of dotnet restore will do what it does on any platform but a custom variant might use some other command to find only the distro-available dependencies.

@Petermarcu said:

One thing I've liked is a package convention I've seen which has "platformname" for the execution runtime, and "platformname-dev" for the development assets for that package. Is that a convention we should follow?

Just an FYI, some distributions use -dev, others use -devel here. Most often, these are used for C/C++ packages, where -dev (or -devel) contains the headers and unversioned shared object.

I like the dotnet-1.0 and dotnet-sdk idea. I also prefer -runtime over -sharedframework, but we should pick a name that will make .NET Core internally consistent. If .NET Core only ever refers to "Shared Framework" then calling the distro package something different might be confusing.

A dotnet package which is equivalent to the latest dotnet-x.y might need some considering. Python, for example, suggest #!/usr/bin/python to invoke python2, not python3. What are our API compat goals?

I could see there being a challenge when installing the dotnet-sdk package (latest) brings in dotnet-1.1, then we update the sdk to have 2.0 in it and updating dotnet-sdk would remove dotnet-1.1 and install dotnet-2.0.

Am I correct on that being the behavior of the package manager?

Most package managers are not that smart. In fact, I don't think anything is ever removed automatically. Both Debian and Fedora remove automatically installed packages on explicit uninstall.

Petermarcu commented 7 years ago

Good input. Sounds like "-sdk" is the way to go for the name. Sounds like we also don't need "-runtime" because thats what "dotnet" is when it doesn't have "-sdk".

  1. Latest version of dotnet-host (or equivalent)
  2. Latest version of dotnet-sdk (or equivalent)
  3. Multiple versions of dotnet-sharedfamework (or equivalent)

dotnet/core-setup#1 and dotnet/core-setup#2 are correct.

For dotnet/core-setup#3 it's unclear to me if over time users will want to be able to get multiple copies of "-sdk". That would depend on the SDK always being backwards compatible. If it ever broke, the fix would be to explicitly install some version of the "-sdk" package. To me, that says that dotnet-sdk needs to have a version option as well like dotnet-1.0-sdk.

leecow commented 7 years ago

To touch on a question in Matt's original post, the current Ubuntu debs are authored to resolve and download dependencies at install time. eg apt-get install dotnet-dev-1.0.0-preview2.1-003177 also downloads and installs the defined dotnet-host, dotnet-hostfxr, dotnet-sharedframework and dotnet-sdk (in that order) packages.

I would want to be very careful about automatically removing past versions.

gkhanna79 commented 7 years ago

When a newer version of the host or resolver comes out, this package would have its version number bumped.

@ellismg Does this mean the new package will be dotnet-host-1.0.1?

dotnet-sharedframework-1.0 is offered an update and that update would remove the 1.0.4 folder from shared/Microsoft.NETCore.App and lay down a 1.0.5 version

How would the user revert back to 1.0.4 if they ran into issues with 1.0.5?

I can image that dotnet-sharedframework-X.Y depends on the dotnet-host package as well

Correct.

This makes me think we might want to tweak those package names a bit so the most common usage is made easiest

I think we want to have the packages be componentized and something like dotnet-1.0 is what would be equivalent to the Windows bundle we generate (whether, on RHEL, it is a bundle or triggers the chain is an implementation detail). dotnet-sharedframework-*is still worth having in its own right.

then we update the sdk to have 2.0 in it and updating dotnet-sdk would remove dotnet-1.1 and install dotnet-2.0

Is the convention for things like dotnet-sdk to be the version distro maintainer are chosen to be default or is it intended to be latest? I ask this because what I have seen (e.g. with lldb) is that packages like lldb seem to be distro blessed versions and to pull down the specific package you need to call it out (e.g. lldb-3.8). Thus, if maintainer chooses to update the defaults (from 2.0 to 2.1), then uninstalling the older version maybe fine - though uninstalling across versions (2.0 uninstalling 1.1) does not sound right.

ellismg commented 7 years ago

When a newer version of the host or resolver comes out, this package would have its version number bumped.

@ellismg Does this mean the new package will be dotnet-host-1.0.1?

There are two components to the package. The name of the package (in my proposal, that's dotnet-host) and the version. I'd advocate just bumping the version. That means anyone who does a dnf update and has dotnet-host installed would get the newer version.

This is in contrast to how I would say we should handle the shared framework. In that world we bake the major and minor number into the package name and bump the version of the RPM when patch releases come out. When a new major or minor version of the SF is released, packagers produce a new package (with a 1.0.0 version).

How would the user revert back to 1.0.4 if they ran into issues with 1.0.5?

The same way they revert to an older version of any software that their package maintainer updates if there are issues. Sometimes the rpm's are archived and there's tooling to downgrade a package, sometimes you may need to build it yourself (from the source RPM).

Is the convention for things like dotnet-sdk to be the version distro maintainer are chosen to be default or is it intended to be latest? I ask this because what I have seen (e.g. with lldb) is that packages like lldb seem to be distro blessed versions and to pull down the specific package you need to call it out (e.g. lldb-3.8). Thus, if maintainer chooses to update the defaults (from 2.0 to 2.1), then uninstalling the older version maybe fine - though uninstalling across versions (2.0 uninstalling 1.1) does not sound right.

My view on the situation (and I'm sure I'll be corrected by folks more in the know like @omajid) is that packages which need to be installable side by side bake version numbers into their package names. python vs python3 is a representative example here. clang and friends sometimes do this because certain ABIs are not stable across versions and so it's reasonable to have multiple installed.

If we think it makes sense to have the ability to install SDKs side by side, then we probably want to include part of the version in the package identity. If we don't expect that's something we want to support (easily) because we believe the move to latest story is what we want everyone doing, then we should not include version information in the package name.

omajid commented 7 years ago

Matt's entirely correct. A few points that I thought might be worth clarifying.

A (distro) package is generally identified by two components: name and version. The version in turn often consists of two components: the upstream version and the distribution version. As an example for python on my system, the name is python, the upstream version is 2.7.13 and the distro version is 1.fc25 (Ubuntu might do it as 2ubuntu1). The upstream version is the version that the upstream release corresponds to. The distro version is essentially how many times a distro has built/patched the upstream version.

When it comes time to upgrade packages, the package manager looks at names to find "same" packages and versions to determine if it's an upgrade. For example, if it sees I have python-2.7.13-1.fc25 installed and it sees there's a package named python-2.7.13-2.fc25 available, it uses the name to identify that they are both the same package. It then compares the versions to identify that the new package is an upgrade for the old package. When asked to perform the upgrade, the package manager removes the old version and installs the new version. It's a bit more complicated than that to ensure that a no point there are missing files, but conceptually a removal followed by an install is what happens.

So if we call two packages as foo, it tells the package manager they are different versions of the same package and one should be replaced with the other on an upgrade/downgrade.

In other words, if we want to make two packages be installable side-by-side, we should give them different names. If we want to make one package be an upgrade for another, we should give them the same name.

Often implicit in this entire convention is that the package stays compatible. A bump from one version to the next should not break anything, including the user experience and the command line interaction. This isn't true for distros with rolling releases, but they announce an upcoming break in advance.

So if foo version 2 is not compatible with foo version 1, and both are being added to a distro they may get added as foo and foo2 or foo1 and foo or even foo1 and foo2. This is up to the package maintainers.

Given that we want 1.0.1 shared framework to be replaced by 1.0.2, we should call it something like dotnet-1.0-sharedframework(or dotnet-sharedframework-1.0), and give them versions of 1.0.1 and 1.0.2.

I prefer dotnet-$VERSION-$COMPONENT over dotnet-$COMPONENT-$VERSION, as that allows users to do sudo dnf install 'dotnet-1.0*' or sudo apt install 'dotnet-1.0*' to install all components for dotnet-1.0.

Also, my understanding is that upgrading never removes a package. There's two notable exceptions. One, a package foo can explicitly say that it's a replacement and upgrade for bar and the package manager will remove bar and install foo to comply. Another is basically a special case for the kernel package, lets ignore that for this discussion.

RheaAyase commented 7 years ago

Sorry for the somewhat late reply, I had a bunch of PTO last week and didn't get to sit down long enough to go through this.

@bleroy said:

dotnet -> latest runtime / shared fx dotnet-major.minor -> specific runtime / shared fx dotnet-sdk -> latest sdk

Yes that is what I was thinking about (I don't like the current naming for debian branch...)

@omajid said:

Also, my understanding is that upgrading never removes a package. There's two notable exceptions. One, a package foo can explicitly say that it's a replacement and upgrade for bar and the package manager will remove bar and install foo to comply. Another is basically a special case for the kernel package, lets ignore that for this discussion.

Not even if its dependency no longer exists? I thought that it would remove said dependency.

So to sum the whole discussion up and what we could agree on I hope. Please correct me if I mixed something up =)

Packages:

This scenario has one right now relevant problem. Someone may want to stick with project.json for a while longer, but as soon as they run update their sdk will roll over to the csproj tooling and they are screwed... I think that this is a breaking change we all have to get over and it will be fine =) If someone would strongly insist on having multiple versions of the sdk, then it would have to be handled the same as the framework. (Whether dotnet-version-sdk or dotnet-sdk-version should be up to the maintainer / distro conventions, same as dev/devel)

(Note that the above update refers to package manager update command, such as apt-get update or dnf update)

Lets please iterate on the above suggested conventions and lets come up with something that could be considered guidelines so we don't have complete chaos between different Linux distributions... Can anyone summon other people involved in packaging for other distros?

omajid commented 7 years ago

Not even if its dependency no longer exists? I thought that it would remove said dependency.

The case that I am referring to is the following:

In this case the dependency on bar does not exist when foo, 2 is installed. Upgrading from foo, 1 to foo, 2 does not remove bar. Both Fedora and Debian have explicit steps to make a package (baz here) replace another by uninstalling that other package (bar here).

I just tried this out. I created 3 packages:

foo requires bar. Build and install foo and bar:

$ sudo dnf install ./RPMS/x86_64/foo-1-1.fc25.x86_64.rpm ./RPMS/x86_64/bar-1-1.fc25.x86_64.rpm 
Last metadata expiration check: 1:07:49 ago on Wed Mar 15 17:12:09 2017.
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                              Arch                                                    Version                                                     Repository                                                     Size
==============================================================================================================================================================================================================================================
Installing:
 bar                                                  x86_64                                                  1-1.fc25                                                    @commandline                                                  6.0 k
 foo                                                  x86_64                                                  1-1.fc25                                                    @commandline                                                  6.0 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  2 Packages

Then version bump foo to 2, and make it require baz instead of bar. Then try and install that:

$ sudo dnf install ./RPMS/x86_64/foo-2-1.fc25.x86_64.rpm ./RPMS/x86_64/baz-1-1.fc25.x86_64.rpm                                                                                                                         
Last metadata expiration check: 1:08:16 ago on Wed Mar 15 17:12:09 2017.
Dependencies resolved.
==============================================================================================================================================================================================================================================
 Package                                              Arch                                                    Version                                                     Repository                                                     Size
==============================================================================================================================================================================================================================================
Installing:
 baz                                                  x86_64                                                  1-1.fc25                                                    @commandline                                                  6.0 k
Upgrading:
 foo                                                  x86_64                                                  2-1.fc25                                                    @commandline                                                  6.0 k

Transaction Summary
==============================================================================================================================================================================================================================================
Install  1 Package
Upgrade  1 Package

And to confirm, the package manager knows nothing requires bar:

$ rpm -q --whatrequires bar
no package requires bar
omajid commented 7 years ago

dotnet-sdk where the sdk can be substitued by devel or dev - whatever is the distributions convention. [Depends on dotnet]

I would prefer if the .NET Core project made a recommendation (of course, distros can ignore it if they want). I personally prefer sdk, over -dev/-devel since that's what https://www.microsoft.com/net/download/core calls it. Is there another name for it used here on github or on microsoft.com?

bleroy commented 7 years ago

Yes, sdk would be preferable as far as we're concerned. Let's go for that.

RheaAyase commented 7 years ago

Updated to reflect your preferences, I'm also inclined to it, was just wondering how strong are the dev/devel conventions... Do we have anything else to add to this, or can we write it up somewhere in a better .md file perhaps, as guidelines?

Packages:

fsateler commented 7 years ago

Not even if its dependency no longer exists? I thought that it would remove said dependency.

The case that I am referring to is the following:

-foo, version 1 depends on bar -foo, version 2 depends on baz

In this case the dependency on bar does not exist when foo, 2 is installed. Upgrading from foo, 1 to foo, 2 does not remove bar. Both Fedora and Debian have explicit steps to make a package (baz here) replace another by uninstalling that other package (bar here).

On debian and derived distros, this is true, but only if you use apt-get. If you use apt or aptitude, automatically installed packages will be removed if nothing else depends on them. This can be easily worked around by creating a virtual package dotnet-runtime, that all dotnet-major.minor packages Provide. dotnet could then Recommend this virtual package, which would cause apt to consider the package still needed. Mutatis mutandis the same for the -sdk variants if it is so desired.

gegenschall commented 7 years ago

Hey everyone,

I'm currently maintaining .NET Core on Arch Linux (as AUR packages) and want to chime in on this discussion. Thanks @omajid for notifying me about it.

First of all: Arch Linux is not an officially supported distribution by .NET Core, so I (and the maintainers before me) had to jump through some hoops to get it running. Specifically I haven't been able to build .NET Core from source because of the bootstrapping mechanism, but that's an entirely different topic. Currently the packages build the native components from source and replace the corresponding files from an official Ubuntu 16.04 build. (ugh...)

There are 4 packages that I maintain:

Package Purpose Current Version Dependencies
dotnet-sdk Contains the latest sdk 1.0.1-1 dotnet, dotnet-cli
dotnet Contains the latest shared framework 1.1.1-1
dotnet-lts Contains the latest LTS shared framework 1.0.4-1
dotnet-cli dotnet binary and libhostfxr 1:1.1.1-1 dotnet

Notes:

As far as I can see, this pretty much corresponds to what @RheaAyase wrote with the exception of dependencies which I'll adapt accordingly.

bleroy commented 7 years ago

Thanks all, this is very useful. We'll certainly add documentation with recommendations on package naming based on your input.

@gegenschall can you give more details about some scenarios where someone would install dotnet-cli and not dotnet-sdk? I understand you said this is something you inherited, but is there a compelling reason to keep it around?

RheaAyase commented 7 years ago

@gegenschall: Yep your scheme is close enough to the idea, with a small difference - that we would prefer dotnet.major.minor instead of dotnet-lts because... which lts? :p

gegenschall commented 7 years ago

@bleroy If you were to deploy a server you wouldn't need the SDK, right? But this is a case you most likely wouldn't do with Arch + .NET Core as its officially unsupported, you're right. Apart from that I guess I would annoy several people if I remove or rename dotnet-cli...

@RheaAyase You are absolutely right! I'll upload dotnet-major.minor packages, point dotnet to the latest, and ask for a deletion of dotnet-lts.

bleroy commented 7 years ago

@gegenschall if you deploy to a server, you should only need the shared framework release, no?

RheaAyase commented 7 years ago

@bleroy that should depend on the host because that is what decides which version of framework to use for said application you want to run. You can have as many versions of framework as you wish.

bleroy commented 7 years ago

oooh, no, ok, sorry, so @gegenschall's dotnet-cli is just our dotnet-host... That was actually clearly stated...

bleroy commented 7 years ago

I put a draft of the recommendation doc here: https://github.com/bleroy/core-docs/blob/dotnet/build-from-source/docs/core/building/distribution-packaging.md

tmds commented 7 years ago

@bleroy Looks great.

dotnet is a latest runtime package? Since apps need a specific major.minor to work, I don't get why this package is useful. Can it be removed? Does that make sense?

Petermarcu commented 7 years ago

@rakeshsinghranchi @vivmishra to make sure they see this and follow along on the proposed changes.

RheaAyase commented 7 years ago

@tmds we've discussed this for a month, I don't see why would we not have the dotnet package with latest version. That should be the default thing when someone installs sdk, framework, etc. If they want specific version, there are specific versions.

RheaAyase commented 7 years ago

Also @bleroy this document does not reflect what was discussed here at all.

tmds commented 7 years ago

@tmds we've discussed this for a month, I don't see why would we not have the dotnet package with latest version.

@RheaAyase In the discussion I miss the rationale for the package. I am asking why this package is useful.

RheaAyase commented 7 years ago

dotnet-sdk-[major].[minor]: the SDK with the specified version. The version specified is the highest included version of included shared frameworks. This means users can easily relate an SDK to a shared framework.

This should be in the "Optional additional packages" list, and the version should reflect the sdk, the tooling, not the framework, as it doesn't really care about the framework version... And whoever installs it, will care about the specific version of the tooling, especially right now in the transition period with project.json and csproj.

We did not discuss version naming of this kind of optional sdk package at all, and frankly i've no idea how could we name it because major.minor doesn't cut it with "preview" being the difference between json and csproj. Hence I suggest not packaging specific version at all, as in the future it should not matter - there should not be more major breaking changes like this one was.

dotnet-sdk: the latest SDK. dependencies: dotnet-host, one or more dotnet-[major].[minor] (one of those is used by the SDK code itself, the others are here for users to build and run against).

I Suggest:

This is because of already mentioned "I want a fresh install of sdk and stuff." If the package depends on the specific version then the user will not get the latest version when installing the sdk, which might not be updated as often as the framework. @tmds it serves merely as dependency here, for someone installing the sdk (or just installing dotnet as the latest thing they want to install.)

Edit: I, as user, would want to install dotnet right now, and I would want to install an sdk that depends on it.

bleroy commented 7 years ago

Some context about versioning... We've observed massive confusion about the version of packages being the version of the SDK, which is what we've been doing with early 1.x versions. We had to change that: almost nobody understands the distinction between SDK and runtime versions, and everybody is interested in the version of .NET Core that will effectively run their code, not the version of the CLI. Starting with 2.0, the confusion over that will be reduced as we present a unique and consistent version number for all outgoing things.

Versioned SDK packages are useful because we are allowing side-by-side SDKs, and users can specify in global.json what version of the SDK should be used for a particular app.

I agree that the mainline scenario is that you get the latest everything.

RheaAyase commented 7 years ago

Some context about versioning... We've observed massive confusion about the version of packages being the version of the SDK, which is what we've been doing with early 1.x versions. We had to change that:

Oh did I miss the discussion about that change?

everybody is interested in the version of .NET Core that will effectively run their code

What if I'm not, I'm interested in project.json vs csproj. What if I want latest framework and the sdk of my choice. I mean that's the only instance when I would opt into installing "dotnet-sdk-[major].[minor]" over dotnet-sdk. And if I were in the need of the other thing, being specific framework version, only then I would install both the dotnet-sdk and dotnet-major.minor

and users can specify in global.json what version of the SDK should be used for a particular app.

...what version of... sdk package (framework) or sdk tooling? Are they supposed to be the same version number starting 2.0 or how am I supposed to understand this?

I think that it is relevant to re-open the whole discussion maybe. Just imagining the sdk having dotnet dependency and it changing on someone overnight, their application would stop working without any warning, since they target specific version, right? In the ideal scenario the dotnet package would install the new version, but also keep the old one as dotnet-major.minor which would solve this problem, regardless of whether they wanted sdk with latest framework when they started, or just the latest framework.
This won't work though, so what other options do we have? Dropping the whole latest dotnet package and going with just major.minor? Can we still have install dotnet spit out the latest dotnet-major.minor? That would work well I guess. What would be the sdk dependent on, the latest major.minor where with the new release we would have to update the sdk package as well, just to fix that? I don't think that it would be correct to keep the sdk dep on the old version...
(I don't see the need for dotnet-sdk-major.minor where the version is the underlying framework in either case...)

Edit: To expand on that idea, having the sdk updated dep. with new framework release, it would not uninstall the old framework, it would only add the new framework package when the user runs update which seems okay.

bleroy commented 7 years ago

I'm sorry if I gave the impression that discussions about packaging recommendations happened outside of this thread.

What if I want latest framework and the sdk of my choice?

Then you would install the specific dotnet-sdk-major.minor sdk version you want, and then dotnet. You would even be able to target different SDK versions per app.

Installing dotnet-sdk and then a specific dotnet-major.minor is also an option. Both are valid choices.

In all cases, the disk layout is versioned, so none of this should be breaking or stepping on each other's toes (except for dotnet.exe of course).

what version of... sdk package (framework) or sdk tooling? Are they supposed to be the same version number starting 2.0 or how am I supposed to understand this?

Yes, SDK and shared framework will rev together from now on, with consistent version numbers. As a consequence, "latest" runtime and sdk packages will need to be updated together (at least major and minor versions), even in cases where the SDK has no real code change.

I updated the doc to cover preview versions as well as side-by-side update explanations in the disk layout section.

I hope this helps. Please do let me know if I missed something else that should be reflected in the recommendation doc. It is very far from being set in stone: it is still in my own fork of the docs, and I haven't submitted a PR yet.

RheaAyase commented 7 years ago

@tmds what about the "dotnet package is kinda useless?" Any thoughts?

I brought it up in a meeting/call last night with the following scenario as a problem:

The user installs dotnet-sdk depending on dotnet (or just the dotnet, works both ways) and they start working on their project or deploy it somewhere using this latest framework. This project is of course depending on this specific major.minor, but when they update all packages, their shared framework version will change, and the application will no longer work.

Solution: Remove the dotnet package completely, while the sdk would dep on the latest available dotnet-major.minor, if updated, it will simply install the new package for new major.minor and still keep the old one. So in other words, the sdk would always keep the original major.minor framework packages, and the latest.

Questions: Would this make sense at all? Maybe we should remove the dotnet-sdk package as well and simply force people to explicitly choose what do they want?

cc @gegenschall what's Arch thinking? ;P

tmds commented 7 years ago

Solution: Remove the dotnet package completely, while the sdk would dep on the latest available dotnet-major.minor, if updated, it will simply install the new package for new major.minor and still keep the old one. So in other words, the sdk would always keep the original major.minor framework packages, and the latest.

:+1:

Questions: Would this make sense at all? Maybe we should remove the dotnet-sdk package as well and simply force people to explicitly choose what do they want?

I'd keep the sdk meta package as it provides an easy way to install the sdk without knowing which is the latest. The sdk should have a somewhat consistent user experience across versions.

RheaAyase commented 7 years ago

@bleroy @ellismg any thoughts about that from your side of things?

Solution: Remove the dotnet package completely, while the sdk would dep on the latest available dotnet-major.minor, if updated, it will simply install the new package for new major.minor and still keep the old one. So in other words, the dotnet-sdk would always keep the original major.minor framework packages, and the latest.

bleroy commented 7 years ago

I like the certainty that comes with fixing what dotnet-major.minor the dotnet-sdk package references. But what if the dotnet package was itself an empty package that just references the latest dotnet-major.minor? Updating would get the new latest, but wouldn't remove the old one, and this way we can be both side-by-side and have a simple way to get the latest. Wouldn't that work?

tmds commented 7 years ago

ut what if the dotnet package was itself an empty package that just references the latest dotnet-major.minor? Updating would get the new latest, but wouldn't remove the old one, and this way we can be both side-by-side and have a simple way to get the latest. Wouldn't that work?

That would work, but it is not adding anything useful. Apps require a specific dotnet version. The sdk meta package points to a specific sdk. The specific sdk points to specific dotnet version. So, it's not useful as a dependency.

As part of an update, you want to get the fixes for the versions you have installed (patch update). Updating to the latest (major/minor update), installs a version of dotnet you are not using.

bleroy commented 7 years ago

Yes, apps require a specific version, but that's just one scenario. There is value in being able to install "dotnet", and having updates that add the latest to the system. I agree it's not useful as a dependency, but it's useful by itself. I'm fine with making it optional, however.

tmds commented 7 years ago

it's useful by itself

How did you reach that conclusion? I miss the rationale for the package when reading this thread.

bleroy commented 7 years ago

"There is value in being able to install "dotnet", and having updates that add the latest to the system". Makes it easier to maintain a runtime environment in a back compat way. "dotnet" without version number is also more discoverable. But, again, if you're unconvinced, and would rather not ship a dotnet package, I can move that one to be optional.

RheaAyase commented 7 years ago

My opinion from my two points of view:

As a user I see no reason to have always the latest major.minor on my server. When I update my server-side application's code, to use newer version, then I'll also manually update the framework - always keeping but one installed to save HDD space for economical reasons (I don't have freebie Azure VM)

As a developer I would like my SDK to be the latest, and my framework to be -all of them- because I will probably work with more than one at a time with different projects.

Based on these two points of view, I see no point in having the dotnet package, however, the -sdk that will add new frameworks (without removing anything) would be nice.

bleroy commented 7 years ago

Yes, those are two important scenarios.

There's also the scenario where as a hoster, I want to keep my servers ready to welcome apps written for all versions of .NET Core. This can be achieved in two ways: I can manually install versioned shared frameworks as they come out, or I can install dotnet once and update it to get new versions side by side.

Another is the one where as a a new user, I can easily discover the dotnet and dotnet-sdk packages and start experimenting.

I do agree however that dotnet is the least necessary of all these packages, and could be made optional. I'll go ahead and make that change in the recommendations.

tmds commented 7 years ago

I like the idea of a dotnet package as a user-experience but it doesn't work well:

Now, what if we rename the dotnet-host package to dotnet? When installed, the user gets the dotnetexecutable which can tell him many interesting things like:

Thoughts?

bleroy commented 7 years ago

That's an interesting idea, especially as the host consists of a "dotnet" executable, but the host remains pretty useless on its own, which is why the package so far has been considered as a dependency only, not as a standalone thing.

It also comes with strong constraints on how much it can do without an SDK, and what help messages we can put in there (and how they can be localized). We also can't make assumptions about the name of the dotnet-sdk package, since those are only recommendations, or even the acquisition mechanism, so we'd have to adopt language that is more vague and as such less useful than what you suggest.

If the new user experience is what we want to aim dotnet at, then making it a synonym of dotnet-sdk would probably be more useful than pointing it to the host. Another issue with aliasing it to either SDK or host however is that there's a symmetry between versioned and unversioned SDK packages that would be lost between the dotnet and dotnet-major.minor packages, which would cause confusion.

In the end, we seem to be stuck between the idea of having a dotnet package at all, and the possibility to make it useful. At this point, you've convinced me that it would actually be better to not have it at all, since I don't see what experience we could put behind it that would match any reasonable expectations.

RheaAyase commented 7 years ago

There's also the scenario where as a hoster, I want to keep my servers ready to welcome apps written for all versions of .NET Core. This can be achieved in two ways: I can manually install versioned shared frameworks as they come out, or I can install dotnet once and update it to get new versions side by side.

True, I haven't thought of that, however in that case they could just install the sdk which will effectively do the same thing. The only in-production actual use of this that comes to my mind is an university, where every student has an account on some powerful server to run their heavy calculations or projects, etc... They won't cry for the extra 100MBs from having an sdk instead of just the framework...

If the new user experience is what we want to aim dotnet at, then making it a synonym of dotnet-sdk would probably be more useful than pointing it to the host.

I think so, and I would be inclined to this solution, if we are to have some dotnet package...

Can I assume then, that these are our current go-to names or is there anything else?

bleroy commented 7 years ago

Yes.

I'll remove the dotnet package from the doc entirely then.

RheaAyase commented 7 years ago

//bump

Is this published already?