search

dotnet / runtime

.NET is a cross-platform runtime for cloud, mobile, desktop, and IoT apps.
https://docs.microsoft.com/dotnet/core/
MIT License
15.29k stars 4.74k forks source link

Support system-wide installation on unix-like platforms #5068

Closed snuxoll closed 4 years ago

snuxoll commented 8 years ago

Currently CoreCLR and CoreFX do not provide any facility for system-wide installation on unix-like platforms, with the current build framework only outputting binaries with no installation mechanism. This is extremely important to get .Net Core packaged and included in Linux distribution repositories. In addition, the lack of soname and so versioning as well as the requirement to pull down NuGet packages during build prevents certain distributions (especially Fedora and the EPEL repository where I am personally interested in getting .Net Core packaged for) from providing packages (I can work around the lack of so vesions downstream if I have to, but it is a matter of last resort in the Fedora packaging policy).

In essence:

I'm happy to work on contributing solutions to these problems, as I personally would like to see .Net Core in the Fedora Software Collection - but I am filing an issue to discuss these requirements and get input.

jkotas commented 8 years ago

cc @piotrpMSFT

myungjoo commented 8 years ago

I've been building .net core with rpmbuild and .spec files general enough for non Ubuntu Linux systems and putting things (coreclr) into /usr/bin/dotnet/* and made symbolic link from /usr/bin/corerun to /usr/bin/dotnet/corerun as my initial draft. (at least, it works for x64 systems. I'm trying for armv7l now)

If we are going to make some dotnet installation standard for Linux systems, I'd like to know it beforehand and make my policy accordingly. For such policies, let's consider the rules how to install (and their locations and search rules) class libraries or DLL files. We are not going to put ALL .DLL files at the same directory with runtime (no subdirectories), are we? If we can have some general rules applicable to all Linux distros or Linux-based platforms, that'd be great.

ayende commented 8 years ago

Please let us know if you can get it working on ARM, my last attempt ended with invalid instructions on Linux.

Hibernating Rhinos Ltd

Oren Eini* l CEO l *Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

On Thu, Feb 4, 2016 at 6:27 PM, MyungJoo Ham notifications@github.com wrote:

I've been building .net core with rpmbuild and .spec files general enough for non Ubuntu Linux systems and putting things (coreclr) into /usr/bin/dotnet/* and made symbolic link from /usr/bin/corerun to /usr/bin/dotnet/corerun as my initial draft. (at least, it works for x64 systems. I'm trying for armv7l now)

If we are going to make some dotnet installation standard for Linux systems, I'd like to know it beforehand and make my policy accordingly. For such policies, let's consider the rules how to install (and their locations and search rules) class libraries or DLL files. We are not going to put ALL .DLL files at the same directory with runtime (no subdirectories), are we? If we can have some general rules applicable to all Linux distros or Linux-based platforms, that'd be great.

— Reply to this email directly or view it on GitHub https://github.com/dotnet/coreclr/issues/3025#issuecomment-179930032.

myungjoo commented 8 years ago

@ayende If you are working on an ARMHF-Ubuntu14.04, it should work without any serious work unless there are serious regressions (I just don't know on this) in coreclr/corefx. We are experiencing some troubles only because we are not ARMHF and we are not using Ubuntu (our own "baremetal" Linux targeting embedded systems.

The cross build environment supplied is for ARMHF/Ubuntu-14.04. "Ubuntu part" is not that hard to resolve as long as you can identify which libraries are used. (like icu-52 is used with "rename" option. ours uses icu-5x without "rename" option. so I've installed two instances of icu. and some other libraries have some minor issues as well: lttng-ust, systemtap, and so on). More complicated part is on ARMHF; the current "legacy" JIT does not emit ARM softfloat. We have some workarounds to try for now; but I'm not sure if such will be sufficient for now. (but, at least we are not experiencing "invalid instructions".)

snuxoll commented 8 years ago

@myungjoo Great point.Considering libcoreclr.so and corerun/coreconsole should be put in separate locations (/usr/lib/ and /usr/bin), which would require corerun to need a shim script to pass the correct -c argument, I propose the following.

shahid-pk commented 8 years ago

one thing should be clear .net core (coreclr+corefx) is designed to be application local and not system wide. This decision has been taken long ago [https://blogs.msdn.microsoft.com/dotnet/2014/12/04/introducing-net-core/] and i don't think right now that could change or should change so late in the release cycle. The new way is to use a tool for example dotnet/cli which will acquire the run time along with only the libraries that you require and when you deploy you deploy the run time along your application. Not using any tool and doing this all by hand is also possible but not recommended. The post that i have provided link above explain why this decision was taken.

for packaging on linux distros senario you will have to package the tool (dotnet/cli) and yes i i agree that currently (dotnet/cli) tool does'nt come with corefx libs it should come with them too so that hitting the internet is not required for atleast Microsoft provided libraries on environments that the op mentioned.

Disclaimer : I am not a Microsoft employee and everything that i said is what i have learnt by following this project from the start.

TheRealPiotrP commented 8 years ago

Yea, today we're building the CLI just as if it were any other CoreFx app: we include our own dependencies in a flat directory and do not enable sharing with the applications CLI builds. There has been some effort to make this more efficient, as suggested on this issue, by making the CLI dependencies sharable with apps. Unfortunately, the packages with sharable dependencies will likely be quite a bit larger than what's in box today.

If folks want to participate in the conversation of improving this CLI experience I encourage creating an issue in https://github.com/dotnet/cli.

snuxoll commented 8 years ago

@piotrpMSFT I am not talking about dotnet-cli at all here, only the CLR and CoreFX. If you choose to bundle CoreCLR and CoreFX with your application that is your prerogative, I can modify it to use system installed libraries.

As it stands, not supporting any method of installing CoreCLR and CoreFX system wide is going to prevent inclusion of any application that use it in the software collection of any popular Linux distribution where static-linking or vendoring dependencies in a strict no-go for security reasons (every single package that vendors these dependencies requires they be repackaged with newer versions of a library instead of just updating the single .so installed in /usr/lib).

Ultimately, .Net Core can be used without dotnet-cli, and my goal is to focus on that usage - I'm not buying into the fad of bundling your application with all it's dependencies and neither is the Fedora project.

.Net Core is useful for a lot more than internal or SaaS deployed applications where such behavior is deemed acceptable, we have plenty of things that use Mono in the Fedora Software Collection since it does support this usage, and I would like to see the same of .Net Core.

shahid-pk commented 8 years ago

@snuxoll i think a system wide scenario could be supported by the community if interested.We know that it won't be supported by Microsoft to be frank [Although someone from Microsoft should confirm] , personally i don't have a problem with that i like the new idea but i think that their should be options. This project is completely open source and one great thing about open source is that you can make it work whatever way you like it. So if a system wide installation is offered by a third party that is free and open and does'nt conflict with the app local model [in a sense that user can choose to use one or the other] i will support it :smile:

snuxoll commented 8 years ago

@shahid-pk Absolutely, I see no reason why the app-local model cannot continue to exist, as there certainly is a bunch of people who want it (and it certainly makes deployment of newer versions easy where you don't care to rely on your distribution repositories). All I seek is the option to allow a system-wide installation so that those that DO prefer to use their standard package manager (myself included, obviously) can do so without needing to bundle everything with their applications.

Petermarcu commented 8 years ago

This is a totally valid request and is something we are exploring actively.

OtherCrashOverride commented 8 years ago

I think its important to clarify the part that is NOT being said and that is this is a request to make CoreCLR a replacement for Mono.

apt-get install coreclr
coreclr myapp.exe

This mean adding back in things like the global assembly cache (GAC). Without a GAC, you are right back in the situation of having multiple copies of private .dlls for every non core library. With CoreCLR being "core", its of very limited use without additional libraries. Every app will need to provide its own copy of the extra libraries even if multiple apps share the same dependencies.

[edit] The other part that is not being said is there is a desire to remove Nuget and replace it with the distribution specific packaging method. The reason for this is to provide a source that is curated, trusted and accountable. This also requires a GAC.

snuxoll commented 8 years ago

@OtherCrashOverride - Java doesn't have a GAC yet we support multiple JAR versions quite nicely by using shim scripts that modify the CLASSPATH. Many distributions don't even support installing multiple versions of libraries in the first place (debian and derivatives being notable exceptions) - I don't see a reason this needs to be handled by the runtime.

OtherCrashOverride commented 8 years ago

I don't see a reason this needs to be handled by the runtime.

http://branchandbound.net/blog/java/2015/09/java-module-system-first-look/

When you assemble and run an application consisting of different jar files (read: every non-trivial application), you put them on the classpath. And then hope for the best. Because there's no way to tell if you put everything on the classpath that your application needs. Or whether you inadvertently put the same classes (in different jars) on the classpath. Classpath-hell (analogous to DLL-hell) is a real thing. This leads to bad situations rearing their ugly head at runtime. Also, the knowledge that a class was ever in a jar file is lost at runtime. The JRE just sees one big collection of classes. But jars need other jars. It's just not encoded explicitly in any form of meta-data at the moment.

snuxoll commented 8 years ago

@OtherCrashOverride Classpath/DLL hell is caused by poor package management, an issue that is not present on the intended target systems. Supporting something similar to soname's used my ldd would be nice to allow compat libraries to exist, but that's the furthest I'd go with extending the runtime. The GAC is designed to solve a problem that exists within Windows, and I don't see .Net Core replacing the full .Net runtime on Windows platforms for this use-case.

OtherCrashOverride commented 8 years ago

Classpath/DLL hell is caused by poor package management, an issue that is not present on the intended target systems.

If the target system is GNU/Linux based, "dll hell" is very much alive, well, and thriving there. Mesa proves this daily.

The GAC is designed to solve a problem that exists within Windows, and I don't see .Net Core replacing the full .Net runtime on Windows platforms for this use-case.

There is no correlation between CoreCLR using a GAC and it replacing .Net runtime on Windows as proven by Mono on Windows. The GAC as implemented by Mono is a simple file structure (edit: meaning structuring of files, not a structure within a file). Nothing scary or fearful about it. When Mono is embedded in an application, the GAC file structure exists in the application folder and is local. This does not preclude an embedded app from using the system GAC in addition to its "local" GAC. This is a feature that should have been implemented in CoreCLR from the start. Again, all we are talking about here is a well defined file hierarchy in a known or discoverable location. The runtime's job is to traverse the hierarchy in a well defined manner to locate dependencies.

snuxoll commented 8 years ago

@OtherCrashOverride Makes sense. I could agree with that.

krytarowski commented 8 years ago

I just hit it on NetBSD.

$ tree ./bin/Product/NetBSD.x64.Debug/                                                                                                              
./bin/Product/NetBSD.x64.Debug/
├── Loader
│   └── NativeLibs
│       └── FromNativePaths_lib.so
├── System.Globalization.Native.so
├── bin
│   ├── hfa_nested_f32_native_cpp.so
│   ├── hfa_nested_f64_native_cpp.so
│   ├── hfa_simple_f32_native_cpp.so
│   ├── hfa_simple_f64_native_cpp.so
│   ├── jitstructtests_lib.so
│   ├── libMarshalBoolArrayDLL.so
│   ├── libStructABILib.so
│   ├── mirror.so
│   ├── native_i0c.so
│   ├── native_i0s.so
│   ├── native_i1c.so
│   ├── native_i1s.so
│   ├── native_i3c.so
│   ├── native_i3s.so
│   ├── native_i5c.so
│   ├── native_i5s.so
│   ├── native_i6c.so
│   ├── native_i6s.so
│   ├── native_i7c.so
│   ├── native_i7s.so
│   ├── native_i8c.so
│   └── native_i8s.so
├── coreconsole
├── corerun
├── crossgen
├── gcinfo
│   └── gcinfoencoder.cpp
├── ilasm
├── ildasm
├── inc
│   ├── cfi.h
│   ├── cor.h
│   ├── cordebuginfo.h
│   ├── corerror.h
│   ├── corhdr.h
│   ├── corinfo.h
│   ├── corjit.h
│   ├── corprof.h
│   ├── gcinfoencoder.h
│   ├── gcinfotypes.h
│   ├── opcode.def
│   └── openum.h
├── lib
│   ├── libcoreclrpal.a
│   ├── libcorguids.a
│   └── libpalrt.a
├── libcoreclr.so
├── libdbgshim.so
├── libmscordaccore.so
├── libmscordbi.so
├── libryujit.so
├── libsos.so
├── libsosplugin.so
└── sosdocsunix.txt

6 directories, 53 files

I need to research how these files work together to have an idea how to install them to the system.

Please add a standard installation target under desired --prefix.

krytarowski commented 8 years ago

I'm rejecting my proposal. I'm happy with the current installation method.

jkotas commented 8 years ago

This is being worked on in the cli repo https://github.com/dotnet/cli/blob/rel/1.0.0/Documentation/specs/canonical.md#acquisition

snuxoll commented 8 years ago

@jkotas I do not agree that dotnet-cli is the appropriate place to solve the usage of a shared runtime and libraries. I want to use core without needing to be tied at the hip to cli.

jkotas commented 8 years ago

@snuxoll We are doing the packaging work in the cli repo because of it fits well with how our engineering system is system. It does not mean that the shared runtime cannot exist without the full CLI. You will be able to install the shared runtime, without installing the full CLI.