Support for Internalizing/Merging external Dependencies

[avanderhoorn]

tl;dr

In a world where an ever growing number of packages are depending on other packages for "internal" functionality, the amount of unresolvable conflicts that occur between packages will increase. This specification is designed to help package authors to explicitly declare these internal dependencies so that the runtime or build process can allow multiple versions of the same package to run, and so that users aren't faced with unresolvable conflicts.

Definitions

Internal Dependency

An Internal dependency is defined as a dependency that a controlling package needs for its own internal functionality. To uses/consumers of the controlling package, they should not be affected or constrained by which version of a package the controlling package chooses to use. An internal dependency should not "leak" outside of the controlling package and should be able to operate in its own memory space (meaning if 2 versions of the same package are present, there should not be any conflict between the 2 packages).

Example Glimpse is a tool based package which users add to their system to gain insights on how their system is running. INTERNALY we are using JSON.Net to seralize data for our own internal consumption. The JSON.Net types are never leaked and the users are never aware that we use or what version we are running. To the user we might as well have written our own seralizer.

Public Dependency

A Public dependency is defined as a dependency that an authors package extends and/or exposes the dependent packages types. To uses/consumers of the authors package, they need to be aware of the version of a package the controlling package chooses to use as it adds/extends a package (at a specific version) that they are using. An public dependency most likely has types that "leak" outside of the authors package and should be able to operate in the same memory space (as it needs to interact with types and memory in that public dependency).

Example LinqKit is a set of extensions that is built on top of Entity Framework. It needs to publicly depend on Entity Framework as it is extending EF types/functionality and is dependent on the version of EF that the user is using.

Summary

When you are extending something, you don't internalize (as that obviously doesn't work) but if you depend on something just for internal use, Internal is used to avoid issues

Overall Problem

In a world were package authoring and consumption is becoming ever easier, it should be expected that the number of conflicts between packages will only increase. In this world there there are Public dependencies which MUST be resolved in order to function as expected and Internal dependencies which shouldn't effect the package version choices of someone using the internal dependences. Moving forward, the amount of packages that have truly internal dependencies will most likely increase and the pain that authors and consumers experience increase as a result.

Version conflicts will be a problem In my app, if I use 5 - 7 different packages for middleware functionality, its highly unlikely that all these will use the same version of JSON.Net or log4net. In fact it is more likely that they will want to use 5 - 7 different versions of JSON.Net or log4net. Expecting the user to resolve and test these conflicts is a high expectation and they mightn't even be able to resolve those conflicts (as they might all be using different incomparable versions). And asking package authors to change there versions wont really work either, as no change will work for everyone.

Internal dependencies are designed to address these issues and allow package authors to be explicit about there Internal dependencies.

Specification

Not leaking types When a package author defines a dependency as being Internal, none of the types from that Internal dependency should leak through the public API.

This limitation simplifies the problem space and reflects a constraint which most users internalizing dependencies via ILMerge (or other similar methods are used to) are used to. When types can't be leaked it means that we need to build abstractions to let users toggle given configurations or proxy given given providers, but that "limitation" also acts as a point of clear separation between the my code and the package that I (as a package author) take a dependency on (like JSON.net or log4net, etc). It also means that a package author can update internal dependencies without having to effecting their users.

Dependent packages operate in their own memory space When a package author defines a dependency as being Internal, any state or memory that the internal package sets, should not conflict with any other instance of that package that is running.

Depending on how the feature is implemented, this might come as a result, but if a package that I depend on has a static variable, I don't want this to conflict with the same package that the user may be depending on directly. This might also come as a side effect of not leaking types (if that is implemented by changing things to internal) or might have to be done by changing namespaces or some other method, but it would be expected that dependent packages operate in their own memory space.

Support in the platform When a package author defines a dependency as being Internal, it should be supported by the platform so that authors and consumers alike have a consistent and supported experience

Having support in the platform for this problem means that instead of everyone going off and trying to solve it in different ways, we are combining our collective intelligence to have a process that is provided out of the box. It also means that consumer of package who use this, have a common expectation of how things will work.

Guidlines and Expectations

When a author internalizes a package, it now "owns" that dependency If my package depends on JSON.Net and I declare that dependency as Internal, I am now taking responsibility for that code. For consumers of my package, they should no longer think of my package as depending on JSON.Net, but just my package. They probably won't even know that have a dependency on JSON.Net. As far as they are concerned, I might as well built my own JSON seralizer. This is consistent with the fact that Types aren't leaked.

Package author controlling the version I've seen it said that "what if the package user wants to change the version of JSON.Net/log4net that we are using in Glimpse". I struggle to understand what the requirement is here. If we use version 1.1 of log4net and a user try's to change it to 1.2, things will most likely not work correctly. When we currently ILMerge something in, that is us (the package author) taking responsibility for that dependent package. How can we make an guarantees about our code running correctly, if people have the ability to change the version of one of our dependencies without us having tested that. If a bug is found in a package that we depend on, it is up to us to rerelease our package or the user to get the source and rebuild it. Again trying to make simple rules. Semver is great, but it doesn't guarantee that expected APIs or behaviours wont change/break.

Don't have to support every use case, this isn't a silver bullet I think that its ok if support for this feature doesn't work in absolutely every use case. For instance, Castle requires that some types must be public and hence can't be fully Internalized, that is ok. If we end up supporting the use case where we can change namespaces and log4net is doing string comparison on its types, pressure will soon come from the community to change it. In the end, with power like this also comes great responsibility. Its up to the author to make sure things work.

Implementation

At this point, this specification isn't worried about how this could be implemented. This is intended to document a use case and state what the base expectations are, we can work on implementation details later.

Example

Current thinking is that package authors could express internal dependency via the project.json. This being the case, changes to the project.json could look as follows

"aspnet50": {
  "dependencies": {
    "EntityFramework": "6.1.2-beta1",
    "NLog": { "version": "1.0.0.505", "type": "internal" },
    "Newtonsoft.Json": { "version": "6.0.1", "type": "internal" }
  }
},

In the above example, the package has a Public dependency on Entity Framework, but Internally depends on NLog and JSON.Net for its internal logging and serialization.

---

Original Comment

Ideally/Long Term: Currently, dependencies can we setup for external packages:

"aspnet50": {
  "dependencies": {
    "Microsoft.Framework.Logging.Interfaces": "1.0.0-*"
  }
},

These dependencies can also be marked as build dependencies only:

"aspnet50": {
  "dependencies": {
    "Microsoft.Framework.Logging.Interfaces": { "version": "1.0.0-*", "type": "build" }
  }
},

It would be nice if one could mark the dependency as being "internal":

"aspnet50": {
  "dependencies": {
    "Microsoft.Framework.Logging.Interfaces": { "version": "1.0.0-*", "type": "internal" }
  }
},

In the end the definition might be different to the above, but the end result is that the dependency would be merged into the "main" dll.

Hack/Short Term In the mean time, it would be great if there was a support way for us to be able to do this (probably with a script) so that it works with the existing kpack process.

If the correct hook is already in place, can you please point me in the right direction for this "Hack/Short Term" solution.

This is an extension of a discussion that I had with @davidfowl on JabbR.

[damianh]

+1e9

Seriously one of the major pain points in .NET. As we enter the world of lots and lots of packages and, consequently, composable web applications (think RavenDB.Database, IdSrv3, Glimpse, all of my own middleware) who may use different versions of various frameworks entirely internally, we need a way to encapsulate these dependencies such that neither consumers nor the various components impact each other (intellisense, references, version conflicts, runtime exceptions, etc).

The current solution is to either use ILMerge, ILRepack, Fody Costura or similar. These are sub-optimal workarounds for numerous reasons. The real solution is platform / runtime / compilation support.

[shiftkey]

cc @davkean

[davidfowl]

What's wrong with ILMerge? I'd like to hear some of the pros/cons.

[avanderhoorn]

Do you mean why do we want it in the box?

[davidfowl]

Nope I'm wondering if there's something fundamentally wrong with it (besides the fact that it confuses mono).

[damianh]

Couple of things to start:

1. It's hard to use, hard to get right.
2. Prone to errors & mistakes (i.e. /internalize should fail if one of the merged types is available via target asms)
3. Usually done as a post build step, I dislike that my tests cover the pre-merged asm, and not the one that is shipped.
4. Is it even officially supported?
5. Friction friction friction.

[damianh]

As an example of a commercial use case, we're composing our applications via a suite of components that expose a MidFunc. These component may be considered as middleware, they may be "big", they may be "small" (it doesn't matter). I am predicting that we're going to see a lot more of this sort of thing:

app.UseGlimpse();
app.UseIdentityServer();
app.Map("blog", blogApp => blogApp.UseBlogEngineDotNet());
app.Map("cms", cmsApp => cmsApp.UseDotNetAge());
app.UseSomethingHefty();
app.UseMyOtherThing();
app.UseCompanyPortal();

... if BlogEngineDotNet uses WebApi 2.x and DotNetAge use WebApi 3.x (or whatever) this app must still work. It must be easy for developers to create such components safely. ILMerge is not easy (enough).

[davidfowl]

And the assumption is that the merged components never leak into public API right?

[avanderhoorn]

Yep. Hence why I think internalizing the types works.

[damianh]

Yes, we (I am anyway) am specifically talking about internalizing, not just bundling.

[leastprivilege]

We also use internalizing in IdentityServer to handle the dependencies. It would be nice to have that feature in the box.

[avanderhoorn]

Another example of this really helping is for all the packages in the ecosystem that will popup that will depend on different versions of json.net. Internalization here will really help!

[davidfowl]

That's only if those packages don't expose JSON.NET though. For example, configuring the serializer settings etc. It would be interesting to understand how many projects would use this feature (I guess the ones that use ILMergr today)

[damianh]

Yep. For example, NEventStore is such a package that uses an internalized Json.Net and doesn't expose the settings. If the user wants to customize that, they can implement their own ISerializer.

https://github.com/search?utf8=%E2%9C%93&q=ilmerge&type=Code&ref=searchresults https://github.com/search?p=100&q=ilmerge+internalize&ref=searchresults&type=Code&utf8=%E2%9C%93

IMHO, more projects should be doing it.

For example, Hangfire before http://www.nuget.org/packages/HangFire.Core/1.0.2 and after I suggested ILMerging the internal dep stuff: http://www.nuget.org/packages/HangFire.Core/1.1.0

The point is, Hangfire didn't do the ILMerge initially up front. Why not?

[avanderhoorn]

We do the same thing in glimpse. We depend on several different packages that we internalize and if we need to let people config things, we provide abstractions for that (because we can't let the types leak out)

[leastprivilege]

Same here - we have our own CookieOptions, DataProtector etc abstractions that we translate to the internalized ones.

[davidfowl]

And you guys aren't worried about the bloat?

[damianh]

As in bytes on disk? No. Bigger assemblies held in-memory? Not particularly.

[damianh]

@avanderhoorn +1 on that approach.

[distantcam]

Fody Costura uses a different approach to internalizing. Instead it bundles the assemblies as embedded resources, and then hooks the appdomain when it tries to resolve an assembly and provides the correct assembly.

If you're already controlling the appdomain as part of vNext, this might be a viable option.

The pros of this approach is that the embedded assemblies keep their own identity, including strong naming. Also you can include assemblies that are obfusctated, or otherwise difficult to ILMerge. Costura also has handles for native and mixed mode assemblies.

And lastly, using ILMerge is problematic when dealing with licenses like the GPL, as you're effectively merging all the source together. And there are licenses that restrict decompilation, which also could not use ILMerge.

[damianh]

@distantcam Hi, can you clarify something for me? Two components, A & B, both using Costura, both have embedded different and non-compatible visions of another component, say JSON.NET v5 and v6. Then we have an app C that references A & B. At runtime, which version of JSON.NET is loaded into the appdomain?

[distantcam]

@damianh Well, using Costura for libraries is not technically supported, precisely for this reason. :/ But I know people use it that way anyway.

The AppDomain assembly resolution is a first answer is the winner approach. So in the case you describe, which ever Costura hook from A & B that finishes first will be the one that's provided.

Costura was designed originally to bundle everything into a single exe to ease deployment. It was not designed for libraries.

[avanderhoorn]

Ok so I'm wondering, @davidfowl mentioned that he thought there was some ways that could do this given the new vNext world. @davidfowl what do you think might be possible here?

[damianh]

@distantcam I suspected such, thanks for the response. Pinging @ayende, you'll need to re-evaluate using Fody Costura for RavenDB.Database and embedded scenarios. Someone using it with a different version of WebAPI (or the other embedded dlls) are going to run into interesting (and probably non-deterministic) runtime issues.

[ayende]

I think that being able to hide what we depend on is important, especially incompatible stuff. IIRC, you can provide multiple versions of the same assembly if it is different version via AssemblyResolve

[distantcam]

@ayende Unfortunately not. AssemblyResolve is only used when the AppDomain cannot already find the assembly. So once the assembly is in the AppDomain that's the one that's used.

[dennisdoomen]

@ayende, @damianh, that's exactly what we ran into this week. Using RavenDB 3.0 in an ASP.NET web site that uses WebAPI 1 causes all kinds of really weird and annoying assembly loading conflicts.

[davkean]

First thing, I think really thing we should be working with and filing bugs/shaming projects that are breaking compatibility left, right and center. If you're popular enough, whether you like it not, you have to be in the compat business. Internalizing/ilmerging also doesn't help when you need to pass types across libraries and agree on their location (thing types like Stream, IServiceLocator, etc). Internally, we call these "exchange types" - any assembly that contains these, must be be compatible across versions.

With the new changes coming with NuGet, I'm wondering if you really need to internalize/ilmerge. NuGet's considering introducing a concept of private dependencies in v3, where these are not automatically passed to the compiler but are still deployed. After of which, you just need to figure out how to side-by-side load them. There's a couple of options:

1. Make use of LoadFile[1]. This gives every assembly it's own universe (AssemblyResolve is called back for every reference), and doesn't require you to put things in the GAC to enable side-by-side loading. However, it would be complex to figure out if a reference is needed just for implementation details or is actually something that needs to be unified and shared across libraries.
2. In ASP.NET vNext on top of CoreCLR, we have introduced a new concept called AssemblyLoadContext. This gives a similar result to LoadFile, where you can give each assembly it's own universe.

Unfortunately, neither 1 or 2 will work anywhere other than the server or full .NET Framework. Certain environments (ClickOnce, Silverlight, Phone, Store, .NET Native and probably Android/iOS) have restrictions where there cannot be duplicate assemblies in an app package and you are unable to side-by-side load.

The third option could be to simply rename automatically (using a metadata reader/writer) any private dependencies and then rewrote callers to pick up the new name.

For example given the following simple graph

Library1 -> System.Runtime (4.0) [Public]
         -> Newtonsoft.Json (3.0) [Private]

Library2 -> System.Runtime (4.0) [Public]
         -> Newtonsoft.Json (2.0) [Private]

We (ecosystem) introduce something that rewrites just before deployment to:

Library1 -> System.Runtime (4.0) [Public]
         -> Newtonsoft.Json_Library1 (3.0) [Private]

Library2 -> System.Runtime (4.0) [Public]
         -> Newtonsoft.Json_Library2 (2.0) [Private]

We make this general purpose so that it runs and works everywhere on all project types.

We should note however, that this doesn't fix the situation where Newtonsoft.Json is exchanged between Library1 and Library2. There's no way to make that work other than push on these popular libraries to be compatible between versions (not picking on James here, just using him as theoretical example).

We could also look at using a combination of options, 1 & 2 for the server and 3 for the client.

[1] If you use LoadFile, please call AppDomain.ApplyPolicy on the resulting name in AssemblyResolve, otherwise, policy (binding redirects, publisher policy unification, portable) will not be applied.

[ayende]

I like the idea of renaming dependecies.

Hibernating Rhinos Ltd

Oren Eini* l CEO l *Mobile: + 972-52-548-6969

Office: +972-4-622-7811 l Fax: +972-153-4-622-7811

On Fri, Nov 7, 2014 at 8:49 PM, David Kean notifications@github.com wrote:

First thing, I think really thing we should be working with and filing bugs/shaming projects that are breaking compatibility left, right and center. If you're popular enough, whether you like it not, you have to be in the compat business. Internalizing/ilmerging also doesn't help when you need to pass types across libraries and agree on their location (thing types like Stream, IServiceLocator, etc). Internally, we call these "exchange types" - any assembly that contains these, must be be compatible across versions.

With the new changes coming with NuGet, I'm wondering if you really need to internalize/ilmerge. NuGet's considering introducing a concept of private dependencies in v3, where these are not automatically passed to the compiler but are still deployed. After of which, you just need to figure out how to side-by-side load them. There's a couple of options:

1.

Make use of LoadFile[1]. This gives every assembly it's own universe (AssemblyResolve is called back for every reference), and doesn't require you to put things in the GAC to enable side-by-side loading. However, it would be complex to figure out if a reference is needed just for implementation details or is actually something that needs to be unified and shared across libraries. 2.

In ASP.NET vNext on top of CoreCLR, we have introduced a new concept called AssemblyLoadContext. This gives a similar result to LoadFile, where you can give each assembly it's own universe.

Unfortunately, neither 1 or 2 will work anywhere other than the server or full .NET Framework. Certain environments (ClickOnce, Silverlight, Phone, Store, .NET Native and probably Android/iOS) have restrictions where there cannot be duplicate assemblies in an app package and you are unable to side-by-side load.

The third option could be to simply renamed automatically (using a metadata reader/writer) any private dependencies and then rewrote callers to pick up the new name.

For example given the following simple graph

Library1 -> System.Runtime (4.0) [Public] -> Newtonsoft.Json (3.0) [Private]

Library2 -> System.Runtime (4.0) [Public] -> Newtonsoft.Json (2.0) [Private]

We (ecosystem) introduce something that rewrites just before deployment to:

Library1 -> System.Runtime (4.0) [Public] -> Newtonsoft.Json_Library1 (3.0) [Private]

Library2 -> System.Runtime (4.0) [Public] -> Newtonsoft.Json_Library2 (2.0) [Private]

We make this general purpose so that it runs and works everywhere on all project types.

We should note however, that this doesn't fix the situation where Newtonsoft.Json is exchanged between Library1 and Library2. There's no way to make that work other than push on these popular libraries to be compatible between versions (not picking on James here, just using him as theoretical example).

We could also look at using a combination of options, 1 & 2 for the server and 3 for the client.

[1] If you use LoadFile, please call AppDomain.ApplyPolicy on the resulting name in AssemblyResolve, otherwise, policy (binding redirects, publisher policy unification, portable) will not be applied.

— Reply to this email directly or view it on GitHub https://github.com/aspnet/KRuntime/issues/819#issuecomment-62201571.

[davidfowl]

@davkean Would the rename also internalize publics?

[avanderhoorn]

I imagine it would have to be or at the very easy the default option. When we take on a dependency like that we are internalizing it not only because we don't want it to cause conflicts but also because we don't want people playing with those types and we want the freedom to version/replace those dependencies without fear of breaking people. If we want people to be able to configure or work with elements of the dependent library, if up to us (the author) to provide the right abstractions.

On Saturday, November 8, 2014, David Fowler notifications@github.com wrote:

@davkean https://github.com/davkean Would the rename also internalize publics?

— Reply to this email directly or view it on GitHub https://github.com/aspnet/KRuntime/issues/819#issuecomment-62251614.

[adamralph]

Note that internalization doesn't always 'just work'. E.g. in FakeItEasy we have to exclude the following types from internalization otherwise Castle.Core blows up at runtime https://github.com/FakeItEasy/FakeItEasy/blob/master/Source/ILMerge.Internalize.Exclude.txt. Of course, Castle.Core is doing some pretty funky stuff with its DynamicProxy but it's worth being aware that internalization isn't a silver bullet for the general case.

[avanderhoorn]

Totally agree, we have the same exact thing for Castle.Core in Glimpse. I think the point here is that if we can come up with something that works for everything great! But if Castle.Core needs to be handled manually and we get out of the box support for everything else, I can live with that. With powerful tools comes responsibility and its up for the dev in the end to use the right tool for the right job.

[JamesNK]

My experience is most issues with dependencies are either caused by strong naming + assembly binding or by people putting exact or upper limits on dependency versions in NuGet.

Assembly binding is no longer an issue in vnext. If something is done to give users an option to override NuGet complaining about incompatible version numbers then dependencies should be much easier.

[NTaylorMullen]

Talked with @avanderhoorn in person. It's a tough problem. In vNext we're preaching the thought of app separation by opting in to what you want via Startup.cs (whatever you want, just add it to your Startup.cs and its yours). This is fantastic and works for the majority case but can be frustrating for package authors who are depending on third party or MS packages who aren't in their loop.

Without some sort of solution to this I do see us leading authors down the wrong path. Ex:

UserLib => Razor 3.0 (no tag helpers)

Glimpse => Would like to depend on Razor 4.0 (tag helpers)

Glimpse wouldn't be able to use TagHelpers in their stack and would be forced to update their package anytime Razor went up a major version (assuming breaking changes). I don't think making a type: "internal" piece is the right solution but maybe making that portion of project.json easily extensible?

[davidfowl]

Making it extensible sounds worse. I think it's time we do some case studies on other ecosystems that have these problems. This isn't unique to .NET. Does anybody have any experience with other ecosystems and internalizing dependencies?

Incompatibilities are incompatibilities. We give you more control over dependencies in vNext but there's no silver bullet that will make everything work.

[avanderhoorn]

NPM is the first one that comes to mind, but its not relevant given the way it handles dependencies.

[leastprivilege]

Well C/C++

It is really nice that IdentityServer is only a single nuget and we can encapsulate all our dependencies. And I hear this feedback all the time from customers as well.

https://github.com/thinktecture/Thinktecture.IdentityServer.v3.Samples/blob/master/source/SelfHost%20(Minimal)/SelfHost/packages.config

[davidfowl]

What happens if one of your components needs to be serviced after you've decided to merge/internalize them? Is that just a problem with your library and a user is no unable to work around the potential issue? You need to release a new version of your library?

In C/C++ the decision to statically/dynamically link is usually something you think about and I think it makes more sense for client apps tbh not sure about libraries.

[leastprivilege]

If a library needs to be serviced we have to update the whole package - that is true. But in most cases even just a service releases of a library you depend on requires you to revisit/test your own code.

[leastprivilege]

But yeah - there are certainly two angles to it - the dependencies (and their conflicts) and internalizing.

For us the main motivation was to simplify dependencies (see the JWT handler v3 vs v4 debacle) - the single nuget was a side effect of that - but it is handy.

[damianh]

Yep, need to ship an updated version of my lib. People "servicing" one of your dependencies on your behalf causes problems too - you've shipped have tested with version x but people start using it with version y. Even with semver and the best intention in the world, version y can break my lib, I won't even know it and I get in-the-wild-runtime issues.

Java has static linking.

[davidfowl]

@damianh Can you point to the java static linking support? Which component in the system does it?

[damianh]

Solution to having to service stuff quickly is continuous integration + nuget + CI feeds etc. When .NET was originally designed ~2000/2001 these facilities weren't wildly available (or were complicated / hard).

Also from a "bloat" perspective, typical computer memory & hard drive sizes were ~128MB & ~60GB respectively. Times have changed.

[davidfowl]

We should just IL Merge .NET into a single blob called DotNet.dll :smile:

[damianh]

lol

Edit: don't we already have one, mscorelib.dll?

[damianh]

This is what I could find on java linking http://njbartlett.name/2014/05/26/static-linking.html Seems they have similar problems. (I've very little knowledge of Java, so .... pinch o'salt)

[adamralph]

I really don't mind servicing my packages when new versions of dependencies are released. I see the usage of a new version of a dependency as an enhancement to my package so I'm quite happy to bump the version number accordingly.

The problem of passing types between libs is an issue in the current runtime, but I expect assembly neutral interfaces to be the path out of that problem in vnext.

[davidfowl]

That sucks though, I wouldn't want all of my dependencies to do this. Everything revs when any dependency changed sounds much worse than what we have today.

[damianh]

What's the versioning story for assembly neutral interfaces? Shouldn't we just have structural typing and be done with it :trollface: