Dependencies with dependency_overrides are grabbed differently for -spath and -sgit

[zzwx]

Environment

Pub 2.14.4

Dart SDK version: 2.14.4 (stable) (Wed Oct 13 11:11:32 2021 +0200) on "windows_x64"

windows_x64

Problem

Dependency with dependency_overrides enabled for an executable tool, with a bin folder, used privately within a team, hence published to a private git repo - at least for the time being. Basically it's a css purging library in pure Dart, nothing fancy. A csslib is overridden to grab a patched version of csslib from a github cloned repo like so:

dependency_overrides:
  csslib:
    git: https://github.com/zzwx-forks/csslib

Cloned and patched repo has a modified version number as well, different from a published one (last digit increased). Another library in direct use, html, has csslib as its dependency as well. Obviously there's no way to patch a child dependency (actually I tried to clone and override it as well, patching its dependency, and all observation remained the same).

Now there's two ways to activate the tool and make it available within a team.

1) dart pub global activate -spath <path> This one assumes, that someone has a local clone of the tool's repo.

2) dart pub global activate -sgit <git>

This one is supposed to be the easiest way, not requiring anyone to clone the repo, just run this and you are supposed to be up-to-date.

The fun part is that only 1) actually grabs patched dependencies. 2) in fact gets the published official dependencies instead, not adhering to dependency_overrides request. This is observed by different output that the tool produces when grabbed using 1) and 2) ways.

I've tried to clear cache and stuff. I'm wondering if this is a security measure. If so, is there a way to overcome this behavior? Is it a bug? The docs at https://dart.dev/tools/pub/dependencies don't have much information on this.

[sigurdm]

This is kind of working as intended, though I can see it is a bit confusing.

When you global activate a hosted or a git package foo pub will make a in-memory package depending on foo, do a pub get for it and store the pubspec.lock in PUB_CACHE/global_packages.

dependency_overrides are only taken into account for the root package of a resolution.

Path-packages are special when global activated, they use the local resolution instead. This is to enable global-activation of a package being developed. Therefore they are the root-package for resolution.

https://github.com/dart-lang/pub/blob/master/lib/src/global_packages.dart#L52

Could it be that you could use the tool as a dev-dependency of your main project?

main project pubspec.yaml (If the css-tool is called ):

....
dev_dependencies:
   tool:
      git:
        url: <giturl>

dependency_overrides:
  csslib:
     git: https://github.com/zzwx-forks/csslib

And then you can run it with (no global):

dart run tool

[zzwx]

The other side is not using the tool within any Dart environment, they only have Dart installed for the sake of dart pub global.

I keep reading...

[sigurdm]

I'm sorry, but that use-case (using dependency_overrides when doing a global activate from git) is not covered, I'm not sure we want that either. @jonasfj do you have an opinion here?

A possible solution is to vendor the patched css package into the tool. (Currently that is a manual process).

The other side is not using the tool within any Dart environment, they only have Dart installed for the sake of dart pub global.

Hmm - in that case perhaps you should really make a binary with dart compile exe and distribute that? Then they would not need dart at all...

[zzwx]

I guess I'm so used to the idea of a "predictable build" that is popularized by Go. As soon as something is announced in the pubspec, it should build the target lib/bin using the same set of versions and repositories, no matter how you are getting it and who gets it.

With the absence of cross-platform compilation, a possibility of receiving something from a remote git and build in in-place becomes especially appealing.

[sigurdm]

I guess I'm so used to the idea of a "predictable build" that is popularized by Go.

You can pin the version in pubspec.yaml by using a specific version in the dependency. But not override the version resolution logic.

[sigurdm]

If we started using dependency_overrides when activating a git package it will be breaking backwards compatibility.

We could work around this by having different behavior for different language versions.

This is not a high priority for our team currently.

[zzwx]

If we started using dependency_overrides when activating a git package it will be breaking backwards compatibility.

Breaking anything is certainly inappropriate!

I'm only brainstorming right now of what would be appropriate instead. I'd dig into this a little if you would say it could be done. Let's say this behavior would be activated only with a flag for dart pub global (like --follow-deep-dependency-overrides)?

[jonasfj]

--follow-deep-dependency-overrides

Following dependency overrides beyond one-level would IMO always be wrong.

---

When doing a deployment, it makes sense to respect the dependency_overrides of the root package. One could also argue that when deploying an application it makes sense to just use the pubspec.lock from the root package.

When using dart pub get in the root folder of a project (whether flutter app or dart server app) this is also what we do, and this makes sense here. The dependency_overrides allows the application developer to overrule the solver because the application developer knows that the way packages are used in their application, ignoring the version conflicts is acceptable.

In practice when deploying an application you should get the dependencies from pubspec.lock, because this ensures you get the exact dependencies your application was tested with. This is also what happens when you use dart pub get in a Flutter application, or when building a docker image containerizing a Dart server application..

---

It can therefore be argued that dart pub global activate should respect pubspec.lock, because after all global activation of a package is a form of application deployment. IMO this is a reasonable opinion.

The counter argument is that if we do this, then anyone deploying an application using pub.dev (deploying as globally activated package) will have to update their package whenever they want updates from their transitive dependencies. So users of a globally activated package won't get updates until the author of the package updates it, even if the dependencies are updated...

Hence, there is value in not respecting pubspec.lock, which makes respecting dependency_overrides when globally activating packages a reasonable middle ground. And if I were to design globally activated packages from scratch today, maybe this is a design consideration I would take into account.

There is, however, still one issue to consider: People are currently using dependency_overrides for many things (including local development), so respecting them during global activation would possibly break existing users.

This could be solved by saying that starting with a lower-bound SDK constraint of 2.16 we interpret dependency_overrides differently when a package is globally activated (ie. when the package has a lower-bound SDK constraint of 2.16). But:

• Changes like this causes friction.
• We have no good way to communicate this change in the intepretation of dependency_overrides, at-least doing so would likely require us to spam a lot of people who don't care and don't need to know.
• Using dependency_overrides is rarely needed in a globally activated package.
• Few packages are design to be globally activated.

So even if theoretically, I can talk myself into thinking that respecting dependency_overrides for globally activated packages is reasonable. It seems like a lot of friction, for very small gain.

---

It is also reasonable to argue that as an application deployment platform globally activated packages relies on version constraints because we want users of such packages to get updates for their dependencies. If the developer of a globally activated package really wants to lock down dependencies they can use version pinning.

Look at other package systems like debian apt-get, here packages specify which libraries they depend on, and they specify a version range. Most packages I've seen don't require a specific version of glibc, gtk or qt, they require a version later than x.y.z and less than (x+1).0.0 (which is the same as ^x.y.z in pub semantic version constraints).