Allow multiple versions of the same package to be used by an app

[amirh]

Pub currently forces resolving to a single version of each package in the dependency graph. This can be over restrictive for second-level dependencies.

The failure scenario (which I have hit multiple times) is:

1. An app depends on package_a: ^1.0.0 and package_b: ^1.0.0.
2. Packages package_a v1.0.0 and package_b v1.0.0 both depend on package_second_level: ^1.0.0.
3. package_second_level gets a breaking change and the major version is bumped to 2.
4. package_a v1.0.1 is updated to depend on package_second_level: ^2.0.0, package_b was not updated to use the new package_second_level yet.
5. Something forces package_second_level: >=2.0.0 for the app (this may be a result of e.g flutter upgrade updating pinned dependencies, or of the app developer needing a new feature available in the newer versions of package_a).
6. pub cannot solve the dependency constraints of the app 😢 .

If package_second_level is only used as an internal implementation detail in package_a and package_b I would say it is reasonable to build the app with 2 copies of package_second_level's code at different versions (binary size is the cost, but I'd argue it's preferred over not being able to build the app).

We still need to have the kind of dependency that allows only a single version for an app; this is for second level dependencies that are exposed as part of a package's public API.

cc @jonasfj @mit-mit @collinjackson @ianh

[mit-mit]

I'd much rather work on better tools and bots for ensuring that package_a and package_b quickly get onto package_second_level: ^2.0.0.

cc @munificent who can share more details about the historical considerations that lead to the current design.

[amirh]

I'd take quick updates of all packages over this thing any day 😄 Though it is worth considering that updating both packages may require non-trivial changes and may take awhile regardless of how good the incentives and tooling are.

I'd love to use this issue to get a good understanding of the pros and cons of this suggestions, and if it seems like an overall good idea we I'd suggest we understand the amount of effort required to make this happen before deciding on a priority.

[jakemac53]

Dart library identity is defined by the uri the library is imported with. So two imports of package:foo/foo.dart always necessarily must refer to the same library. This isn't a pub thing but a Dart language thing, and its pretty fundamental.

Even if that somehow changed - now you would get very confusing behavior where Package A and Package B leak some instance/type from package_second_level which look equivalent but aren't actually - leading to confusing a Foo is not a Foo errors.

Additionally there is only a single .packages (or soon package_config.json) file per application - individual libraries cannot pick and choose their own versions of packages.

[amirh]

Dart library identity is defined by the uri the library is imported with. So two imports of package:foo/foo.dart always necessarily must refer to the same library. This isn't a pub thing but a Dart language thing, and its pretty fundamental.

Even if that somehow changed - now you would get very confusing behavior where Package A and Package B leak some instance/type from package_second_level which look equivalent but aren't actually - leading to confusing a Foo is not a Foo errors.

Additionally there is only a single .packages (or soon package_config.json) file per application - individual libraries cannot pick and choose their own versions of packages.

What if we add something like internal_dep: package_second_level ^1.0.0 to pubspec.yaml which will rename the package (and the Dart code of package_a will actually be importing the internal name of the package).

(probably there are many edge cases I'm not considering right now, but just throwing an idea)

I'd guess we could potentially do some enforcement that "internal dependencies" are not leaked through public APIs.

[jakemac53]

What if we add something like internal_dep: package_second_level ^1.0.0 to pubspec.yaml which will rename the package (and the Dart code of package_a will actually be importing the internal name of the package).

All imports in the package itself would still have the wrong name :(

I'd guess we could potentially do some enforcement that "internal dependencies" are not leaked through public APIs.

This starts getting really complicated and restrictive though - types leak all over the place.

[jakemac53]

Note that this would also quite likely not just mean copying package_second_level but also possibly all of its transitive dependencies as well.

[jakemac53]

See also https://dart.dev/tools/pub/versioning#shared-dependencies-and-unshared-libraries which has some discussion on the philosophy here :)

[jakemac53]

Fwiw it took me a while to find but this issue has come up before and was closed with wont fix - https://github.com/dart-lang/pub/issues/462. That was a long time ago though.

[amirh]

Thanks for all the feedback!

My gut feeling (based on anecdotal experience) is that this problem is painful enough (for Flutter apps, where it's probably amplified as Flutter is pinning some dependencies) to justify considering remedies (even radical ones).

What if we add something like internal_dep: package_second_level ^1.0.0 to pubspec.yaml which will rename the package (and the Dart code of package_a will actually be importing the internal name of the package).

All imports in the package itself would still have the wrong name :(

I was thinking that all imports in the package will use the internal name for an "internal dependency".

This starts getting really complicated and restrictive though - types leak all over the place.

It just requires that a package author is explicit about leaking a type (by making sure the dependency is not an "internal dependency").

Note that this would also quite likely not just mean copying package_second_level but also possibly all of its transitive dependencies as well.

My guess is that it will be common to prefer a binary size hit over not being able to build your app. Ideally there could be a solution that only takes the binary size hit when it's required.

See also https://dart.dev/tools/pub/versioning#shared-dependencies-and-unshared-libraries which has some discussion on the philosophy here :)

Thanks! I read it again. It's a good read, the argument there against "unshared libraries" is totally valid IMO when you apply this approach to all dependencies, always. I guess what I'm suggesting here is allowing "unshared libraries" for "internal only" dependencies, where that problem does not apply.

Fwiw it took me a while to find but this issue has come up before and was closed with wont fix - #462. That was a long time ago though.

Thanks for the reference, IIUC that issue is talking about applying some sort of the "unshared libraries" approach to all dependencies, which isn't what I'm suggesting to consider here.

FWIW I find this discussion productive - if the decision is that this isn't something we'd want to do, I think it's useful to have the reasons against it documented (which is what this issue seems to be producing already).

[mit-mit]

where it's probably amplified as Flutter is pinning some dependencies)

Perhaps that is where we start then...

[jakemac53]

FWIW I find this discussion productive - if the decision is that this isn't something we'd want to do, I think it's useful to have the reasons against it documented (which is what this issue seems to be producing already).

Ya - I was surprised at my inability to find in depth similar discussions already, as I known it has come up in the past but maybe just more anecdotally.

There are valid arguments on either side of the debate to be sure - to be clear my goal hasn't been to shut down debate just highlight the reasons I am aware of that would make it difficult :).

where it's probably amplified as Flutter is pinning some dependencies)

Perhaps that is where we start then...

I do think flutter pinning dependencies is likely contributing negatively to the problem. It does allow them to be more confident about their releases but at the expense of holding their users back to old versions of packages :man_shrugging:.

[natebosch]

Another downside to this is that it is fundamentally at odds with the philosophy of our internal third party package management.

https://opensource.google/docs/thirdparty/oneversion/

Anything we do to make it easier for the package ecosystem to have long-standing discrepancies in version support for their dependencies will increase the burden of resolving these discrepancies while importing and updating packages internally.

[amirh]

Anything we do to make it easier for the package ecosystem to have long-standing discrepancies in version support for their dependencies will increase the burden of resolving these discrepancies while importing and updating packages internally.

Good point. IMO this is the strongest argument against this proposal. Thanks.

[jonasfj]

Maybe allowing packages versions to be retracted or flagged broken would alleviate some of the concerns around version pinning in Flutter.

Personally, I find the cargo model very attractive. But authors can also just publish mypkg2 when breaking compatibility, in those cases where the author suspects downstreamers would prefer multiple versions of mypkg.

[jupanubv92]

Because flutter_simple_shopify >=0.0.25-alpha depends on graphql ^3.1.0 and no versions of graphql match >3.1.0 <4.0.0, flutter_simple_shopify >=0.0.25-alpha requires graphql 3.1.0.
And because graphql 3.1.0 depends on http_parser ^3.1.3, flutter_simple_shopify >=0.0.25-alpha requires http_parser ^3.1.3.
And because firebase_auth >=1.0.0 depends on firebase_auth_web ^1.0.0 which depends on http_parser ^4.0.0, flutter_simple_shopify >=0.0.25-alpha is incompatible with firebase_auth >=1.0.0.
So, because flutter_store depends on both firebase_auth ^1.0.0 and flutter_simple_shopify ^0.0.25-alpha, version solving failed.

How do you fix these kind of issues when pub does not allow multiple versions of the same library?

[jonasfj]

Dependency overrides is an option: https://dart.dev/tools/pub/dependencies#dependency-overrides

It's also possible that you need to downgrade other packages..

[jakemac53]

@jupanubv92 The ideal solution here is to file issues on your dependencies that don't allow the latest versions asking them to update. These messages can be a lot to decipher (usually because its actually a complex interaction they are trying to describe), but in your case the issue looks to be that the version of graphql that flutter_simple_shopify uses doesn't allow the latest http_parser, but the latest version of that is required by the latest flutter_auth_web. So I would file an issue on flutter_simple_shopify to update to the latest version of graphql.

You could try downgrading firebase_auth until flutter_simple_shopify does update, or using a dependency override on http_parser in the meantime (set it to the latest).

[jupanubv92]

@jonasfj, @jakemac53 thanks for your response. I gave up upgrading to Flutter 2.0 because as you can see I have package dependencies that haven't upgraded to the latest pub packages optimized for Flutter 2.0.

The only thing that I can do is to ask about the upgrade on these packages that haven't upgraded yet. However, the reason for my message here is to ask again the question to see if it is worth considering supporting multiple versions of the same library. This will help with major Flutter release in future such as Flutter 2.0 where people have to wait weeks/months until all packages upgraded so they can benefit from the latest features.

[jakemac53]

@jupanubv92 yes I understand the use case. However I believe quite strongly myself that the cons of that approach would strongly outweigh the benefits. See my above comments to describe what I mean. The language itself really doesn't support this either (also described above).

[munificent]

if it is worth considering supporting multiple versions of the same library.

It's not really a question of "cost", at least in terms of Dart team cost to implement the support. It's not like if we had more time we could just implement it and be done with it.

Package sharing is fundamental to how pub and the Dart language itself works. Changing it would require significant change not just in our implementations, but in how users write code and all of the workflows and best practices around code reuse. It would significantly increase the complexity of reusing code and likely increase the size of shipped applications.

[gh-pap]

Not here to ask advice (though that would be wonderful) but to add a real-world use case. I'm new to Flutter and building a PWA which needs to generate as well as read (in the same app) EAN-13 (standard barcodes) and GS1-compatible data matrix codes.

I must use the "barcode" package to generate codes (because it's the only package which supports GS1-compatible data matrix, a spanking new feature). And there's only one web-compatible package (that I can make work) that can read EAN-13/data matrix: "ai_barcode".

tl;dr my project needs the latest "barcode" package and AFAIK any version of "ai_barcode".

"I have everything I need, this is going to be a piece of cake" I said, not realising = famous last words.

The dreaded error:

Because ai_barcode >=3.2.1 depends on ai_barcode_web ^3.0.1 which depends on qr_flutter ^4.0.0, ai_barcode >=3.2.1 requires qr_flutter ^4.0.0. And because qr_flutter >=4.0.0 depends on qr ^2.0.0 and barcode >=2.2.0 depends on qr ^3.0.0, ai_barcode >=3.2.1 is incompatible with barcode >=2.2.0. So, because plainli_web depends on both barcode >=2.2.2 and ai_barcode >=3.2.4, version solving failed.

In short, barcode depends on qr ^3.0.0 whereas ai_barcode depends on qr_flutter ^4.0.0 which in turn depends on qr ^2.0.0. Hmmmm...

I did the usual googling, tried lots of suggested workarounds including downgrading ai_barcode (a flurry of null-safety issues) and dependency overrides (many combinations attempted, nothing worked). But I wasn't worried because I said to myself "these packages are open source, I'll just make a copy of the source from one package or the other, point pubspec at the copy's path and voilà! Sure my project will grow in size, but who cares, what I need is a working system."

If I'm interpreting this thread accurately then there are no such options and the onus is on me to ask the owner of qr_flutter (last touched 19 months ago) to release a new version compatible with qr ^3.0.0. Of course I will do that and though someone might say "great, see, the system is self-organising" I would argue that the frustration endured and time spent (not invested, spent, thus a real opportunity cost to the dev) trying to resolve package dependency issues isn't doing Flutter any favours.

Solutioning (sorry, can't help myself). As a package selector, I'd love a prominent (and filterable) badge on pub.dev saying whether this package "plays well with others". Perhaps this idea goes into the "too hard" bucket: too many permutations, like asking for a giant pubspec.yaml in the cloud -- impossible; but IMO a badge would be ideal. I would then (personally) only consider packages which display the play-well-with-others badge and, theoretically, this should motivate serious package developers to maintain the credentials (so to speak) of their package(s) (similar to how the "Null-safety" badge was used). I'm sure there are better ideas, but the essential requirement is to reduce the time spent trying to sort out package dependencies.

As Flutter grows in both users (especially newbies like myself) and packages this issue may get worse.

P.S., just found a thread (https://github.com/theyakka/qr.flutter/issues/174) with a potential workaround for my specific issue which will involve me making a copy of ai_barcode, modifying it a bit, then pointing at a new qr_flutter (currently only on github). I will try it, of course, but independent of whether it works or not, it all feels quite messy and is definitely time-consuming.

[sigurdm]

We have been discussing this quite a lot - and there are definitely pros and cons to allowing multiple versions of the same package in the same resolution. I don't think we are going to build support for it in the near future.

P.S., just found a thread (https://github.com/theyakka/qr.flutter/issues/174) with a potential workaround for my specific issue which will involve me making a copy of ai_barcode, modifying it a bit, then pointing at a new qr_flutter (currently only on github). I will try it, of course, but independent of whether it works or not, it all feels quite messy and is definitely time-consuming.

FYI @jonasfj has made https://pub.dev/packages/vendor that makes this a bit easier!

[driver4567]

Download the package and import it directly in to you project. I had to do this recently.

For the file I imported I just put a version number in to the naming of it, to have it referred to differently

[nishantgta]

Download the package and import it directly in to you project. I had to do this recently.

For the file I imported I just put a version number in to the naming of it, to have it referred to differently

Can you specify how to achieve this ????

[driver4567]

So you can download the source code for the second version of a package you'd like to work with, and then make a folder in your project and place the package you downloaded in to that folder, and just import the root code file from that project, in my case I changed the name of it before importing. And now I can use both versions of packages.

[mateusfccp]

In our case, we have the following case:

We have a package a that we use in our code. This package is provided by we ourselves, so we have control of its releases and everything.

We are releasing a new major version of a that breaks some APIs and works differently. We are migrating the main app code to match the new version's changes. However, we don't want to release this immediately, and we also want to the control to quickly revert the changes if necessary (unexpected critical bug, for example). When this is the case, we put the new changes behind a feature toggle.

Thus, for this feature toggle to work, we have to keep two versions of the code, one with the old version and one with the new version. Then, we would simply toggle between versions as needed.

I tried to provide two versions of the same package through different names in pubspec, but the package name doesn't match and I get an error. What was I expecting is that if I did something like:

a:
  git:
    url: git_url/a
    ref: v2.2.0
a_next:
  git:
    url: git_url/a
    ref: v.3.0.0

And then, in the code I would simply import package:a/a.src or package:a_next/a.src, for instance.

Alternatively, we can clone the repository and change the name of the package in the cloned repository to use it in the meantime, but it's burdensome and more error-prone to maintain two identical code bases.

[driver4567]

In the code where you are importing a.src twice this will clash, change the names of one of them.

For example make one a2.src. I'm not sure that it will then be needed in pubspec, just import the code directly in to a project folder

On Thu., 19 Jan. 2023, 11:32 pm Mateus Felipe C. C. Pinto, < @.***> wrote:

In our case, we have the following case:

We have a package a that we use in our code. This package is provided by we ourselves, so we have control of its releases and everything.

We are releasing a new major version of a that breaks some APIs and works differently. We are migrating the main app code to match the new version's changes. However, we don't want to release this immediately, and we also want to the control to quickly revert the changes if necessary (unexpected critical bug, for example). When this is the case, we put the new changes behind a feature toggle.

Thus, for this feature toggle to work, we have to keep two versions of the code, one with the old version and one with the new version. Then, we would simply toggle between versions as needed.

I tried to provide two versions of the same package through different names in pubspec, but the package name doesn't match and I get an error. What was I expecting is that if I did something like:

a: git: url: git_url/a ref: v2.2.0 a_next: git: url: git_url/a ref: v.3.0.0

And then, in the code I would simply import package:a/a.src or package:a_next/a.src, for instance.

Alternatively, we can clone the repository and change the name of the package in the cloned repository to use it in the meantime, but it's burdensome to maintain and more error-prone.

— Reply to this email directly, view it on GitHub https://github.com/dart-lang/pub/issues/2272#issuecomment-1396981048, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADHCASDMUU7LSTEKN4T6AB3WTE66VANCNFSM4JU5PZ6A . You are receiving this because you commented.Message ID: @.***>

[MBjoern]

I understand that this may not be the preferred approach, but it seems that it may be necessary given the circumstances. The conversation here has been valuable, but it is important to consider the practicality of the situation as well. For example, if there are significant bugs or security issues with a particular library that is used as a dependency in many other libraries, and the majority of those libraries are not updating their dependencies, a company may need to make the difficult decision to either remove or rewrite a significant portion of their features, or forego the security update. I understand that this is not ideal, but it is important to remember that companies and managers often have financial considerations to take into account as well, which can lead to problematic situations which can be circumvented by finding a solution here.

[MBjoern]

Download the package and import it directly in to you project. I had to do this recently.

For the file I imported I just put a version number in to the naming of it, to have it referred to differently

Even though not pretty, nor maintainable, this is the most practical solution for me at the moment. I'm glad I found someone thinking alike :D

[FrancoisG-WIMT]

I wonder, would restricting this requested ability to an isolate level (similar to how .NET achieves it via multiple active Application Domains), make this more tenable?

[jakemac53]

I wonder, would restricting this requested ability to an isolate level (similar to how .NET achieves it via multiple active Application Domains), make this more tenable?

This would already work afaik - isolates can take a package config which is different from the root isolate package config. You might run into some weirdness just because I doubt it is used much and might have corner cases, but it should be possible.

I don't know what the workflow would look like for pub though.

[sigurdm]

I don't know what the workflow would look like for pub though.

One (trivial?) workflow would be to make multiple packages with seperate pubspec.yamls and have a program running in one start an isolate with code from another, with the resolution from that package...

That would give you full control over packages used each isolate and can work already today.

[martin-braun]

To quote myself:

I wouldn't care if my app grows in size. This is just a frustrating experience over and over. I think we need a way to declare a package as stale, so that it will embed its dependencies separately from the normal dependencies, so it can work without any version mis-match.

This [issue] is a very frustrating experience with such a big impact that I would never pick Flutter for a new project, because I know what bitter after-math can result in this, unless you only use official packages.

I want to be able to define a package as stale, so that it keeps its dependencies separately isolated. It would increase the app size, but the current method is just a blocker and causes unnecessary cost from the business point of view.

Having the ability to stale a package would also motivate people to build Flutter components, because they don't have to be worried to be stressed by constant blocking issues and PRs. They can take time to modernize their packages without people being stuck. They still should maintain their packages, but at least it won't block people who decided to pick their package in the past.

[jonasfj]

Hmm, interesting... Ideas for how to quantify this issue would also be useful (Please upvote this issue).

My impression of how pressing this issue is might be incorrect.

I wouldn't be surprised if the single version limitation will eventually cause a lot of pain. So one day we should find a way to address it better.

@martin-braun, have you tried: https://pub.dev/packages/vendor

To vendor the stale dependency and any stale dependencies it might have. Perhaps it could be smarter and more guided, and yes it copies a lot of ugly files into your repository.

And sure if doing flutter plugins or something like that it probably won't work well.

[martin-braun]

@jonasfj Yes, this solution isn't pretty. It would've been better if this was implemented into flutter pub instead, but man, I'm grateful for your comment. I never heard of this package and I thank you for guiding me to this package. It would really be a tool for the last resort, given the fact that you end up committing dependencies, but it's certainly quicker than forking and adjusting a package.

I still hope that we can come up with something that integrates into Flutter itself.

[natebosch]

Vendoring dependencies can cause failures or unexpected behavior, and should be a last resort solution applied with significant care. No types from the vendored package can be exposed as arguments or returned values from public APIs. In a strict sense the API "surface area" is much wider than the directly referenced types, but depending on risk tolerance it captures the most likely static problems. We don't have any tooling that would compute whether a type from the vendored package is exposed in the transitively reachable public API surface area.

There can also be runtime behavior changes that are impossible to detect statically. Types defined in a vendored copy behave differently for is and as than the types in the sources they were copied from.

[jonasfj]

Yes, this solution isn't pretty.

Agreed, it's a lot better than forking a package you don't plan to maintain. But generally it's a last resort.

Full disclosure: I kind of prototyped package:vendor with the idea that it would allow us to play with multiple versions and experience some of the other issues that will bring along.

No types from the vendored package can be exposed as arguments or returned values from public APIs.

This is only a concern if you're developing a package yourself. If you're vendoring stale packages into your application you don't have a public API to worry about.

I do wonder if we could extend package:vendor to detect this using static analysis, we should be able to check if the signatures declared a free of types defined in censored packages. What actually comes out at runtime is hard to predict, but I'm guessing such types wouldn't be different from private classes.

Types defined in a vendored copy behave differently for is and as than the types in the sources they were copied from.

This will almost always be possible in any scenario where multiple versions are allowed. It's a major caveat, doesn't mean we shouldn't explore it.

If you're making a package and you don't expose types from a vendored package in the public API. And you remove the dependency on the vendored package, so that you only have the vendored version. Then I figure you are less likely to cause this sort of issue.

Of course then all you've done is copy/paste a stale dependency into your package, such that you could do a few hacks on it...