Sub-package auto-export

[TekExplorer]

Something that I see a lot is packages that would work really well together, but can't directly import the package in question in order to remain pure (ex flutter widget libraries supporting the mix package)

I propose some mechanism to conditionally federate packages based on what packages are imported in the application.

For instance, my package would like to generate utilities for package x And let's say I made a separate package for it. The problem is, users don't know that package exists. And what if I support a ton of packages? How many mini packages are users expected to find? Across multiple packages? If my package simply included the sub packages when x is depended on, then users just get first party support automatically.

It has a ton of possibilities. We could easily expand on conditional imports and exports if we wanted to keep it in-package, which would be even better.

[lrhn]

Sounds like a configurable export based on a package being available in the current package configuration, with your package not having a strong dependency on the package. (But it does have a weak dependency with a version constraint, so if the package is included at all, it will be version-constrained.)

Requires two features, one language and one Pub:

• Weak package dependencies. A dependency on another package which a package resolution solution doesn't have to satisfy unless there is at least one non-weak dependency on the package in the solution. Something like a dependency that can be tree-shaken, not really because there is no attempt to solve for it unless necessary.
• Package-conditional import/export: An import or export can use a condition like if (dart.package.foo) to only perform an import if package:foo is available in the current package configuration, the same way if (dart.library.io) only uses the following URI if dart:io is available to the current compilation.

Both features seem technically possible.

The Pub feature requires the constraint solver to recognize that all dependencies on a particular package are weak, in which case the solution is trivial (no dependency), and to recognize that adding another non-weak constraint requires resolving all of the version constraints. The library will likely still be considered as strong dev-dependency, so you can write code using the package to begin with.

The language feature is trickier, because it introduces the ability to query the package configuration for whether a package is there. That may get in the way of modular compilation. You can't compile a package independently of how it will be used, because the conditional import depends on the dependencies of other packages. That's why conditional imports are so restricted today: Each package only needs to be compiled once for each target platform, and the compilation artifact can then be reused. That only works if conditional imports can only behave in one way per target platform. That issue may be enough to disqualify the feature.

[jakemac53]

Agreed about the modular compilation issue. In that world you would have to compile as if all dependencies were present when doing a modular compile, which not only might be inefficient but also could make behavior between modular and whole world compilation provide a different result, which probably isn't viable.

We had previously explored package "features" in pub, which would basically allow depending on specific sub-libraries of a package, with their own dependency constraints. That would make things more discoverable at least, but users would have to import specific libraries still in order to get the features that are applicable to them, and depend on those features separately.

[TekExplorer]

I'm not too hungry up on any specific implementation

Rust has features, which only includes code if you specifically enable the feature on the package, which can be easily documented. Not as automatic as weak dependencies, but not that bad either.

The most straightforward option is to have features correspond to the export of sub-packages. (I've been recently experimenting with the new workspace feature, and it could be connected)

That way, it's basically like adding a package, but implicitly, and it's in one place.

I think that just giving authors some way to bundle compatibility with other packages is crucial to an improved ecosystem, and gives some packages the opportunity to actually get popular, where they simply can't without widespread use.

I don't necessarily care how

[lrhn]

In that world you would have to compile as if all dependencies were present

Which means making the dependencies be present, the "weak dependency" would be treated as a strong dependency.

[TekExplorer]

Just so I can contribute to the conversation, what exactly is modular compilation?

[jakemac53]

Just so I can contribute to the conversation, what exactly is modular compilation?

In this context we mean compiling libraries (or groups of libraries) separately, such that they can be shared across multiple applications. This doesn't come into play much for external projects, because this style of compilation isn't common there (for Dart), but internally we use almost exclusively modular compilation.

An external equivalent would be if when you download a package from pub, you didn't get the dart files but instead got a single kernel file (this is the intermediate format for Dart code). When compiling that kernel file, it doesn't know how it is used by any apps, it only knows about its own (strong) dependencies. So, it isn't clear how this could work with the feature as described (I proposed above you would probably have to compile it as if all weak deps were present, but that defeats the purpose largely, and could even lead to an incorrect result).

[TekExplorer]

In that case, one possible option is to have the "features" compile into their own, independent mini-kernels that can be included or not

In a lot of cases, the core package doesn't actually need to be aware of the extra code, and just need to prepare itself with interfaces and such if that's even necessary at all.

If it's extensible, then you can extend it by just adding code without directly modifying the original kernel.

[jakemac53]

In that case, one possible option is to have the "features" compile into their own, independent mini-kernels that can be included or not

Yes, if we use "features", with separate libraries that you import to get support for each of the packages you use, then I see no major issues. The original proposal was I think asking for a more magic thing, where new APIs are exported from a single library based on your dependencies.

independent mini-kernels that can be included or not

We can do this if they are separate libraries, but don't have the ability to separate a library across multiple kernel files, at least today.

[TekExplorer]

Yes, if we use "features", with separate libraries that you import to get support for each of the packages you use, then I see no major issues. The original proposal was I think asking for a more magic thing, where new APIs are exported from a single library based on your dependencies.

Sure, but i understand that would be much more complicated

We can do this if they are separate libraries, but don't have the ability to separate a library across multiple kernel files, at least today.

Sounds like this is the most straightforwards solution then

In addition, this simple "federation"-like solution doesnt block more complicated solutions later, so i believe it should simply be added.

It also doesnt block automatic-feature-enabling in the future

[jakemac53]

That just becomes a package manager feature request then, @jonasfj is there an open issue already for package "features"? What is the general pub team stance on this? I know there was a version of it at one point which was reverted, but I don't know the details.

[TekExplorer]

Just a note on naming: this could be called "includes", which basically just has a string:package map, essentially.

Otherwise, we could use "features" with the same, and reuse it for future, more complicated functions.

I love cool features that leave room for awesome stuff in the future.

I can already see a bunch of packages that could benefit from this. (That being said, let's make sure it can be IDE auto-completed)

[jonasfj]

Years ago we had "features" in pub, but it was never really launched.

It allowed a package to declare dependencies that would only be required, if the package depending on it enabled the feature.

As there was no mechanism for declaring that a particular library required a particular feature to be enabled, this didn't have any practical use.

Hence, we ended up removing "features", I wouldn't be opposed to reintroducing it. BUT: we'd have to find a way to make libraries dependent on a feature.

And for that to work, there is a LOT for work for the analyzer, which would need to be aware. The modular compilation issue might pop up.

---

I think we should learn more about what use cases we're trying to solve. The initial comment talks about discoverability as a major driver, which I think can be addressed in other ways:

• Custom topics in pubapec.yaml
• Links in README
• Mentioning other packages in relevant documentation.
• Use of a verified publisher (maybe one with more than one person behind it)
• Maybe related packages on pub.dev?

When the word sub-packages comes up, I imagined the motivation would be multiple version numbers. To version libraries in a single package with different versions. That probably crazy, and hard for users to follow :)

If it's about reducing risk of dependency conflicts, I think options could be:

• "features"
• Optional dependencies (constraints in siblings dependencies)
• private-dependencies (my personal favorite)
• Optional SDK constraints.

I think we could get far with optional dependencies or even just optional SDK constraints. We likely will need that for native assets when that stuff is supported.

---

If people are breaking up packages I think we should try to understand why first. A crystalize the different scenarios that leads people down such a path. It's not necessarily wrong, but it can make life harder.

I'm certainly not sure federated plugins turned out to be an unmitigated succes -- they so have upsides, but how often are 3rd parties really implementing support for a specific platform independently of the plugin author.

[TekExplorer]

One option is:

If we decide that features are just the exporting of a completely separate package (which may have extensions or which may add new subclasses)

Then it would be simple to list features in pub.dev such that they link to unlisted packages

The more complicated options of optional dependencies and weak-depends would need much more thought.

Plus, there's no reason we can't start with the simpler option (leaving room for extension later) and seeing how it gets used. Especially as it can be added to existing packages by letting them declare features in-place, possibly even for already published versions in pub.dev.

It can enable say... firebase to include auth or messaging and would be nice to use.

More usefully is providing micro-packages through features that add special compatibility with specific packages, such as dart-only packages including a flutter feature, which, depending on how we represent "features" in this way might even help clean up, or at least lessen the whole "dart" and "flutter" naming in the future. Plus, they wouldn't need to be separately visible packages. Just one, and the feature can add its own documentation.

All in one place. Easily achievable, I think.

Just a few thoughts.

Weak dependencies and such can also piggy back off of this function later.

Also, package authors could easily declare _thirdparty packages as features, federation-style.

Perhaps riverpod might include my pagination package? (As a theoretical example)

A lot of fragmentation could be cleaned up, in addition to encouraging inter-compatibility (like widget packages including mix utilities)

[jonasfj]

Just to reiterate, what are the concerns we're trying to address. Can we enumerate them and enumerate the scenarios in which they occur?

[jakemac53]

As I understand it, the core ask here is for a single package to be able to provide tooling related to a variety of other packages, without introducing a dependency on those packages that a given user doesn't actually use.

One of the core use cases for this would be testing. Today if you want to release testing related utilities which themselves have testing specific dependencies, you have to release those as a totally separate package, because you don't want those dependencies leaking out to all users of your package (especially, transitive users of those packages).

It would be much nicer imo, if you could instead release those as an optional feature of your package, which could be depended on separately, and importantly would only be a dev dependency for the consumer package. So, lets say I am writing a Builder, from package:build. My dependencies would look something like this:

dependencies:
  build: ^1.0.0
dev_dependencies:
  build#test: ^1.0.0 # Hypothetical syntax for a sub-feature, not trying to rathole on that right now.

This would mean only users of the "test" feature from "build", get those extra transitive deps related to testing. And a regular dependency on build (especially, a transitive one), doesn't pull in those deps.

I do think it is likely you would want to version features separately though, and that might get overly complicated, so maybe forcing separate packages is better, I don't know.

[lrhn]

The test scenario sounds like a "transitive dev-dependency". Except that that doesn't really make sense in the current dependency model.

If we have a package which provides libraries for helping with testing its own framework, then since all a package can give to other packages are the files in lib/, the test-helper library must be in lib/. But then the test-helper library's dependencies must become non-dev dependencies of the package, up to and including package:test, or at least package:matchers.

The alternative, as I understand it, would be to make the test-helper library a "feature" that is only enabled for code compiled with access to package:test. (Which ... won't help, since every package in existence has package:test in its package_config.json, even if almost none of them use it inside lib/. The distinction between dev-dependencies and non-dev dependencies is enforced by analyzer warnings only. Should we mark "dev-dependencies" in package_config.json, and make it strongly enforced that you can't access a dev-dependency package from inside the packageRoot (lib/)?)

I can see why one would make a second package instead, with those non-dev dependencies, so that the down-stream user can have a dev-dependency on the second package.

[jakemac53]

The test scenario sounds like a "transitive dev-dependency". Except that that doesn't really make sense in the current dependency model.

You wouldn't just want to pick up all the dev dependencies of the package you depend on, that could involve many other deps that aren't related to testing (for example, codegen tools that are local to the development of that package).

The feature also has usefulness outside of just development time related stuff (the original case in this issue isn't related to dev time only things afaik), it was for a package which provides helpers for a variety of other packages, and only wants to introduce those dependencies if they already exist.

I can see why one would make a second package instead, with those non-dev dependencies, so that the down-stream user can have a dev-dependency on the second package.

Yeah this is the current strategy (see build_test). But it doesn't quite seem right that you have to do this, just to share a few test related helpers like mocks/fakes etc.