Optional Dependencies for conda v2

[travishathaway]

This is a second attempt to draft a CEP for optional dependencies for use by conda compatible package managers (e.g. conda and mamba).

Please see the CEP itself for more information.

[wolfv]

Can packages in the repodata also depend on other packages + optional dependencies?

[mbargull]

This CEP was mentioned by @chenghlee in a discussion at https://matrix.to/#/!ugIkjExjZULQyVDDNz:gitter.im/$vMqrBnqcqeOfqNq27A4epLJ_xDuIjXiu-59XN_KeBbk?via=gitter.im&via=matrix.org

I didn't yet read the text or discussion much, but just wanted to copy my points from the chat:

> There are multiple steps; main points (list not necessarily exhaustive): > > 1. Define the actual semantics on what "optional"/"extra" would mean (ask different people and you'd get different answers...). > I.e., figure out how this fits into the general dependency resolution and what defaults should be (and look out for typical corner cases, e.g., differences between env `create`/`install`/`update`.) > 2. Define the data storage/communication layer, i.e., "repodata CEP update" and stuff. > 3. Define the user interface for this; e.g., (amongst other things) how would a user say "install foo with bar". > - This cannot be `install foo[bar]` since we have `[...]` already reserved for "match spec". > - Would have to be `foo[extra=bar]`/`foo[extras=[bar]]` or the like if this would use he match spec syntax; could be something entirely different, too, though.

[xhochy]

I don't understand why we need this feature. We already use in several cases multi-output recipes for optional dependencies where a package[feature] on the pip side is translated to a package-feature package. For the sake of simplicity, I would prefer to avoid to define an additional way of doing it and keep the current way of doing it.

[h-vetinari]

We already use in several cases multi-output recipes for optional dependencies where a package[feature] on the pip side is translated to a package-feature package.

Yes, but it's a crutch, and a persistent paper cut that people run into and contributes substantially to friction and user-frustration for requirement-incompatibilities between PyPI and conda. Even a bunch of conda-forge recipes get this wrong (and don't even get a warning that this doesn't do what's expected!). It should IMO absolutely be fixed just based on those two points alone.

If we zoom out a little bit further, this "string-based work arounds instead of sufficiently expressive syntax/metadata" is a constant source of pain (the other huge pain point on this being the completely overloaded build strings), which I mention here because IMO both should be solved with the same syntax.

conda install package[feature,build_number=1,cuda=True]

would be orders of magnitude more user-friendly than the following jumble of symbols

conda install package-feature=*=cuda*_1

and that's even before getting into https://github.com/conda/conda/issues/11442, https://github.com/conda/conda-build/issues/4498, https://github.com/conda/conda-libmamba-solver/issues/446, etc.

[xhochy]

My main concern is that I want to avoid is adding an attribute to repodata.json as this will take longer to propagate to all implementations.

It would be nice if the problem could be solved by mapping the square bracket syntax to plain packages, i.e. make this a client-side convention.

One example could be to map jaxlib[build_number=1,cuda,docs] to jaxlib=*=*_1 jaxlib-cuda jaxlib-docs.