Support for optional dependency groups

[savente93]

As some context around this feature request: I currently work on a project that has a plugin system, and each of the plugins have some quite heavy dependencies, so we would like a way for these plugins to install their dependencies but not have to include unnecessary ones, without having to maintain separate environment files. Our currently pyproject.toml has a table something like this:

[project.optional-dependencies]
io = [
    "s3fs",             # S3 file system
    ...
]
extra = [
    "xugrid",
    ...
]
dev = [
    "black",       # linting
    "ruff",        # linting
    ...
]
test = [
    "pytest>=2.7.3", # testing framework
    "pytest-cov",    # test coverage
    "pytest-mock",   # mocking
]
doc = [
    "nbsphinx",                     # build notebooks in docs
    ...,
    "hydromt[jupyter,extra]",       # examples are included in the docs
]
jupyter = [
    "jupyterlab",      # run examples in jupyter notebook
    ...
]

full = ["hydromt[io,extra,dev,test,doc, jupyter]"]
slim = ["hydromt[io,extra,jupyter]"]

The exact way that these groups are specified doesn't actually matter that much to me, but the idea that I'm basing all of this off is cargo and it's feature groups. For me personally this feature would consist of three components in decreasing order of importance.

1. being able to specify which groups of dependencies are needed and have those be installed
2. the ability for groups to reference each other (e.g. dev could require "docs" and "testing")
3. Having some information about which groups were used in the lockfile.

In an ideal case I would also love the ability to "incrementally" up or downgrade between these configs, but I feel like that is a bit much for just one issue, so I'll open that discussion in a separate one

[baszalmstra]

This is related to #239 as well as https://github.com/conda-incubator/ceps/pull/55

[olivier-lacroix]

This would be great.

Not quite sure how that would interact with the lockfile(s): optional dependencies do not have to be 'compatible' between each other. Would there be one lockfile per optional group?

[savente93]

A lockfile is a record of exactly which versions have been installed right? In that case I think the simplest option for both users and the devs is that a lockfile doesn't really need these notions. There is the "the lockfile" of whatever was installed. If projects want lock files per group (or subset thereof) it should be on them to generate and commit them. I beleive this is how cargo does it too? (though don't quote me on that) Honestly I'm inclined to say "do whatever cargo does" because that's seen quite some use already, so they would have some experience in this already I assume

[paugier]

PDM uses both [project.optional-dependencies] (https://peps.python.org/pep-0621/#dependencies-optional-dependencies) and [tool.pdm.dev-dependencies] (which are both useful) and a lockfile taking into account these groups of optional dependencies.

[pavelzw]

This is fixed now with the pixi multi env feature