search

conda / conda

A system-level, binary package and environment manager running on all major operating systems and platforms.
https://docs.conda.io/projects/conda/
Other
6.35k stars 1.64k forks source link

Resurrecting `libconda` for better downstream stories #11925

Closed jaimergp closed 5 months ago

jaimergp commented 1 year ago

Checklist

What is the idea?

With the introduction of CalVer based releases, there was a discussion about API guarantees for projects that depend on conda as a library, not as a CLI tool. While the deprecation cycle can be used as an approximation of API changes (6 months or so) for dependency pinning, it's not an ideal solution.

One potential idea was to create a library libconda to encapsulate most of the objects that might be required downstream. This is not a new idea, though, since libconda already existed for conda v4.0:

However, it was slowly abandoned. libconda was never a dependency of conda and it had to manually kept in sync (source).

What I am suggesting now is to recover this idea but, this time, use it within conda! Thanks to the vendoring mechanism implemented à la pip, conda can vendor libconda as it does with other libraries. Projects will be able to pin libconda as a dependency more freely since it won't affect conda itself.

For this to make sense maintenance-wise, libconda would need to follow semver. It's ok if it breaks often and as needed by conda, since now downstream can pin to whatever version is needed (or even vendor, like conda-lock does), but it needs to follow the semver rules.

I am not sure what would go into libconda, but the previous version had basically everything but the CLI. I'd start with:

This might mean "almost everything but the CLI (and shell stuff?)" again. This would suggest that most of the conda/conda repo would indeed become conda/libconda. In that case it might be easier to extract the non-libconda parts to a conda/conda-app repo or something, but that's a decision about an implementation detail that can wait.

Why is this needed?

Better maintenance story downstream. Note this includes both community projects (mamba, conda-lock, grayskull) but also Anaconda projects (conda-build, constructor), to name a few!

What should happen?

A project should be able to depend on libconda in a API-predictable way or, at least, in a way that doesn't restrict conda's release cycle.

Additional Context

Tagging people who were part of the discussion or might have a say as maintainers of downstream tools that depend on conda: @beckermr @wolfv @marcelotrevisani @mariusvniekerk

beckermr commented 1 year ago

Thanks @jaimergp! I have one question at a technical level. Is most of the technical debt with respect to deprecations in the CLI sections of the code or in the parts of the code that would go into libconda? I ask because if we move most of the code that has pending deprecations back to SemVer, then we've sort of gone full circle.

jaimergp commented 1 year ago

I'd say there's technical debt all over the place 😬

jezdez commented 1 year ago

For the record, conda releases were not SemVer (e.g. see number of 4.x releases and the fabled 5.0). Or at least for quite a while from what I can read in the release notes. This would have implied an actual deprecation policy, which was not clearly spelled out (AFAIK) until very recently (currently being voted on in https://github.com/conda-incubator/ceps/pull/27!).

Also, while I wished we would have had this discussion while reviewing and voting on CEP 8, I'm glad for bringing this up, @jaimergp and @wolfv!

I'm yet to see a clearly spelled out reasoning why the CalVer releases can't be used as a basis for 3rd party tools. I'm inviting you to read other people's opinion on why SemVer will not save you from Hyrum's Law.

Besides the abstract discussion, there is an added complication: With the work on plugin hooks, conda would be tricky to decompose in a way that allows plugins to exist at the same time next to libconda. (E.g. where would the plugin hooks live, in libconda, or conda?)

Both strategies are sensible on their own, but follow different goals:

We've known about plugin use cases for a while and have received many requests over the years, which triggered the work on them (now being in phase 2).

For a reusable version of conda, I don't think I've seen a list of use cases though, which is why I'd suggest to NOT start from the past libconda (which seemed to have taken a broad approach). But instead start collecting actual use cases we need today and the future, before we overcommit to decomposition into "library" and "UI" just for simplicity's sake.

jezdez commented 1 year ago

This is somewhat related to https://github.com/conda/conda/issues/11234 (adding to the context above)

wolfv commented 1 year ago

I think if the deprecation cycle is properly followed, pinning ~6 months into the future might work well enough for mamba's use case. If there is a plugin API defined, then that is probably going to have a positive impact on conda's "stability" and deprecation cycles.

So I was actually somewhat convinced that CalVer might work just as well as SemVer in this case.

dholth commented 1 year ago

Nice idea. Getting there (libmamba) from here is the question, we don't have a superb set of objects to expose to the outside world the way that conda is factored now.

jezdez commented 1 year ago

I'm putting this into the backlog to have a larger discussion around this at some point.

github-actions[bot] commented 7 months ago

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

  1. Verify that you can still reproduce the issue at hand
  2. Comment that the issue is still reproducible and include:
    • What OS and version you reproduced the issue on
    • What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

jezdez commented 7 months ago

not stale

jaimergp commented 5 months ago

Continues at https://github.com/conda/conda/issues/13789, where we can focus on the implementation details.