add provides functionality

[nerdvegas]

Snippet from https://github.com/nerdvegas/rez/issues/673:

In VFX there are often cases where a DCC ships with its own copies of other libraries (python is a classic example). Having rez resolve an environment that includes such a DCC, but also includes a standard installation of such a library, can be problematic.

To help manage this situation, it would be useful if a package could describe that it also provides another package as part of its own installation. This is another case where ephemerals come to the rescue - along with some changes to the solver.

To indicate that a package provides its own copy of another package, you would include it as a package feature:

requires = [ ".{this.name}.feature.provides.python-2.7.3" ] When the solver encounters a "provides" ephemeral, it would use it in lieu of its possibly already-resolved, real package counterpart. If a provided package ends up in the resolve, it plays no part in configuring the environment (the responsibility for that is left to the package - typically a DCC - doing the providing).

Note that the distinction between (eg) python-2.7.3 and python==2.7.3 as a provides statement is important. Whereas python-2.7.3 would take precedence over any python package within that range (including for eg python-2.7.3.1), python==2.7.3 would provide for that exact version only. If python-2.7.3.1 ended up in the resolve in this case, it would cause a conflict.

We will also need to consider what it means if >1 package provide the same package. Perhaps a configurable setting would cause a warning or error; if a warning, perhaps the first package encountered wins.

[bfloch]

We are having active discussions about this very problem. Is it something you foresee being addressed anytime soon?

[nerdvegas]

I'm working on it as we speak!

The basic idea:

• A request for '.provides-foo-1' implies that foo-1 is already provided by something
  • if a pkg specifies it as a requirement, the pkg means that it's providing it (eg python within dcc)
  • if specified by user (ie in rez-env call) or as an implicit - this means that the system itself provides it.
  • for eg, user may not want to have python itself under rez control and instead just use system installation
• If a package is already found during a solve, but hits up on an equivalent .provides later, that .provides is ignored and a warning is printed.

That last point is important. The thing is, a resolve may pull in say, foo-1 early in the solve, and later another pkg pulls in a .provides.foo-1. But by then the damage is done - the resolve has been influenced by foo-1 in a way that can't be reversed (eg foo-1's requirements are already pulled in). It also wouldn't be helpful nor intuitive to treat this as a conflict and backtrack the solve either. Perhaps this behaviour should be configurable though, I'm open to ideas.

In any case I'll let you know when the PR is up and will leave it for comment for a while before we get it merged.

Cheers A

On Thu, Jun 24, 2021 at 3:47 AM Blazej Floch @.***> wrote:

We are having active discussions about this very problem. Is it something you foresee being addressed anytime soon?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1100#issuecomment-867037706, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSUIBF5KJE3V6YX25RLTUIM3DANCNFSM47CSPNGA .

[instinct-vfx]

Great to hear this is in the works! One small question before really thinking about it: Is there a specific reason to have the definition of a pkg providing another package as a requirement? It seems a bit unintuitive to me. Was initially expecting a provides = [] package attribute.

Cheers, Thorsten

[nerdvegas]

Hey Thorsten,

So, the underlying mechanism that drives the 'provides' feature is ephemeral packages, and that will be the case for package 'features' also. Doing it this way greatly simplifies the implementation since we're working with the same thing (package requests) across the board. The solver is already quite complicated, and introducing other kinds of primitives would make it more so (as well as making graph generation a nightmare).

My take on this stuff is to not hide what's going on initially (ie expose the user to exactly what's going on, even if it's not that intuitive). However, I absolutely agree that a syntax like the one you've suggested makes sense. I'm just thinking that that should be a separate effort. I'd like to concentrate on the functionality first, then once it starts getting some use, we'll be better informed on what the new syntax(es) should be.

Cheers A

On Thu, Jun 24, 2021 at 5:47 PM Thorsten Kaufmann @.***> wrote:

Great to hear this is in the works! One small question before really thinking about it: Is there a specific reason to have the definition of a pkg providing another package as a requirement? It seems a bit unintuitive to me. Was initially expecting a provides = [] package attribute.

Cheers, Thorsten

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1100#issuecomment-867418463, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSWJS7GO6KNQBCOGB2DTULPH7ANCNFSM47CSPNGA .

[instinct-vfx]

Hey Allan,

that makes perfect sense, thanks for the feedback!

Cheers, Thorsten

[fnaum]

Hi @nerdvegas , are you still working on this?

[nerdvegas]

Hey Fede,

Yes this is still in the works but I've had to concentrate on other things recently, so it's on hold. I'll be getting back onto it as soon as I can, as it's a feature that we at Method very much want.

Cheers A

On Thu, Sep 30, 2021 at 5:26 PM Federico Naum @.***> wrote:

Hi @nerdvegas https://github.com/nerdvegas , are you still working on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/nerdvegas/rez/issues/1100#issuecomment-930920786, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAMOUSTSXEPTOLY4IOQLSY3UEQGK3ANCNFSM47CSPNGA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[nerdvegas]

Just a ping on this - efforts to migrate to ASWF meant that I moved away from this work. We should pick it up again soon though.

[herronelou]

This is quickly becoming my # 1 wish

[herronelou]

Sorry to poke again, but I'm running into corners that are almost preventing us to release our new Rez-based pipeline which are really weird to solve currently but would likely be solved by the provides functionality.

[JeanChristopheMorinPerso]

I found an old issue that described something similar to that: https://github.com/AcademySoftwareFoundation/rez/issues/298.

[EnLewis]

Hey @nerdvegas,

I'm looking at implementing the provides feature and was wondering if there was any WIP to build from? My current hunch is to make provided packages special ephemerals which don't try to bring in their upstream dependencies.

Thanks!

[instinct-vfx]

@EnLewis Here is the bigger REP that this was pulled from to add some more context: https://github.com/AcademySoftwareFoundation/rez/issues/673

There is no WIP yet afaik and as a heads-up Allan has stepped down from being part of the TSC. He is still around but answers are not guaranteed and will likely be delayed.

That said it's great to see someone actively looking into moving forward with provides. It's one of the most anticipated features for rez.

[EnLewis]

@instinct-vfx Aha I see. Thanks for the quick response! I'll keep working on the my current approach then if the most up to date info on the feature is in that thread.