Build script "hook" for Rez plug-ins

[mottosso]

Goal

Eliminate vendor lock-in.

Motivation

In order for a Python project to be both PyPI and Rez compatible, you need two project definition files, a setup.py and package.py. Likewise for a project to compatible with conan or vcpkg or yum or any other package manager out there, you need 1 more package definition.

With rez pip --install, we've demonstrated that Rez as a build-system can be entirely stripped out and delegated to a third-party. That the workflow is sound and that it avoids the need to write a second-class package definition file for each package in order to benefit from what makes Rez strong - dynamically creating environments from a self-curated repository of packages.

I'm integrating Rez with a studio at the moment, and one of the things we'll have to do is replace our existing (internal) package manager file-format (not unlike Rez) with Rez's package.py file. If at any point in the future we decide to move to the next great thing, history will repeat itself. If instead packages had been written for PyPI and VcPkg, our eggs would not all have been in the same basket.

Implementation

Before	After

By removing the build-system aspect from Rez, we can instead focus on bridging the gap between external build systems, allowing both existing projects from working natively and instantly with Rez - such as those already written for Conan or Vcpkg - and make any build definition written for Rez be natively dual-compatible with an external service, such as PyPI or yum.

In practice, the workflow could look like this.

Before

1. Author your project
2. Write your package.py for Rez
3. Write your setup.py for PyPI

After

1. Author your project
2. Write your setup.py for Rez and PyPI

What about package.py?

To clarify, there are two package.py files.

1. The one you author
2. The one released

(2) remains unchanged, making this idea compatible with all of your existing packages and repositories.

(1) however may no longer be necessary. Each external build-system would effectively generate (2). A setup.py covers most if not all requirements of a package.py, and being plain-Python means it can be extended to contain additional fields. Likewise for Conan or CMake definitions. Odds are the essence of a package.py can be reduced into additional fields of an existing package definition format.

Integration

Akin to rez pip, Rez wraps sub-managers to provide additional infrastructure, such as what options and cmake modules to provide.

$ rez pip --install .
$ rez conan --install .
$ rez vcpkg --install .

The subcommand, e.g. pip, identifies the format the package definition file is in, and handled similar to rez/pip.py, although it should probably be delegated to a plug-in.

Benefits

• [x] Instant familiarity A new user with an existing package library on PyPI and local repository is already able to start using Rez with little to no change. Likewise users familiar with any other (supported) build system
• [ ] No vendor lock-in Encourage new users to start using Rez without having to commit, and provide existing users with ease of mind that their investment into writing package definition files are safe and transferable to another system, e.g. everything was written for Conan.
• [ ] Less maintenance Rez is both a build-system and environment management system, the less there is to manage, the more focus it can get.
• [ ] Cross-repository dependencies Build a C++ project with dependencies coming straight out of conan, vcpk, pip and more. No need to convert the dozens of dependencies prior to making it Rez-compatible.

Testcase

Can this work? rez pip has already proven that it can. With it, it's possible right now to stop writing package.py files for any Python project, and in addition have your project instantly compatible with PyPI. The same can be true for just about any established build system out there.

Discussion

Even though your existing package repositories remain compatible, this is a big change to the fundamental nature of Rez. It would turn it into something other than what it is. Something stronger, more welcoming of new users and more forgiving to those that commit long-term. But it is still a big change.

[mottosso]

On a similar note, maybe build can be another plug-in "category", like shell and package_repository? Then, along with #27, we could extract both rezbuild and cmake into their own plug-ins, that claim rezbuild.py and CMakeLists.txt file patterns respectively.

[mottosso]

Ooo, by the looks of it, there already is a build_system plug-in category. I wonder whether this could be used here.