Feature/rez on pypi part2

[mottosso]

NOTE: WIP PR

Not ready to submit just yet

Hi all,

Here's a version of Rez compatible with PyPI, that as a result is somewhat slimmer.

• 181 additions
• 16,860 deletions (mostly build_utils/)

Try current version

$ pip install git+https://github.com/mottosso/bleeding-rez.git@feature/rez-on-pypi-part2 -U

All tests pass on both Windows and Linux (the PR includes an AppVeyor Windows environment for tests). See below for current minor obstacles.

Overview

I've approached this in stages, whereby you are welcome to merge either one or all.

• [x] Part 1 - Scripts Convert the scripts/ directory to ["entry points"](), so as to get rid of the manual creation of binaries on all platforms
• [x] Part 2 - install.py is no more I got rid of install.py and all that goes with it (i.e. build_utils/)
• [ ] Part 3 - Clean up Address below issues and ensure all is working.

Highlights

Let me know if any of my assertions are wrong; they are based on looking through the Git history comments but some amount of filler has been necessary!

• [x] PyPI Rez is now compatible with PyPI, and as proof and testing ground I've uploaded my fork under the name bleeding-rez, which is what you're seeing above. There's not much to say about it other than it works as you would expect. No tricks.
• [x] PyPI Auto-deploy Rez is automatically submitted to PyPI whenever a GitHub release (or tag) is made. Keeping PyPI in sync with GitHub
• [x] Windows tests I've replaced Travis with AppVeyor; you shouldn't notice any difference, apart from tests now also running on Windows. Travis is still there for the time being, to make you comfortable with the switch.
• [x] Windows tests On-top of that, I noticed some tests hadn't been run on Windows before and fixed those. I suspect it was related to how some tests would silently fail (a.k.a. get skipped) if certain binaries wasn't found on the PATH. The problem was, the way binaries was discovered on Windows made it impossible to find cmake, and thus cmake tests were never run. I did however run into the following issue..
• [ ] Visual Studio on Windows As a side-effect of addressing tests on Windows, I found that vcvarsall.bat doesn't work with the way the cmd shell is implemented; cmd is inheriting PATH from the system registry, whereas Visual Studio assumes PATH is appended interactively. As a consequence of this, it does mean Rez now inherits PATH more generally, which may or may not be what we want. See 1
• [ ] system.is_production_install is no more This problem may be better addressed in another way, see 2
• [ ] sup_world test disabled I couldn't figure out how to address this; somehow, the CMAKE_MODULE_PATH got appended with a Windows-backslash which threw cmake. See commit message for details.
• [ ] --target Custom install directory with --target doesn't work with how Rez manages your PYTHONPATH. Because if you do use --target you would also need to manually add Rez to your PYTHONPATH but you can't, since rez env will take control of that PYTHONPATH and remove it once you've entered into an environment. Without --target, rez will still be accessible as site-packages is accessible despite not being on your PYTHONPATH. Workaround is adding the target to your ./~bashrc on Linux and system PATH on Windows.

Standing Issues

1. [ ] Rez binaries are intermixed with system binaries (see 3
2. [ ] CMake tests on Windows and PATH (see 4)

No Virtual Envrionment

From what I can tell, the embedded virtualenv was used to let packages gain absolute control over PYTHONPATH, whilst still being able to expose the rez Python package to rez env.

• Reference 1
• Reference 2

Now that there is no virtual environment, rez is made available via the site-packages of the target Python install.

$ c:\python27\lib\site-packages\rez

But that means you can't install Rez elsewhere anymore, which can be a problem if you host Rez itself as a package.

To solve that, I can think of two options.

1. Use your own virtualenv
2. Append sys.path.

(1) is simple enough, as you can install any PyPI package in a virtualenv, including this Rez from this PR.

(2) is a feature addition, as far as I can tell, via e.g. adding a site package path, and inserting to sys.path.insert(0,os.path.dirnam(rez)` on entering an environment.

So the question is, can the original problem be resolved (pun!) by either (1) or (2)?

PATH

On Linux, the environment inherited by Rez is coming the call sh echo $PATH which reads its PATH from the usual suspects.

~/.bash_profile
~/.bash_login
~/.profile
~/.bashrc

On Windows, prior to this PR, PATH was coming from the Windows registry; in similar vain to the above Bash scripts.

In this PR however, it's coming from os.environ["PATH"] for the sole purpose of enabling tests involving CMake to pass on Windows.

Is Production Install

Previously, a file called .is_production_install was embedded in the Scripts\rez directory alongside the Rez binaries. I was placed there to help identify the install having been made via install.py, which in turn was designed to solve the aforementioned issue of PATH.

With this PR, that safety guard is off per default, and made possible by installing into your own virtual environment.

$ python -m virtualenv my_rez_env
$ my_rez_env\scripts\activate
(my_rez_env) $ pip install bleeding-rez

The advantage is two-fold.

1. Rez on Pip, improve adoption and reduce learning curve
2. Simplified getting-started process for new users

I'd imagine the README would then be split between a "I just want to take Rez for a spin" and "I like Rez, now I want to use it in my large corp", whereby the latter involves a virtual env.

Post-merge

In order to test deployment, I've deployed to my account under a different name and version.

Once merged, you'll need to:

1. Create an account on AppVeyor
2. Replace my username and password in appveyor.yml

[instinct-vfx]

Stage 1 may bring up an issue for some depending on how Rez is provided to end-users. For us this won't be an issue i think but for reference and discussion i would like to add some information.

AFAIK (and that is not that far, so there may be other reasons?) the main reason for patching the executables in the installer is to have a single directory with ONLY the Rez commands. That way you can safely add this folder to %PATH% to provide Rez commands to end-users.

If the commands reside in Scripts\ (or bin\ on linux) adding that folder to %PATH% also adds all other scripts of the specific python version to %PATH%. With multiple python versions installed this may be a problem.

I am pretty sure fetching PATH from the current session creates unexpected problems. I have been bitten by the implementation before, but i think that weakening the separation between Rez's own execution environment and the environments it creates is not a good idea. I'd like to hear Allan's feedback here. I think i would prefer to just enable cmake usage without using vcvarsall. If you have to manually select a version that could also happen inside a resolved shell, no?

[mottosso]

Thanks @instinct-vfx just FYI this PR isn't actually on the Rez repo yet, so we'll possibly have to re-have our conversation later. I'm keeping this here to format my WIP markdown message before submitting.

the main reason for patching the executables in the installer is to have a single directory with ONLY the Rez commands.

Yes, this was my impression too. At the moment, you'd get Rez, but also anything else you already had in your Scripts/ directory which isn't very clean.

In this case, I think one could still safely use their own virtualenv (i.e. no need to embed/force one).

$ python -m virtualenv my_safe_rez
(my_safe_rez) $ pip install rez

Whereby you could then also copy/pasta the virtualenv directory wherever, like you can with it embedded.

I am pretty sure fetching PATH from the current session creates unexpected problems.

Yes, I expect to discover those problems first-hand; at the moment, it solved at least one issue and at surface level appears a step in the right direction.

Production-wise, it would be identical to fetching from the system PATH in all cases but one; when the user has (1) entered a shell, (2) made some changes, and (3) then entered Rez. It's the (2) that's the killer, but that's highly manageable I would imagine? I.e. on automated tasks to a render farm, a shell would be spawned (inheriting system PATH) and Rez immediately executed. I.e. on an app launcher script, the app launcher itself would control whether (or not) to modify PATH prior to calling Rez. So I really don't see the problem here, other than being careful just because we can.

[instinct-vfx]

I think that from a Rez point of view the problem is not really (2) as that's the whole point (the user making changes to control behavior). The problem is, that rez commands are also available from within Rez created shells. And at that point things can get really ugly i assume (and that's really just an assumption, i'd like to hear Allan's thoughts!)

[mottosso]

Having seen this particular issue of PATH around the mailing list and several GitHub issues over the past few days, I've changed my mind. It appears well thought through, and there's no reason not to keep it, other than fixing the tests that break with out it.