[Feature] Cascading Package

[mottosso]

A fully backwards-compatible addition of one package augmenting another, as though each package was a CSS document added over another.

User Stories

1. As a DEVELOPER, I want to leverage Rez for project configuration management, so that I can avoid developing a second but similar system
2. As a DEVELOPER, I want a project or asset to override a studio-wide configuration, so that I can avoid duplication
3. As a USER, I want an environment tailored to my task, so that the right tools I require are made available
4. As a USER, I want predictable and dependable availability of tools, so that I can focus on using them

The Problem

You are 6 months into production, with 3 months remaining and everything is running along smoothly, when suddenly one of the shots require a change or addition to the pipeline that would adversely affect the rest of production; what do you do? Do you ignore the requirement and force the artist to conform to the existing pipeline, even though it would take him 4 weeks instead of 1? Do you give in, and sacrifice the stability and iteration time for the surrounding 50 artists?

If only there was a way to isolate the change to this one shot, thus saving 3 weeks at virtually no cost.

The Problem, Really

So what's stopping anyone from solving this problem today?

Normally, if a package exists on two or more paths, the search lasts until a first match is found. In order to change or add anything to a specific shot you have at least four options.

1. Separate System
2. Duplicate and Extend
3. Use a Bundle Package
4. Use Conditional Requirements

Option 1: Separate System

The most commonly used approach (it seems) is to develop a separate system that generates requirements for a task environment.

$ setproject alita
(alita) $ rez env maya -- maya

The setproject command would then establish the REZ_PACKAGES_PATH with relevant entries, like one for Alita-specific packages. The problem here being that setproject duplicates what Rez is already doing, such as version control, requirements and establishing an environment in a subshell, resulting in two similar systems to develop and maintain.

Option 2: Duplicate and Extend

You could copy the existing maya package into a shot, and use that in-place of the studio-wide maya.

$ set REZ_PACKAGES_PATH=/projects/alita/shots/ra1600/rez:$REZ_PACKAGES_PATH
$ rez env maya  # shot-specific Maya picked up

However the problem with this approach is that the shot would also need to duplicate an either large package (containing multiple gigabytes) or logic to handle an external reference on the local disk, e.g. c:\program files\autodesk\...

# /projects/alita/shots/ra1600/rez/maya/package.py
name = "maya"
version = "2018.0.2"

def commands():
    import os

    if system.platform == "windows":
        path = r"c:\program files\autodesk\maya2018\bin"

    elif system.platform == "linux":
        path = "/opt/maya2018/bin"

    assert os.path.exists(path), "Missing files: %s" % path
    env["PATH"].prepend(path)

Once duplicated, you can start making changes.

# /projects/alita/shots/ra1600/rez/maya/package.py
# name = "maya"
# version = "2018.0.2"

requires = [
    "shot_specific_requirement-2",
]

# def commands():
#   import os
# 
#     if system.platform == "windows":
#         path = r"c:\program files\autodesk\maya2018\bin"
# 
#     elif system.platform == "linux":
#         path = "/opt/maya2018/bin"
# 
#     assert os.path.exists(path), "Missing files: %s" % path
#     env["PATH"].prepend(path)

Duplicated lines grayed out for clarity

Option 3: Bundle

One straightforward method of increasing specificity for a project or asset is to "bundle" two or more packages together, and append additional requirements and variables.

maya
alita_maya
alita_vector_maya
nuke
alita_nuke
alita_vector_animation_nuke
houdini
alita_houdini
maya
lotr_maya
lotr_frodo_maya
...

Where each bundle contains what you might call "overrides" to the package it itself requires.

alita_maya/package.py

requires = [
    "alita-3",
    "maya-2018",
]

def commands():
    # extended commands

This has a few problems.

• Lots of permutations become increasingly difficult to manage
• Reusing alita_vector_maya for another asset is difficult, e.g. alita_vector_and_chiren_maya or alita_common_assets_maya
• The user must not forget to include the specific package in the request

# Before
$ rez env alita maya

# After
$ rez env alita_maya

Option 4: Conditional Requirements

Another option is to (1) make packages for your assets too and (2) add conditions to a project package.

$ rez env alita vector maya

# alita/package.py
name = "alita"

common_requires = [
    "base-1",
]

@late
def requires():
    if in_context() and "vector" in request:
        return this.common_requires + ["mgear"]

Like with Bundles, the problem here is that the if you wanted to add an override to a previously non-overridden environment, the user must not forget to include vector to the list of requirements.

A Solution

An Override Package is a special type of package that accumulates properties during a resolve or build.

~/packages/maya/package.py

name = "maya"
version = "2018"

def commands():
    env.PATH.prepend(r"c:\program files\autodesk\maya2018\bin")

alita/rez/maya/package.py

name = "maya"
version = "1.0"  # Version of override, rather than overridden package
override = True

requires = [
    "specific_package-1"
]

def commands():
    env.MAYA_ENABLE_LEGACY_VIEWPORT = "1"

Now if the first match - e.g. alita/rez/maya/package.py - carries override = True, then that package is added onto the next match found - ~/packages/maya/package.py.

Example

For a more elaborate example, the package maya exists at each of these 3 locations.

$ LEVEL1=/mnt/packages/prod               # Global packages
$ LEVEL2=/shows/alita/rez                       # Show overrides
$ LEVEL3=/shots/alita/assets/vector/rez  # Asset overrides
$ export REZ_PACKAGES_PATH=$LEVEL3:$LEVEL2:$LEVEL1  # NOTE: Overrides are added first
$ rez env maya

Before this feature

1. rez env maya is called
2. maya is found on LEVEL3
3. LEVEL3:maya is resolved
4. Search complete

After this feature

1. rez env maya is called
2. maya is found on LEVEL3, carrying override = True, search continues..
3. maya is found on LEVEL2, carrying override = True, search continues..
4. maya is found on LEVEL1, no override, search stops
5. LEVEL1:maya is resolved
6. LEVEL2:maya is resolved and added
7. LEVEL3:maya is resolved and added
8. Search complete

[bpabel]

There's another method that I've normally used that is similar to your "bundle" method

I would break out additional configuration options into their own packages. For example

maya
maya_enable_legacy_render_layers
maya_enable_legacy_viewport

You could achieve the same thing that way without the added complexity of adding a new type of package.

When a specific extension is needed, it would just get added to the launch command

rez-env maya-2018 maya_enable_legacy_render_layers 

I've always used a separate system for managing those launch commands -- either a config file for a launcher tool, or more recently, as fields in Shotgun.

[mottosso]

Thanks @bpabel, that works, but is it not the same as the "bundle" option? At least the cons apply here as well.

I've always used a separate system for managing those launch commands

Indeed, this feature could potentially remove the need for this kind of additional system.