OCIO config environment variables distribution

Is your feature request related to a problem? Please describe. At the moment we are having working colorspace distribution with OCIO support, but we are still missing a part where EP, SEQ, SHOT environment variables occasionally used by production are set before a host application is started.

Describe the solution you'd like Pre-lauch hooks woflow should be used which would distill mentioned environment variables from the hierarchy of context and set them, so any OCIO config which is using such env vars will be able to use it. Idea here is to fill those variables if the any of parent above shot is sequence or episode assetType.

Additional context related PR to https://github.com/ynput/OpenPype/pull/4195

[cuID:OP-4724]

Could you clarify slightly further?

Does this mean the OCIO config would be pointed to using for example: /path/to/ocio/{EP}/{SHOT}? Does this mean that - with such a template - that ALL shots must have it like that? Or is it in reverse that the EP or SQ can have a custom OCIO env var config?

Or are you referring to it in another way of 'filling those variables'?

Im currently using the prehook to load OCIO setups in clarisse if there are overrides on the folder structure level. Setting up either global or per task level overrides works nice. Idealy this would go from content creation, lightning and rendering to compositing tasks. All should load the same OCIO to minimize the posible discrepancy in OCIO versions.

I have this function

def get_hierarchy_env(project_doc, asset_doc, skip_empty=True):
    """Returns an environment dictionary based on the hierarchy of an asset in a project

    The environment dictionary contains keys representing the different levels of the
    visual hierarchy (e.g. "SHOW", "SEASON", "EPISODE", etc.) and their corresponding
    values, if available.

    Args:
        project_doc (dict): A dictionary containing metadata about the project.
        asset_doc (dict): A dictionary containing metadata about the asset.
        skip_empty (bool): Whether to skip env entries that we don't have a value for.

    Returns:
        dict: An environment dictionary with keys "SHOW", "SEASON", "EPISODE", "SEQ",
            "SHOT", and "ASSET_TYPE". The values of the keys are the names of the
            corresponding entities in the hierarchy. If an entity is not present in the
            hierarchy, its corresponding key will not be present or have a value of None
            if 'skip_empty' is set to False.

    """
    visual_hierarchy = [asset_doc]
    current_doc = asset_doc
    project_name = project_doc["name"]
    while True:
        visual_parent_id = current_doc["data"]["visualParent"]
        visual_parent = None
        if visual_parent_id:
            visual_parent = get_asset_by_id(project_name, visual_parent_id)

        if not visual_parent:
            break

        visual_hierarchy.append(visual_parent)
        current_doc = visual_parent

    # Dictionary that maps the SG entity names from SG-leecher to their corresponding
    # environment variables
    sg_to_env_map = {
        "Project": "SHOW",
        "Season": "SEASON",
        "Episode": "EPISODE",
        "Sequence": "SEQ",
        "Shot": "SHOT",
        "Asset": "ASSET_TYPE",
    }

    # We create a default env with None values so when we switch context, we can remove
    # the environment variables that aren't defined
    env = {
        "SHOW": project_doc["data"]["code"],
        "SEASON": None,
        "EPISODE": None,
        "SEQ": None,
        "SHOT": None,
        "SHOTNUM": None,
        "ASSET_TYPE": None,
    }

    # For each entity on the hierarchy, we set its environment variable
    for parent in visual_hierarchy:
        sg_entity_type = parent["data"].get("sgEntityType")
        env_key = sg_to_env_map.get(sg_entity_type)
        if env_key:
            env[env_key] = parent["name"]

    # Fill up SHOTNUM assuming it's the last token part of the SHOT env
    # variable
    if env.get("SHOT"):
        env["SHOTNUM"] = env["SHOT"].split("_")[-1]

    # Remove empty values from env if 'skip_empty' is set to True
    if skip_empty:
        env = {key: value for key, value in env.items() if value is not None}

    return env

that I added on context_tools.py:change_current_context:

def change_current_context(asset_doc, task_name, template_key=None):
    """Update active Session to a new task work area.

    This updates the live Session to a different task under asset.

    Args:
        asset_doc (Dict[str, Any]): The asset document to set.
        task_name (str): The task to set under asset.
        template_key (Union[str, None]): Prepared template key to be used for
            workfile template in Anatomy.

    Returns:
        Dict[str, str]: The changed key, values in the current Session.
    """

    changes = compute_session_changes(
        legacy_io.Session,
        asset_doc,
        task_name,
        template_key=template_key
    )

    # Update the Session and environments. Pop from environments all keys with
    # value set to None.
    for key, value in changes.items():
        legacy_io.Session[key] = value
        if value is None:
            os.environ.pop(key, None)
        else:
            os.environ[key] = value

    ### Starts Alkemy-X Override ###
    # Calculate the hierarchy environment and update
    project_doc = get_project(legacy_io.Session["AVALON_PROJECT"])
    hierarchy_env = get_hierarchy_env(project_doc, asset_doc, skip_empty=False)
    for key, value in hierarchy_env.items():
        if value is None:
            os.environ.pop(key, None)
        else:
            os.environ[key] = value
    ### Ends Alkemy-X Override ###

    data = changes.copy()
    # Convert env keys to human readable keys
    data["project_name"] = legacy_io.Session["AVALON_PROJECT"]
    data["asset_name"] = legacy_io.Session["AVALON_ASSET"]
    data["task_name"] = legacy_io.Session["AVALON_TASK"]

    # Emit session change
    emit_event("taskChanged", data)

    return changes

This works perfectly thanks to our customization of the SG leecher that adds the sgEntityType on the assets since without that OP is not really aware of what entity is what. This should be possible to change now with Ayon

Thanks, @fabiaserra, for the solid solution. To implement it in Ayon, we need to address situations where a context change through the workfile tool is triggered. This workflow ties into active environment variables.

Another point to consider: We're not just limited to SHOT, SQ, EP as key names. We can use any of the environment variables. Currently, we're using AYON_PROJECT_ROOT_WORK, AYON_FOLDER_PATH, AYON_PROJECT_NAME, AYON_WORKDIR, AYON_HOST_NAME.

But since we are not having variable alternative for AYON_FOLDER_NAME we will need to have some sort of global prelaunch hook which will set it on demand. We might also need to get path only to a folder parent, perhaps for sequential luts so AYON_FOLDER_PARENT_PATH and AYON_FOLDER_PARENT_NAME keys might come useful.

At least that is how I understand it from here

These contexts are usually based on environment variables, but also allows on-the-fly context switching in applications that operate on multiple shots (such as playback tools)

Edit: Any environment variables could be also remapped inside of any ocio.config https://opencolorio.readthedocs.io/en/latest/guides/authoring/overview.html#environment

This means we could have our own environment declaration as such, and use them accordingly:

environment:
  AYON_FOLDER_NAME: $AYON_FOLDER_NAME
  AYON_FOLDER_PARENT_PATH: $AYON_FOLDER_PARENT_PATH
  AYON_FOLDER_PARENT_NAME: $AYON_FOLDER_PARENT_NAME 
  AYON_FOLDER_PATH: $AYON_FOLDER_PATH
  AYON_PROJECT_NAME: $AYON_PROJECT_NAME
  AYON_WORKDIR: $AYON_WORKDIR
  AYON_HOST_NAME: shell
  AYON_PROJECT_ROOT_WORK: $AYON_PROJECT_ROOT_WORK

This will allow us to use search paths following way:

search_path: |
  $AYON_PROJECT_ROOT_WORK/$AYON_FOLDER_PATH/colorspace/$AYON_HOST_NAME/luts:
  $AYON_PROJECT_ROOT_WORK/$AYON_FOLDER_PARENT_PATH/colorspace/luts:
  $AYON_PROJECT_ROOT_WORK/$AYON_PROJECT_NAME/colorspace/luts:
  luts:

And of course even resolve shot related .cc which are stored on project level:

- !<ColorSpace>
    name: srgb8
    family: srgb
    bitdepth: 8ui
    isdata: false
    from_reference: !<GroupTransform>
      children:
        - !<FileTransform> {src: ${AYON_FOLDER_PARENT_NAME}_${AYON_FOLDER_NAME}.cc}
        - !<ColorSpaceTransform> {src: lnh, dst: lg10}
        - !<FileTransform> {src: film_emulation.spi3d, interpolation: linear}

Thanks, @fabiaserra, for the solid solution. To implement it in Ayon, we need to address situations where a context change through the workfile tool is triggered. This workflow ties into active environment variables.

My solution already addresses that too, might be because I also call this function here?

def prepare_context_environments(data, env_group=None, modules_manager=None):
    """Modify launch environments with context data for launched host.

    Args:
        data (EnvironmentPrepData): Dictionary where result and intermediate
            result will be stored.
    """

    from openpype.pipeline.template_data import get_template_data
    from openpype.pipeline.context_tools import get_hierarchy_env

    # Context environments
    log = data["log"]

    project_doc = data["project_doc"]
    asset_doc = data["asset_doc"]
    task_name = data["task_name"]
    if not project_doc:
        log.info(
            "Skipping context environments preparation."
            " Launch context does not contain required data."
        )
        return

    # Load project specific environments
    project_name = project_doc["name"]
    project_settings = get_project_settings(project_name)
    system_settings = get_system_settings()
    data["project_settings"] = project_settings
    data["system_settings"] = system_settings

    app = data["app"]
    context_env = {
        "AVALON_PROJECT": project_doc["name"],
        "AVALON_APP_NAME": app.full_name
    }
    if asset_doc:
        context_env["AVALON_ASSET"] = asset_doc["name"]
        if task_name:
            context_env["AVALON_TASK"] = task_name

    ### Starts Alkemy-X Override ###
    # Get hierarchy environment variables (i.e., SEASON, SHOW, SEQ...)
    context_env.update(get_hierarchy_env(project_doc, asset_doc))
    ### Ends Alkemy-X Override ###

Another point to consider: We're not just limited to SHOT, SQ, EP as key names. We can use any of the environment variables. Currently, we're using AYON_PROJECT_ROOT_WORK, AYON_FOLDER_PATH, AYON_PROJECT_NAME, AYON_WORKDIR, AYON_HOST_NAME.

That's okay, but I think it's still important to have those env var keys as it's very useful to know at all times the specific folder types and being able to combine them (I have added them on template system too so I can use tokens like {shotnum} or {SEQ} to capitalize the sequence) to fit multiple workflows that require direct access to parent keys by type.

But since we are not having variable alternative for AYON_FOLDER_NAME we will need to have some sort of global prelaunch hook which will set it on demand. We might also need to get path only to a folder parent, perhaps for sequential luts so AYON_FOLDER_PARENT_PATH and AYON_FOLDER_PARENT_NAME keys might come useful.

I personally find that over-complicated and much harder to read than being explicit on using $SHOW, $SHOT, $EP, $SEQ...

This means we could have our own environment declaration as such, and use them accordingly:

environment:
  AYON_FOLDER_NAME: $AYON_FOLDER_NAME
  AYON_FOLDER_PARENT_PATH: $AYON_FOLDER_PARENT_PATH
  AYON_FOLDER_PARENT_NAME: $AYON_FOLDER_PARENT_NAME 
  AYON_FOLDER_PATH: $AYON_FOLDER_PATH
  AYON_PROJECT_NAME: $AYON_PROJECT_NAME
  AYON_WORKDIR: $AYON_WORKDIR
  AYON_HOST_NAME: shell
  AYON_PROJECT_ROOT_WORK: $AYON_PROJECT_ROOT_WORK

The only one you are remapping is AYON_HOST_NAME, there's not much point on redeclaring the ones that point to the same env var that already exists?

This will allow us to use search paths following way:

search_path: |
  $AYON_PROJECT_ROOT_WORK/$AYON_FOLDER_PATH/colorspace/$AYON_HOST_NAME/luts:
  $AYON_PROJECT_ROOT_WORK/$AYON_FOLDER_PARENT_PATH/colorspace/luts:
  $AYON_PROJECT_ROOT_WORK/$AYON_PROJECT_NAME/colorspace/luts:
  luts:

And of course even resolve shot related .cc which are stored on project level:

- !<ColorSpace>
    name: srgb8
    family: srgb
    bitdepth: 8ui
    isdata: false
    from_reference: !<GroupTransform>
      children:
        - !<FileTransform> {src: ${AYON_FOLDER_PARENT_NAME}_${AYON_FOLDER_NAME}.cc}
        - !<ColorSpaceTransform> {src: lnh, dst: lg10}
        - !<FileTransform> {src: film_emulation.spi3d, interpolation: linear}

That's cool but as I said I think we'd still prefer to be explicit in our config with the $EP, $SEQ, $SHOT hierarchies.

That's cool but as I said I think we'd still prefer to be explicit in our config with the $EP, $SEQ, $SHOT hierarchies.

The issue with $EP, $SEQ, $SHOT environment variables is that they require a lot of guessing in Ayon. The folder path and parent might work even it it doesn't follow OCIO documentation.

Our clients' experiences suggest that our approach would work for 95% of projects, which typically involve a sequence or shot hierarchy. However, for projects with seasons, episodes, sequences, and shots, we'd need to lay down some more groundwork. It's unclear if the effort is justified for the remaining 5%. It's not clear if these projects use looks from one OCIO config for the whole season or episode. They might use separate configs for each season or episode. If that's the case, this idea could work too.

ynput / OpenPype

OCIO config environment variables distribution #4299