Support for deep merging of ocio configs

[dekekincaid]

I would like to propose the ability to deep merge config.ocio files. Now with new features such as the NamedTransform, both facilities and app developers would greatly benefit the ability to do this.

For a little background. At facilities we often have a main facility ocio config and then subsequent show level configs. Some facilities even go down the the sequence/shot or even the artist role. This makes it very onerous to update configs on a show. Let's say you need to add a top level colorspace to the show you then have to regenerate all the subsequent configs. We use deepmerge for most of our facility preferences but obviously since OCIO supports more then just python we can't simply use that. https://pypi.org/project/deepmerge/

Facility Benefits:

• Full facility config with Show/Seq/Shot level config can be a thin layer of only overrides to Displays/Views and any additional colorspaces specific to that level.
• NamedTransforms can be a separate config from the facility and can be hidden at certain levels.
• Colorspaces and transforms in general could be more modular rather then having one monolithic config.

App developers:

• Ability to provide a set of app specific colorspaces which are static, ie Nuke's base Colorspace node could in theory be its own ocio config with Nuke namespaced colorspaces in order to not conflict with others.
• DCC's could have a set of roles/colorspaces that always exist and facilities don't need to know that they need to insert them into their facility config. Currently if these are missing many parts of the software breaks, ie Mari's colorspace roles are used to set the default colorspaces for different data types.

[doug-walker]

Thanks for posting this @dekekincaid ! This topic seem to be coming up a lot and it would be great to have some discussions about this at the TSC meetings to see if people are on the same page about how this might work.

As you know, we took the first steps in this direction with the introduction of the interchange roles and the getProcessor calls that take a pair of configs. The next (and harder) question is how to actually merge a config (or part of a config) into another one.

It would be great to have some example configs to get a better grasp of the desired workflow.

[dekekincaid]

I'll work up some example configs based on how I was thinking about them working.

[doug-walker]

@dekekincaid , we would like to start working on this feature. Here is the proposal I made at today's TSC meeting.

If you have some example configs you could post, that would be very helpful.

And if other people in the community have thoughts, now would be a good time to discuss. You may either put comments in the Google Slides above or post them here. Thanks!

[KevinJW]

So as an example of the kind of thing users might 'add' to a base config I've manually created something similar to what I've seen done in v1 for a very simple show with just a single camera space, I've extracted the 'differences' vs a base config, it isn't necessarily best practice especially not in v2 but it is the kind of thing real life ends up with due to different applications only supporting some actions if things are in a colour space etc.

search_path:
  "grades/$SCENE/$SHOT/$ELEMENT:
  grades/$SCENE/$SHOT:
  grades/$SCENE:
  grades:
  luts:
  /my/global/config/location/luts/no-op"

roles:
  show_turnover: Input - RED - Linear - REDWideGamutRGB
  show_grading: Input - RED - REDLog3G10 - REDWideGamutRGB
  show_delivery: Input - RED - Linear - REDWideGamutRGB

environment:
  SCENE: ""
  SHOT: ""
  ELEMENT: ""

displays:
  Rec709:
    - !<View> {name: Show Look, colorspace: showLook}
    - !<View> {name: Show Look Graded, colorspace: showLookGraded}

colourspaces:

  - !<ColorSpace>
    name: showLook
    family: "Output"
    equalitygroup: ""
    bitdepth: 16ui
    description: |
      Simulation of show look output (contextual)
    isdata: false
    allocation: uniform
    allocationvars: [0.0, 1.0]
    from_reference: !<GroupTransform>
      children:
        - !<ColorSpaceTransform> {src: ACES - ACES2065-1, dst: show_grading}
        - !<FileTransform> {src: show_lut.cube, interpolation: tetrahedral}

  - !<ColorSpace>
    name: showLookGraded
    family: "Output"
    equalitygroup: ""
    bitdepth: 16ui
    description: |
      Simulation of show graded output (contextual)
    isdata: false
    allocation: uniform
    allocationvars: [0.0, 1.0]
    from_reference: !<GroupTransform>
      children:
        - !<ColorSpaceTransform> {src: ACES - ACES2065-1, dst: show_grading}
        - !<FileTransform> {src: primary.cdl, direction: forward}
        - !<FileTransform> {src: show_lut.cube, interpolation: tetrahedral}

Basically the idea is to have a hierarchy search path to allow per shot primary grades represented as CDLs found on the filesystem searching in the most specific locations first eventually if no primary.cdl is found for a given scene and shot it finds a "no-op" file. For a show a few roles are defined for 'knowing' the expected colour space of files being interchanged with external vendors, then s few "colour spaces" are defined for a graded and ungraded show LUT and these are used to construct a Rec 709/1886 view for display. The same colour space would be baked in prior to encoding a editorial reference.

This extra 'layer' means the actual config file is fairly generic but not quite as you still need to use show specific roles to find the colorspaces.

This can be generalised by migrating the grading from a n actual cdl file to any file type as we don't care about actual file extensions matching the file contents (which is a bit ugly). You could migrate the grade into a look, then you need to mark up the process space for the look with the colorspace, I've seen this done using additional file transforms which encode the ColorSpaceTransform to the grading space, this also allows you to have shot specific grading spaces but with limitations due to which file transforms you can use (refer to discussions in the meetings about a adding a colour space conversion file type etc)

As a config author I'd favour an include style mechanism similar in concept to https://docs.magnolia-cms.com/product-docs/6.2/Developing/Reusing-configuration/YAML-inherit-and-include.html or https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another

The merging code would dynamically resolve these at run time, if you wanted to bake this out then that would also be sensible, I suspect a mechanism similar to how the C/C++ preprocessor leaves location markers using #line might be handy.

As mentioned int he first meeting I also thing I would want a mechanism to prevent applications from adding things to a config without me knowing using some form of 'immutable' marker in my configs which would force the merge to fail.

[herronelou]

I was pointed at this issue to add our use case as I asked a related question on slack.

I think our use case is not too different from Deke's, we have a "template config", which we copy and modify for each project. The bulk of the config is nearly always the same, and all we do is override a couple of custom colorspaces and roles based on client requests, occasionally adding an extra colorspace for a specific purpose.

Currently, if a change happens in the template, it's not reflected in the other configs, as they were full copies, unless we also update the copy.

Our goal would be to have the possibility to have a single config for all projects, and an option to override/append tweaks in there for each project.

[doug-walker]

Thank you @KevinJW for the great example! I've started a section in the Wiki to share examples. I used your YAML above to create a base config and added an input and merged config. I created a similar example using a Look rather than roles, as you suggested. Please feel free to edit or comment on those or add new examples.

Thanks also for the links with various approaches to adding an !include feature to YAML. I gave some thought about how to get something similar in OCIO but I didn't find an elegant approach. (I added some slides in the deck linked above with my findings.) If you or others have insights into how to implement it, I'd be interested to hear ideas.