Re-Implement Nuke containerizing

davidlatwe commented 4 years ago

Issue/Problem

Loaded nodes were not actually containerized, only data imprinted.
Due to the above, current method for listing container is scaning all nodes and using try-except to find out which node is able to be parsed. I think this could be improved.
Does avalon-core need container data updating method to be implemented here (pipeline.update_container) ? Might need to be disscussed.

Proposal

Containerizing

Using a Group node as an Avalon main container, and after Loader loads subset into root space or anyother sub-space, clone the subset into the container with data imprinted.

The goal of grouping and cloning loaded resources into one place, was to provides a certain block for maintaining. Meanwhile, let artist able to interact those node in place.

Example code:

AVALON_CONTAINERS = "AVALON_CONTAINERS"
def containerise(name,
                 namespace,
                 node,
                 context,
                 loader=None,
                 data=None):
    main_container = nuke.toNode(AVALON_CONTAINERS)
    if main_container is None:
        main_container = nuke.createNode("Group")
        main_container.setName(AVALON_CONTAINERS)
        # Maybe tweaking the tile_color and font_size
        ...

    # Format a container name
    container_name = "{}_{}".format(name, suffix or "CON")
    node.setName(container_name)

    # Clone it into main container
    main_container.begin()
    nuke.clone(node)
    main_container.end()

    # Imprint data and other stuff
    data = {
        "schema": "avalon-core:container-2.0",
        "id": AVALON_CONTAINER_ID,
        "name": name,
        "namespace": namespace,
        "loader": str(loader),
        "representation": str(context["representation"]["_id"]),
    }
    ...

Note that the third arg node is one nuke.Node object, not like in other hosts' implementation were a list of nodes.

Listing containers

If above make sense, now we have a place to look up all containers by the id knob.

Like..

def ls():
    main_container = nuke.toNode("AVALON_CONTAINERS")

    for node in nuke.allNodes(group=main_container):
        # Only look for nodes in main container
        knob = "root.AVALON_CONTAINERS.%s.id" % node.name()
        if nuke.exists(knob) and nuke.knob(knob) == AVALON_CONTAINER_ID:
            data = parse_container(node)
            # Collect custom data via `config.collect_container_metadata`
            ...
            yield data

Or, impelement lib.lsattr just like other hosts and ls from there

# In lib.py
def lsattr(attr, value=None):
    if value is None:
        nodes = nuke.allNodes(recurseGroups=True)
        return [n for n in nodes if n.knob(attr)]
    return lsattrs({attr: value})

def lsattrs(attrs):
    matches = set()
    nodes = nuke.allNodes(recurseGroups=True)
    for node in nodes:
        for attr in attrs:
            knob = node.knob(attr)
            if not knob:
                continue
            elif knob.getValue() != attrs[attr]:
                continue
            else:
                matches.add(node)
    return list(matches)

# In pipeline.py
def ls():
    containers = list()
    containers += lib.lsattr("id", AVALON_CONTAINER_ID)

    for container in sorted(containers):
        data = parse_container(container)
        # Collect custom data via `config.collect_container_metadata`
        ...
        yield data

Which you prefer ?

But in both ways, they all might break the compatibility. Because the data entry were possibly already been prefixed due to lib.set_avalon_knob_data.

So the id could actually be avalon:id or with other prefix.

What do you think ? Does above make sense to you and in production ? Hope we could have a discuss about this big change. :persevere:

tokejepsen commented 4 years ago

clone the subset

I dont know how stable cloning is in the later versions but in Nuke 9 is was definitely very unstable and could corrupt the Nuke scripts. That being said I have not come up with a solution to the problem is sharing nodes, that makes it easy to update while maintaining the users edits. I would be up for trying cloning again, to gain the ability to share nodes.

davidlatwe commented 4 years ago

Sorry for late reply :sweat_smile:

I dont know how stable cloning is in the later versions but in Nuke 9 is was definitely very unstable and could corrupt the Nuke scripts.

Yeah... I have talked with a few senior comp artists here and they have the same impression about cloning nodes in Nuke 9.

Just scanning all nodes to find loaded subsets via implementing lib.lsattr would be the simplest and most stable option here. And I imaging no matter how many nodes created from one Loader.load action, only imprint Avalon container data into one node (usually the most downstream one).

I would be up for trying cloning again, to gain the ability to share nodes.

Really like the idea of cloning nodes, but hard to reproduce the bug to know how to avoid it :( No artists here know how exactly cloning node break the nuke script, could not find from googling too.

tokejepsen commented 4 years ago

only imprint Avalon container data into one node (usually the most downstream one).

Hmm, this will be tricky to update nodes, if an artist start to mess around with a node tree.

Instead of cloning we could also have an ID similar to how @BigRoy and Colorbleed tracks nodes in Maya. We could continue with using a group to store the original nodes, but have a copy outside with a corresponding ID.

Whether we use cloning or some other method the issue will be how to enable an override system in Nuke. In Maya overriding attributes and connections are part of the referencing system, so it's easy, but in Nuke we don't have an equivalent system for overriding knob values and node connections.

Has any thoughts about this?

davidlatwe commented 4 years ago

Instead of cloning we could also have an ID similar to how @BigRoy and Colorbleed tracks nodes in Maya. We could continue with using a group to store the original nodes, but have a copy outside with a corresponding ID.

Sounds a good idea !

Whether we use cloning or some other method the issue will be how to enable an override system in Nuke.

You mean like Maya's reference edit, for updating nodes with knobs' modifications get preserved ?

tokejepsen commented 4 years ago

You mean like Maya's reference edit, for updating nodes with knobs' modifications get preserved ?

Exactly.

jakubjezek001 commented 4 years ago

AVALON_CONTAINERS = "AVALON_CONTAINERS"

Perhaps it is silly question but is this only one container as it is string?

Anyway, I share the idea that all of avalon tracked nodes should be containerized in group and imprinted.

Really like the idea of cloning nodes, but hard to reproduce the bug to know how to avoid it :( No artists here know how exactly cloning node break the nuke script, could not find from googling too.

We could add our own avalon clone menu item to menu/edit could be overwritten as duplicating with expression linked to hidden knobs As the image example suggested the expression is in hidden knob _attr1_Group1 (visible in the example)

tokejepsen commented 4 years ago

We could add our own avalon clone menu item to menu/edit could be overwritten as duplicating with expression linked to hidden knobs

Could you elaborate on this? Not sure I follow the workflow.

jakubjezek001 commented 4 years ago

Not sure I follow the workflow.

It will behave the same way as cloning (included link arrow) but you will be able to overwrite the driving values and it will work on groups. _attr1_Group1 is visible in the example but it can be hidden. The cream colored rectangular annotation is the expression used. The linked hidden knob will be driving insides nodes in group.

Is it more clear now?

tokejepsen commented 4 years ago

We could add our own avalon clone menu item to menu/edit could be overwritten as duplicating with expression linked to hidden knobs

Interesting solution! I see two issues initially though:

The linking lines could get messy in the Nukescript. You might be able to hide the (green) linked connections though somehow?
A plus expression would get confusing for users. If you input a value of 0.1 and you get a result of 0.2.

A workflow for referencing nodes and overrides could also be to compared values when updating the nodes within a group. Let me elaborate:

The user has an (Avalon) group in the scene with a Blur node inside. A copied Blur node is outside with an ID tagged the same as the Blur node inside the Avalon group.
The user updates the Avalon group. We compare the Blur node values inside the Avalon group with the one outside, before updating. This will tells us which knobs has been changed by the user, so those knobs can be ignored. The rest of the knobs will just get updated.

Lastly another solution would be use gizmos, but instead of making a group of nodes one gizmo with exposed parameters, we could make each individual node a gizmo. Gizmos can be simple file path references, so does not need to uniquely named globally.

mkolar commented 4 years ago

A workflow for referencing nodes and overrides could also be to compared values when updating the nodes within a group. Let me elaborate:

I like this a lot. Could be very robust possibly and not particularly hard to implement either.

davidlatwe commented 4 years ago

Okay, here is the code for updating referenced nodes. It seems workable !


import contextlib
import nuke

def get_copies(source):
    """Return nodes that has same 'avalonId' as `source`"""
    def get_id(node):
        id_knob = node.knobs().get("avalonId")
        return id_knob.value() if id_knob else None

    copies = list()
    source_id = get_id(source)
    if source_id:
        type = source.Class()
        copies = [node for node in nuke.allNodes(type, recurseGroups=True)
                  if get_id(node) == source_id and node is not source]
    return copies

@contextlib.contextmanager
def sync_by_id(nodes):
    """Context for Syncing node knobs that haven't been modified"""
    def is_knob_eq(knob_a, knob_b):
        return knob_a.toScript() == knob_b.toScript()

    def sync_knob(knob_a, knob_b):
        script = knob_a.toScript()
        knob_b.fromScript(script)

    staged = dict()
    origin = dict()

    # Collect knobs for updating
    for node in nodes:
        targets = list()

        sources = node.knobs()
        origin[node] = sources

        for copy in get_copies(node):
            for name, knob in copy.knobs().items():
                if name not in sources:
                    continue
                # Only update knob that hasn't been modified
                if is_knob_eq(sources[name], knob):
                    targets.append(knob.fullyQualifiedName())

        if targets:
            staged[node] = targets

    try:
        yield  # Update `nodes`

    finally:
        # Sync update result to all copies
        for node, targets in staged.items():
            updates = origin[node]

            for knob in targets:
                copied, knob = knob.rsplit(".", 1)
                copied = nuke.toNode(copied)

                sync_knob(updates[knob], copied[knob])

And this is my testing example

# 1. Create a transform named "Transform1", and set a random animation to "translate" knob
# 2. Add and set "avalonId" knob to "Transform1"
# 3. Copy Transform1

src_node = nuke.toNode("Transform1")
dst_node = nuke.toNode("Transform2")

src_node["label"].setValue("Source")
dst_node["label"].setValue("Copied")

with sync_by_id([src_node]):
    # Updating animation of "Transform1.translate"
    src_node["translate"].setValueAt(100, 5)

# Transform2's "translate" will be updated, but not "label"

Anyone able to test this ?

getavalon / core