Switching Representations with different loaders

[mkolar]

Problem

Currently we seem to be only supporting switching representations if they are all supported by the same loader. This effectively prevent using this for switching proxy geometries for render time proxies (geometry -> arnold standin for instance)

Solution

From my initial testing it seems that it would be enough to get the loader from the new representation being loaded, rather than container and all should be good. Now obviously if there are more loaders per representation, then there's the question which one to use.

Additional problem is that I have artists asking on a daily basis for being able to keep the representation but change the loader. For instance changing referenced alembic to gpu cache and back. This is very useful for huge scenes.

[BigRoy]

This is an amazingly interesting case, and a big plus if we can figure it out in a stable way.

Unfortunately this isn't trivial. When "switching" from a referenced Alembic GEO to a gpuCache it means that the Alembic GEO reference should actually be removed from the scene, right? Or what other special logic would you apply?

Whenever you remove the referenced Alembic GEO then any local changes to that geometry would be gone, any shader assignments, any tweaks or changes in positioning. As such, switching back and forth makes no sense.

What was your idea on tackling that?

Currently with "switch" it's all done on the Loader itself, as such, any data that needs to persist on the switch can be tracked there. It's up to the Loader (if not a Maya reference, as that would store reference edits even if failed anyway) to potentially store that data somewhere if it needs to remain after switching back and forth. However, if you do the same between different Loaders - then every loader would need to take into account any other loader and what kind of data it might want to preserve?)

In your case, if the removal of data is alright, it's a matter of calling "remove" on the existing instances and "load" on the new ones. Yes?

---

And yes, the other problem is knowing what Loader to switch to if a specific representation and subset supports multiple loaders. (If we are allowed to "switch Loader")

[mkolar]

When "switching" from a referenced Alembic GEO to a gpuCache it means that the Alembic GEO reference should actually be removed from the scene, right?

Correct. that's the only way I can think of at this point.

Whenever you remove the referenced Alembic GEO then any local changes to that geometry would be gone

we're aware of that and don't really have a good solution at this point to be honest. However we would keep the transform of the top group at the very least which is trivial.

At this point I'd approach it with a warning before the actual switching notifying the user that all his changes apart from the position will be lost.

the other problem is knowing what Loader to switch to

I thought a good solution might be adding a a 4th dropdown that lists available loaders, so the user can choose

[mkolar]

Alembic GEO then any local changes to that geometry would be gone, any shader assignments,

actually I had a though. Maybe we could store the changes that can be serialized (like ref edits) in an attribute on the container. So if artist switches back to original loader at later time we could reapply them.

For instance. I have model loaded using reference loader. When switching to gpu cache we take all ref edits and serialize them into json format and save to attribute.\ on container along the lines of

referenceLoader:{attributes: ......, connections: ....... etc.}

when switching back to reference, we try to find referenceLoader in the specified attribute data and apply all of them again. This would of course be quite limited to how far we can push it, but could most probably cover the most used cases. i.e. switching from reference to gpuCache and back, or in our case from reference to arnold standin and back

[mottosso]

Just to throw this in there, could a loader call another loader? That way, maybe there could be a "loader container" of sorts.

class LoaderA(...):
  ...

class LoaderB(...):
  ...

class LoaderContainer(...):
  def load(...):
    if some_expression:
      return api.find_loader("LoaderA").load(...)

Maybe wouldn't even have to be a "container", just one loader calling another.

the other problem is knowing what Loader to switch to if a specific representation and subset supports multiple loaders

Could it default to index[0], and bring up a popup/menu to select if there is any question?

[davidlatwe]

I think I could share something on this. :)

My implementation for switching proxy (GPU cache), take my model loader for example, was done by extracting not only model .mb but also the gpu cache during the publish, and the gpu cache gets wrapped into a mayaAscii file during that process in this way, which could be referenced later on :

gpu_files = ["modelDefault/v001/gpu.abc"]
output_path = "modelDefault/v001/gpu.ma"

MayaAscii_template = """//Maya ASCII scene
requires maya "2016";
requires -nodeType "gpuCache" "gpuCache" "1.0";
"""

gpu_node_template = """
$cachefile = `file -q -loc "{filePath}"`;  // Resolve relative path
createNode transform -n "{nodeName}";
createNode gpuCache -n "{nodeName}Shape" -p "{nodeName}";
    setAttr ".cfn" -type "string" $cachefile;
"""

gpu_script = ""
for gpu_path, node_name in gpu_files:
    gpu_path = gpu_path.replace("\\", "/")
    gpu_script += gpu_node_template.format(nodeName=node_name,
                                           filePath=gpu_path)

with open(output_path, "w") as maya_file:
    maya_file.write(MayaAscii_template + gpu_script)

And then I could swap the reference between model file and the gpu cache with the same model loader. Hope this helps ?

Not sure about other renderer specific like Arnold stand-in or VRay proxy, haven't gone that far, yet. But I think as long as we could wrap everything into a Maya referenceable format, should be good enough for this issue, right ?

[davidlatwe]

Had a deeper look at this since I was also trying to implement cross loader switch for switching in between Arnold Stand-In and model or rig.

I imaging the ultimate goal would be something like this :

        +----------+      +-------------+
    --> | xgenHair | <--> | gpuCacheHat | <..
        +----------+      +-------------+
..         4      4         4        4         ..
  \       /        \       /          \       /
   v     v          v     v            v     v
 +----------+      +---------+     +------------+
 | meshHair | <--> | meshHat | --> | hairSimRig | <-..
 +----------+      +---------+     +------------+

An XGen hair subset for example, can be switched to hair card model, gpu cache, or other representation of asset Hat.

We could switch freely as long as the chosen loader has been implemnted to be able to process the subset which was not loaded by itself.

Implementation proposal

Assume that we all using scene file referencing to load those subset.

Asset Switcher GUI

The switcher GUI must aware of all Loaders that matched with the selected representation. So for this issue:

the other problem is knowing what Loader to switch to if a specific representation and subset supports multiple loaders

I propose to implement like this to reslove:

• If only one loader that matched with the representation

  Display loader icon, and maybe loader name in tool hint

  

• If there are multiple loaders that matched

  Turn into question mark, and click LMB or RMB to show a drop-down menu for selecting the loader.

  

cbsceneinventory

Adding one more optional arg options for indecating which loader been selected.

# avalon.tools.cbsceneinventory.lib

def switch_item(...):
    ...
    options = dict()
    options["loader"] = Loader.__name__
    api.switch(container, representation, options)

api

And implementing one more method covered for loader to judge whether it could switch the subset currently loaded in scene.

# avalon.pipeline(api)

def switch(container, representation, options=None):

    options = options or dcit()
    cross_load = False

    # Assume they are Loader cls
    Loader_opt = options.get("loader")
    Loader = container["loader"]

    if Loader_opt is not None and Loader != Loader_opt:
        if not hasattr(Loader_opt, "covered"):
            raise RuntimeError("Cross load not supported.")
        cross_load = True
        Loader = Loader_opt

    if not hasattr(Loader, "switch"):
        raise RuntimeError("Switch not supported.")

    context = get_representation_context(representation)
    assert is_compatible_loader(Loader, context), "Not compatible."

    loader = Loader(context)

    if cross_load:
        # Whether this selected loader able to replace currently loaded subset to another.
        assert loader.covered(container, representation), "Not covered."

    loader.switch(container, representation, options)

---

With these, as long as the subset stay referenced, I think we could switching in between loaders without a problem.

Please let me know what you think :)

[davidlatwe]

Here's a implementation sneak preview :D

Switching from model to arnold stand-in and back.

[mkolar]

With these, as long as the subset stay referenced, I think we could switching in between loaders without a problem.

So when you load StandIn, it is already wrapped in maya file so you can reference it? Our ass loader create a new standin node and set's its paths to correct values. Functionally I quite like the changes.

[davidlatwe]

So when you load StandIn, it is already wrapped in maya file so you can reference it?

Yes, the .ass file gets wrapped right after it's been exported. The switching part so far no big problems, reference edits will be apply back after switching. But I haven't put into production yet, so much bug to fix right now..

Functionally I quite like the changes.

Thanks ! Those GUI stuff was built upon #376 (obviously), hope it gets merged soon.

[BigRoy]

Just to double check. So it's up to the new Loader to return whether the switch can be done from a container with a different loader.

What is good about this one is that it does not hardcode the idea of References for maya. It allows the studio configuration to "store" ref edits temporarily on a custom attribute in the container like: _edits, maybe even as dict per Loader. Then you could even start storing edits for things that are not references and in other hosts.

I'm a bit afraid of the conplexity and managing the multiple directions. With this I'd love to see an example that switches consistently between a ASS standin (non-referenced) and a Maya reference.

I think it's interesting to hear that you are wrapping the ASS in maya references. Doesn't that takeaway quite a large bit of the loading speed of the scene? As ass proxies I guess are most useful if you tend to have in 100s of them (or if they are super huge files). Whereas when you have 100s of references Maya tends to become a bit sluggish with loading, or isn't that the case anymore in recent releases? - might be a bit off-topic conversation though.

[davidlatwe]

Just to double check. So it's up to the new Loader to return whether the switch can be done from a container with a different loader.

Yes, but still wondering if there are any better name than covered.

I'm a bit afraid of the complexity and managing the multiple directions.

Perhaps (in my imaging) we could use that covered method to decide specifically on which asset or subset, or even version, that allowed to cross loader switching in run-time. By doing this intentionally limitation, should able to reduce some uncertainty. Does this make sense ?

With this I'd love to see an example that switches consistently between a ASS standin (non-referenced) and a Maya reference.

Here's a long GIF I tested on a animated rig from Mixamo.

Not sure why when I switching back to Rig from Ass, the rig require to refresh to pick up animation. :/ Edit: Sorry, the stand-in in above GIF is referenced, missed what you wished was non-ref.

Doesn't that takeaway quite a large bit of the loading speed of the scene?

Haven't think about that before actually, not often using stand-in here (we just begin to adopt Arnold). But that's a good point, hmmm..., maybe that depends on how often you want to switch things... need to think more on this.

[mkolar]

Doesn't that takeaway quite a large bit of the loading speed of the scene?

It does...When we have big scenes that are mostly built up with ass files, the difference trying to recreate this by hand noticeable indeed. I think the same will be true for vray proxies.

[davidlatwe]

When "switching" from a referenced Alembic GEO to a gpuCache it means that the Alembic GEO reference should actually be removed from the scene, right?

For this part, how about just unload that reference, and using a callback like this:

from maya.api import OpenMaya as om  # API 2.0

def before_load_reference(reference_node,
                          referenced_file,
                          clientData=None):
    """Check if the reference is okay to be reloaded"""
    if not cmds.objExists("DO_NOT_LOAD"):
        return True

    dep_fn = om.MFnDependencyNode()
    fn_node = dep_fn.setObject(reference_node)
    ref_node = fn_node.name()

    if ref_node in (cmds.sets("DO_NOT_LOAD", query=True) or []):
        # Or any other condition
        return False
    return True

# Register the callback
om.MSceneMessage.addCheckReferenceCallback(
    om.MSceneMessage.kBeforeLoadReferenceCheck,
    before_load_reference
)

With this callback, the reference will get reloaded only on certain condition, e.g., not being a member of "DO_NOT_LOAD" objectSet.

[davidlatwe]

Adding more input, not having a result, yet.

Using asset container node to hold import-only subset

On the part that switching imported subset to referenceable type or another import-only subset, currently experimenting on using Maya asset container node to hold the import-only subset.

By adding import-only subset nodes into a DAG type asset container node, it could provide a limited editing interface via exposing bonded attributes only, e.g., Arnold stand-in node's frame range attributes and hide the rest.

With that limitation, we can focus only on the attributes that we exposed while switching. And because we could use custom attribute name to bind the subset node's attribute like :

cmds.container("Hulk_01_AST", edit=True, publishAndBind=["node.anyAttr", "abs_name"])

Then we would able to implement a way to remap those attribute's connections and values to the subset that is being switched into scene, by using published attribute name "abs_name" as key.

After the connections and values' remapping and preserving is done, we could safely delete nodes in that asset container, since those were the only variations that artist could apply to. And hide it from outliner if it's being switched to a reference.