search

aws-deadline / deadline-cloud-for-maya

AWS Deadline Cloud for Maya
Apache License 2.0
11 stars 16 forks source link

Bug: Submitter/Adaptor fails to open scene with "... is a layer from a referenced scene and is not allowed to be the current layer" #160

Closed epmog closed 3 months ago

epmog commented 4 months ago

Expected Behaviour

Both the submitter and adaptor should not attempt to switch to a render layer that they are not allowed to switch it to.

Current Behaviour

When launching the submitter, a pop-up appears with a stack trace:

Exception caught:
Traceback (most recent call last):
  File "*REDACTED*/deadline/client/ui/__init__.py", line 38, in gui_error_handler
      yield
  File "*REDACTED*/deadline/maya_submitter/mel_commands.py", line 66 in doIt
      DeadlineCloudSubmitterCmd.dialog = show_maya_render_submitter(
  File "*REDACTED*/deadline/maya_submitter/maya_render_submitter.py", line 473, in show_maya_render_submitter
      set_current_render_layer(render_layer_name)
  File "*Redacted*/deadline/maya_submitter/render_layers.py", line 43, in set_current_render_layer
      maya.cmds.editRenderLayerGlobals(currentRenderLayer=render_layer_name
RuntimeError: Maya command error

with this in maya console:

Warning: '*REDACTED*:defaultRenderLayer' is a layer from a referenced scene and is not allowed to be the current layer.

In the adaptor, when it tries to set the current render layer:

ADAPTOR_OUTPUT: STDERR: Warning: 'defaultRenderLayer1' is a layer from a referenced scene and is not allowed to be the current layer.
ADAPTOR_OUTPUT: STDERR: Traceback (most recent call last):
ADAPTOR_OUTPUT: STDERR:   File "/tmp/openjd/session-340a6dd6816d4f56ad606fa7e24e672ecbc8nkar/venv/lib/python3.9/site-packages/deadline/maya_adaptor/MayaClient/maya_client.py", line 70, in <module>
ADAPTOR_OUTPUT: STDERR:     main()
ADAPTOR_OUTPUT: STDERR:   File "/tmp/openjd/session-340a6dd6816d4f56ad606fa7e24e672ecbc8nkar/venv/lib/python3.9/site-packages/deadline/maya_adaptor/MayaClient/maya_client.py", line 66, in main
ADAPTOR_OUTPUT: STDERR:     client.poll()
ADAPTOR_OUTPUT: STDERR:   File "/tmp/openjd/session-340a6dd6816d4f56ad606fa7e24e672ecbc8nkar/venv/lib/python3.9/site-packages/openjd/adaptor_runtime_client/client_interface.py", line 181, in poll
ADAPTOR_OUTPUT: STDERR:     self._perform_action(action)
ADAPTOR_OUTPUT: STDERR:   File "/tmp/openjd/session-340a6dd6816d4f56ad606fa7e24e672ecbc8nkar/venv/lib/python3.9/site-packages/openjd/adaptor_runtime_client/client_interface.py", line 202, in _perform_action
ADAPTOR_OUTPUT: STDERR:     action_func(a.args)
ADAPTOR_OUTPUT: STDERR:   File "/tmp/openjd/session-340a6dd6816d4f56ad606fa7e24e672ecbc8nkar/venv/lib/python3.9/site-packages/deadline/maya_adaptor/MayaClient/render_handlers/arnold_handler.py", line 83, in set_render_layer
ADAPTOR_OUTPUT: STDERR:     maya.cmds.editRenderLayerGlobals(currentRenderLayer=render_layer_name)
ADAPTOR_OUTPUT: STDERR: RuntimeError: Maya command error
openjd_fail: Error encountered while starting adaptor: Maya encountered an error and was not able to complete initialization actions.
ERROR: Entrypoint failed: openjd_fail: Error encountered while starting adaptor: Maya encountered an error and was not able to complete initialization actions.

The two issues are related but slightly different:

  1. adaptor - we're passing in a display name, which Maya may resolve to a non-valid render layer.
  2. submitter - tries to get all non-referenced layers, but is unable to filter them out properly.

Reproduction Steps

I've attached a minimal reproduction case, but I've also included a README that I'll paste here so you can recreate it yourself without using my scene files:

Folder layout

1. open_this_scene.mb
2. imported_scene.mb
3. ref.mb

Scene layout:

1 imports 2 without a namespace
2 creates a reference to 3

What this does:

1. There are now 3 renderLayers returned in open_this_scene via maya.cmd.ls(type="renderLayer")
    maya.cmds.ls(type="renderLayer")
    # Result: ['defaultRenderLayer', 'imported_scene_defaultRenderLayer', 'ref:defaultRenderLayer']
2. If you filter out via `isNodeReferenced` you're left with:
    # Result: ['defaultRenderLayer', 'imported_scene_defaultRenderLayer']
3. The display names of these layers (melscript):
    renderLayerDisplayName "defaultRenderLayer"
    // Result: masterLayer
    renderLayerDisplayName "imported_scene_defaultRenderLayer"
    // Result: masterLayer
4. Swapping to 'otherscenewithref_defaultRenderLayer':
    maya.cmds.editRenderLayerGlobals(currentRenderLayer="imported_scene_defaultRenderLayer")
    # Warning: 'imported_scene_defaultRenderLayer' is a layer from a referenced scene and is not allowed to be the current layer.
    # Error: RuntimeError: file <maya console> line 3: Maya command error

referenced_render_layer_issue.zip

Code Snippet

Where the submitter grabs all render layers and attempts to switch to all of them

https://github.com/aws-deadline/deadline-cloud-for-maya/blob/mainline/src/deadline/maya_submitter/maya_render_submitter.py#L463-L473

Where the adaptor takes a display name and attempts to get the right unique name:

https://github.com/aws-deadline/deadline-cloud-for-maya/blob/mainline/src/deadline/maya_adaptor/MayaClient/render_handlers/default_maya_handler.py#L57-L73

Here's how you can visualize it without using the submitter/adaptor directly:

Error:

'defaultRenderLayer1' is a layer from a referenced scene and is not allowed to be the current layer.
import maya
maya.cmds.ls(":*", type="renderLayer")

# output
['defaultRenderLayer', 'defaultRenderLayer1']
import maya

def _get_render_layer_display_name(render_layer_name: str) -> str:
    return maya.mel.eval(f'renderLayerDisplayName "{render_layer_name}"')

for name in maya.cmds.ls(":*", type="renderLayer"):
    if not maya.cmds.referenceQuery(name, isNodeReferenced=True):
        print(f"{name} - {_get_render_layer_display_name(name)}")

# output
defaultRenderLayer - masterLayer
defaultRenderLayer1 - masterLayer

And then given the display name masterLayer Maya will then resolve it (mimicing the adaptor)

# data is "defaultRenderLayer"
render_layer_name = self.get_render_layer_to_render(data)
# render_layer_name is "masterLayer"
if render_layer_name:
    maya.cmds.editRenderLayerGlobals(currentRenderLayer=render_layer_name)

with the the referenced one (not sure how Maya determines which one it will resolve to, latest?), which then fails the render or submitter opening with:

'defaultRenderLayer1' is a layer from a referenced scene and is not allowed to be the current layer.
rmv commented 3 months ago

PR to fix this: https://github.com/aws-deadline/deadline-cloud-for-maya/pull/164