Hard crashes while running 'ogsRender' command with proxyShapes

[jufrantz]

Describe the bug

We get hard crashes in some scenarios while running ogsRender command. We noticed it happens when the proxyShape's stage contains instances. It also manifests when working with display layers that have USD memberships. I am joining two reproductions even if the underlying cause is likely the same.

Steps to reproduce

ogsRender and instances

1. Create a mayaUsd proxyShape with some instances

import maya.cmds
import mayaUsd.ufe
import mayaUsd_createStageWithNewLayer
maya.cmds.loadPlugin("mayaUsdPlugin", quiet=True)

proxyShape = mayaUsd_createStageWithNewLayer.createStageWithNewLayer()
mayaUsd.ufe.getStage(proxyShape).GetRootLayer().ImportFromString('''#sdf 1
def Scope "Proto"
{
    def Sphere "Geo" 
    {
    }
}
def "Inst_1" (
    instanceable = True
    references = </Proto>
)
{
}
def "Inst_2" (
    instanceable = True
    references = </Proto>
)
{
}
''')

2. Execute a non-viewport VP2 render, e.g. by calling ogsRender

   maya.cmds.ogsRender()

3. It should crash, we get this stack trace:

   Autodesk::Maya::OpenMaya20230000::MHWRender::MFrameContext::applyViewTransform(Autodesk::Maya::OpenMaya20230000::MColor const&, Autodesk::Maya::OpenMaya20230000::MHWRender::MFrameContext::Direction) const
   pxrInternal_v0_22__pxrReserved__::ProxyRenderDelegate::_GetDisplayColor(std::pair<Autodesk::Maya::OpenMaya20230000::MColor, std::atomic<unsigned long> >&, char const*, bool, Autodesk::Maya::OpenMaya20230000::MColor const&)
   pxrInternal_v0_22__pxrReserved__::ProxyRenderDelegate::GetTemplateColor(bool)
   pxrInternal_v0_22__pxrReserved__::HdVP2Mesh::_UpdateDrawItem(pxrInternal_v0_22__pxrReserved__::HdSceneDelegate*, pxrInternal_v0_22__pxrReserved__::HdVP2DrawItem*, pxrInternal_v0_22__pxrReserved__::HdVP2DrawItem::RenderItemData&, pxrInternal_v0_22__pxrReserved__::HdMeshReprDesc const&, pxrInternal_v0_22__pxrReserved__::TfToken const&)
   pxrInternal_v0_22__pxrReserved__::HdVP2Mesh::_UpdateRepr(pxrInternal_v0_22__pxrReserved__::HdSceneDelegate*, pxrInternal_v0_22__pxrReserved__::TfToken const&)
   pxrInternal_v0_22__pxrReserved__::HdVP2Mesh::Sync(pxrInternal_v0_22__pxrReserved__::HdSceneDelegate*, pxrInternal_v0_22__pxrReserved__::HdRenderParam*, unsigned int*, pxrInternal_v0_22__pxrReserved__::TfToken const&)
   ...

ogsRender and displayLayer

1. Create a mayaUsd proxyShape with a simple stage

import maya.cmds
import pxr.UsdGeom
import mayaUsd.ufe
import mayaUsd_createStageWithNewLayer
maya.cmds.loadPlugin("mayaUsdPlugin", quiet=True)

proxyShape = mayaUsd_createStageWithNewLayer.createStageWithNewLayer()
pxr.UsdGeom.Sphere.Define(mayaUsd.ufe.getStage(proxyShape), "/Sphere")

2. Create a displayLayer containing /Sphere with template display style

   maya.cmds.select(f"{proxyShape},/Sphere")
   displayLayer = maya.cmds.createDisplayLayer()
   maya.cmds.setAttr(f"{displayLayer}.displayType", 1)

3. Call ogsRender

   maya.cmds.ogsRender()

4. It should crash, we get a similar stack trace:

   Autodesk::Maya::OpenMaya20230000::MHWRender::MFrameContext::applyViewTransform(Autodesk::Maya::OpenMaya20230000::MColor const&, Autodesk::Maya::OpenMaya20230000::MHWRender::MFrameContext::Direction) const
   pxrInternal_v0_22__pxrReserved__::ProxyRenderDelegate::_GetDisplayColor(std::pair<Autodesk::Maya::OpenMaya20230000::MColor, std::atomic<unsigned long> >&, char const*, bool, Autodesk::Maya::OpenMaya20230000::MColor const&)
   pxrInternal_v0_22__pxrReserved__::ProxyRenderDelegate::GetTemplateColor(bool)
   pxrInternal_v0_22__pxrReserved__::MayaUsdRPrim::_GetHighlightColor(pxrInternal_v0_22__pxrReserved__::TfToken const&, pxrInternal_v0_22__pxrReserved__::HdVP2SelectionStatus)
   pxrInternal_v0_22__pxrReserved__::MayaUsdRPrim::_GetHighlightColor(pxrInternal_v0_22__pxrReserved__::TfToken const&)
   pxrInternal_v0_22__pxrReserved__::HdVP2Mesh::_UpdateDrawItem(pxrInternal_v0_22__pxrReserved__::HdSceneDelegate*, pxrInternal_v0_22__pxrReserved__::HdVP2DrawItem*, pxrInternal_v0_22__pxrReserved__::HdVP2DrawItem::RenderItemData&, pxrInternal_v0_22__pxrReserved__::HdMeshReprDesc const&, pxrInternal_v0_22__pxrReserved__::TfToken const&)
   pxrInternal_v0_22__pxrReserved__::HdVP2Mesh::_UpdateRepr(pxrInternal_v0_22__pxrReserved__::HdSceneDelegate*, pxrInternal_v0_22__pxrReserved__::TfToken const&)
   pxrInternal_v0_22__pxrReserved__::HdVP2Mesh::Sync(pxrInternal_v0_22__pxrReserved__::HdSceneDelegate*, pxrInternal_v0_22__pxrReserved__::HdRenderParam*, unsigned int*, pxrInternal_v0_22__pxrReserved__::TfToken const&)
   ...

Specs (if applicable):

• CentOS 7.8
• Maya 2023.3
• maya-usd 0.25
• USD 22.11

[pierrebai-adsk]

Here is a temporary work-around: run the following MEL command (or equivalent in Python) before ogsRender:

colorManagementPrefs -e -outputTransformEnabled 1 -outputTarget "renderer";

[jufrantz]

Thank you @pierrebai-adsk,

I gave it a try, but the workaround appears to be effective only when color management is turned on. Unfortunatly in our case it happens in un-color-managed scenes.

Could this be a bug in the OpenMaya sdk ? If so, I might submit a report on Maya's support platform.

[pierrebai-adsk]

It is a bug in the Maya SDK. I fixed it, but it will be only available in a future release and I cannot vouch which Maya version will have it or not. I'm no color management expert, IDK if there would be a way to create a "pass-through" color correction that would not affect the color when applied.

[jufrantz]

Ok great. Would you be able to share the ID for the maya fix? Thanks

[pierrebai-adsk]

It was tracked internally as EMSUSD-1083.