Autodesk: Billboards

[erikaharrison-adsk]

Description of Proposal

Proposal of billboards transformations based on extending the XformOp.

Supporting Materials

N/A

Contributing

• [X] I agree to and accept the Supplemental Terms.

[dgovil]

We have a similar schema for AR use here: https://developer.apple.com/documentation/arkit/arkit_in_ios/usdz_schemas_for_ar/preliminary_anchoringapi?language=objc

The differences I see are:

1. We provide a few different anchor types like image and face. I can understand those not being part of USD proper, though I wonder if we can include them (or reserve their use) since they are common alignment methods in several AR products (not just our own)
2. Our alignment is simply horizontal and vertical. If I understand your proposal properly, there is no secondary rotation alignment along the surface? e.g perpendicular or parallel to the plane. OR would that be taken care of by xformOp:align:plane ?

I do think your proposal captures most of what ours does for rotational alignment. Does this proposal consider positional anchoring as well? I was unclear on that aspect.

If it does include positional anchoring, I might suggest calling out (for a future proposal perhaps) the ability to specify an in scene prim to anchor to, not just an extrinsic camera/viewer.

[nvmkuruc]

I think there's a lot of potential for describing sections of the scene graph as screen anchored. One could imagine shipping assets with macbeth charts and context spheres as siblings to the default prim.

(
    defaultPrim = "model_root"
)
def "model_root" { }
def "context" {
    def "macbeth_color_chart" (references = @color_chart.usd@) { ... }
}

The proposal notes two transformative changes to Xformable-- making ops that are "extrinsic" and "compound" .

My main observation would be that many clients assume intrinsic Xformable. Functions like ComputeLocalToWorldTransform() for example don't provide any inputs to provide context. It seems like state attributes could be seen as a way to address this, but there's many times where there are zero or multiple viewers. At the same time, I'd expect that there would only be one set of billboard ops in a given branch of a hierarchy and that most authored Xformables would still be intrinsic.

Because extrinsic computations add complexity to reasoning about Xformable in usages like simulation, I think it's worth exploring anchoring as a separate typed schema. PointInstancer is an example of schema that similarly complicates the computation of transforms. Consumers can use the schema type as hook for determining whether or not they reason about the transform state. In our "visualization domain" proposal, we called this type of nondiegetic transform a VizAnchor.

A minor consideration regarding compound ops is that the third namespace component is currently used to describe the name of xform ops when an op is used multiple times. A new schema type wouldn't have this conflict.

[andy-shiue-autodesk]

1. We provide a few different anchor types like image and face. I can understand those not being part of USD proper, though I wonder if we can include them (or reserve their use) since they are common alignment methods in several AR products (not just our own)

It looks like, for the AR uses, you want to anchor a vitural object to a real object. I guess there are presumptions to have matched geometry (or bounding geometry?) between the vitual and real object? To support such anchoring, you probably need a combined transformation of tranlation, scaling and rotation. Currently proposed affixPosition, affixScale and align indivisually are too simple to support such transformation.

I think, instead of reserving the anchoring type, you can create a new xformOp to support such anchoring. I probably did not make it clear that more xformOps can and will be added to support other constrained transofrmations. I started recently calling the feature as constrained transformation since the extended xformOp framework supports more than just billboards.

Too bad that there is no way to create a plugin to add new xformOps right now.

[andy-shiue-autodesk]

2. Our alignment is simply horizontal and vertical. If I understand your proposal properly, there is no secondary rotation alignment along the surface? e.g perpendicular or parallel to the plane. OR would that be taken care of by xformOp:align:plane ?

What is the definitionof secondary rotation alignment? Is that something like a rotation constrained on an ordbit of an axis? That requires a new xformOp or new attribues of xformOp:align. I got a request on this axis constrained rotation and kinda inclined to add a new xformOp to support it.

Or you just want more control on the relation to the alignment reference? Like parallel or pependicular to the screen or even an authored angle between the prim and screen? In this case, we can add a new attribute in xformOp:align to support it.

[andy-shiue-autodesk]

Does this proposal consider positional anchoring as well? I was unclear on that aspect.

If it does include positional anchoring, I might suggest calling out (for a future proposal perhaps) the ability to specify an in scene prim to anchor to, not just an extrinsic camera/viewer.

xformOp:affixPosition can anchor a model space position to a screen sapce position. It cannot acnhor to other prims in the stage though. This extension is definetly worth a spot in the future proposal.

It is not clear to me how this can be supported in xformOp though. To anchor one prim to another, you would need the world coordinate of the anchor prim. That data is tricky to get in xformOp since the model2world transformation is computed during the sync phase. We will need to extend UsdGeomXformable to make this work; the thread safty and the denpendt dirtiness could be difficult to handle.

[spiffmon]

To build on @nvmkuruc 's comments, but starting with the quote from @andy-shiue-autodesk

Too bad that there is no way to create a plugin to add new xformOps right now.

It's actually a considerable feature to interchange and native-DCC-ingestion-of-USD that Xformable/xformOps remain intrinsic, simple, and "well supported" (in quotes because not every DCC supports everything the current xformOps compactly encode, but it's all straight-up and easy to translate). I know we haven't provided concrete guidance on how to handle more sophisticated encodings of transformations (e.g. constraints) in USD, but in a nutshell, the thinking is that Xformable remains simple, but its attributes will eventually be "drivable" by rigging/computation powered by OpenExec, whose effects could always be "baked out" in a caching step.

The other thing is that there needs to be a strict separation between "3D scene evaluation" and viewing state, for the reasons Matt mentioned.

What this suggests, in advance of OpenExec, is leveraging prim types that can be easily identified as living in screen-space, and therefore not affecting the scene's 3D bounds, and informing non-rendering USD clients that "here and below is data that only makes sense in the context of a render"; I think this also makes the Hydra update/invalidation process simpler, and no need to try to communicate data back to the UsdStage/evaluation. The Apple anchoring schema I think captures this, and it is very nicely formalized in the Add visualization schema proposal. . It would be fantastic to see Billboards explored in that space!

Thanks, @andy-shiue-autodesk !

[andy-shiue-autodesk]

@nvmkuruc @spiffmon Thank you for pointing out the concerns about the extrinsic states. They are valid points and I made a wrong choice to support the extrinsic states as runtime apis. The extrinsic states need to be "authorable" as part of the schemas.

I started the billboards project to support viewport manipulators. I was kinda limiting my thought process to have the renderer to integrate the billboards xformOp.

Here is what I think I should do (preliminary, of course):

(
    defaultContext = "global"
)
def Context "global" { ... }
def Context "active" { ,,. }

def Cone "cone1" {  ... }
def Cone "cone2"
{
    def "context" (references=[</active>]) { }

    ...
}

This design has extrinsic states part of the authored scene. Composition arcs should work well. External evaluation engine are allowed to compute the context states that drive the transformation.

Two of the big implementation challenges I had (accessing extrinsic states and managing the dirtiness of the transformation) will be addressed easily with the extrinsic states as embedded attributes. A big win for me.

Has this kind of context schemas been discussed before? It could be under UsdRender (as a new UsdRenderContext) or by extending UsdRenderSettings.

[andy-shiue-autodesk]

At the same time, I'd expect that there would only be one set of billboard ops in a given branch of a hierarchy and that most authored Xformables would still be intrinsic.

Yes, I expect people to have one list of billboards ops near the end of the transformation stack. But if consider transformations that are "constrained" in certain ways, math of certain constrained transformations may work as multiples in a transformation stack. There is no hard restriction on how billboards can be used, but the mathematics can make the end transformation unpredictable if people are creative.

[andy-shiue-autodesk]

Because extrinsic computations add complexity to reasoning about Xformable in usages like simulation, I think it's worth exploring anchoring as a separate typed schema. PointInstancer is an example of schema that similarly complicates the computation of transforms. Consumers can use the schema type as hook for determining whether or not they reason about the transform state. In our "visualization domain" proposal, we called this type of nondiegetic transform a VizAnchor.

This is a good point. I am inclining to call it constrained transformation (transformation that is constrained by extrinsic states). Considering the complexity comparing to the current xformOp, having it as a new schema makes a lot of sense.

A minor consideration regarding compound ops is that the third namespace component is currently used to describe the name of xform ops when an op is used multiple times. A new schema type wouldn't have this conflict.

I was considering to do the parsing magic on xformOp:xxx:use. That could look very ugly with the billboards xformop ;). You are right on having a new schema type to avoid this issue.

[spiffmon]

@nvmkuruc @spiffmon Thank you for pointing out the concerns about the extrinsic states. They are valid points and I made a wrong choice to support the extrinsic states as runtime apis. The extrinsic states need to be "authorable" as part of the schemas.

I actually disagree with this. It will be very common to have multiple viewports open and viewing the same stage. It seems neither clean nor practical (due to multi-threading concerns) for each viewport to author its own latest camera info into either the asme attributes, or even different ones. Part of the thinking in moving this into the "VizPrim" domain is in acknowledgment that it is a drawable whose state cannot be fully captured on the Stage itself, I'd argue.