search

PixarAnimationStudios / OpenUSD

Universal Scene Description
http://www.openusd.org
Other
6.14k stars 1.22k forks source link

[Enhancement] Link USDPreviewSurface with an evolving industry standard PBR BSDF. #2119

Closed bhouston closed 1 year ago

bhouston commented 1 year ago

Description of Issue

I wrote this after a discussion with @meshula earlier in the week. He encourages contributions to the USD GitHub repository so I am taking the initiative and proposing this here. If this idea seems good I can try to create a PR.

The State of USD Preview Surface

Currently the USD Preview Surface has support for the following PBR BSDF parameters:

(An aside: Opacity, OpacityThreshold, Normal and Displacement are not truly PBR parameters, rather outside of the BSDF itself and can be used with any BSDF generally.)

A Great Start

This is a great start! Especially when the USD Preview Surface was first created. But today, this is a very limited PBR representation. Much of the industry has moved towards a richer representation. Tools and Companies (e.g. Unreal Engine, Unity, Autodesk, Dassault, Adobe, Three.js, Babylon.js, etc.) and Industry Standard groups (e.g. Khronos's PBR Next group) have developed very well specified and rich PBR models.

Here is a chart I created that compares current PBR models.

Hampering Interoperability in the Medium Term

The fact that USDPreviewSurface is behind the state of the art here is hampering a lot of interoperability of these PBR Models. Many systems are reliant on the USD Preview Surface for the material representation because they do not understand MaterialX or other proprietary BSDF models. Systems that will not likely interpret MaterialX in the near/medium term are Apple iOS Quicklook, Three.js, Babylon.js, Meta's Spark, Snap's Studio, Unity and Unreal Engine.

Solution Parameters

The question then becomes what PBR model should we use for USD Preview Surface? I would suggest some requirements - one picks a well designed, fully specified, standard from a standards body that has wide participation within the computer graphics industry.

Solution: PBR Next

I would like to propose that one could update the USDPreviewSurface to the ratified version of the PBR Next standard from Khronos Group as of the end of 2022. The Khronos Group has been working on PBR Next since 2019 with contributions from Autodesk, Adobe, Dassault Systems, Unity, and another 20 to 30 companies.

Some press releases of progress of the PBR Next initiative:

Proposed New Parameters

The PBR Next BSDF uses the above parameters that USD Preview Surface already has (and it very clarify specifies how they should be implemented) but it adds the following:

2020-Toy-car-model-illustrating-the-use-of-glTFs-three-new-PBR-extensions

2021-blog-glTF-3D-model-of-olives-under-a-glass-dome_1

Future Updates

PBR Next is an incrementally improving standard. It is working on each group of properties as a separate extension so that incremental progress can be made. There are 3 more interesting extension in the pipeline that will likely be ratified in the medium term - which could then necessitate another update of USD Preview Surface to sync up:

Implementation

Implementation can be provided both by the existing MaterialX BSDF for the PBR Next model. And I can help to some degree on a GL implementation based on how it is achieved in Three.js and Babylon.js.

Fidelity Testing

The PBR Next group also has fidelity tests. These ensure that the materials also look correct when rendered, which is a step further than just telling people how to do it -- this ensures that it is done correctly. If USD Preview Surface makes use of the same BSDF, then one could reuse these same fidelity tests.

Ease of Tool Adoption

Adoption of this within content creation tools is also eased if the same BSDF is used as glTF. Many of these tools already have full featured glTF exporters and importers. This means that they already support this BSDF or can convert to it. Also many tools have been converting their implementations of PBR to this specific BSDF. Three.js, Babylon.js, Filament and most recently Blender, are tools that have been adopting this specification.

I Can Make the Initial Contribution

I am asking for permission to contribute some of the work to move towards this goal. I am not asking for USD to do the initial lift to get here.

What is the Harm?

When thinking about this, we should consider any potential downsides to doing this. I have listed mostly upsides, but maybe there are downsides as well.

Steps to Reproduce

Export a model that uses Blender's native Principled BSDF materials from Blender to glTF, and it will have rich materials in Three.js, Android AR Viewer, Babylon.js.

Export to USD and it will only have a simplistic material definition in Apple iOS Quicklook and in Maya - most parameters will be lost. It would be nice to not have that happen any more.

System Information (OS, Hardware)

N/A.

Package Versions

N/A.

Build Flags

N/A.

sunyab commented 1 year ago

Filed as internal issue #USD-7817

spiffmon commented 1 year ago

Hi @bhouston - thanks for engaging on this! I'll start out by saying that Pixar believes that glTF assets should be easily consumable (e.g. by reference) as leaf nodes in a USD composition graph (there are a couple of groups working on the basics of such a plugin), and faithful surfacing is a key ingredient of that.

I spoke with several different groups in the USD ecosystem about this before responding, and there were a few common concerns that I'll enumerate here (which can serve as discussion points for Patrick's January meeting?). Firstly, it's important to note the motivation for the creation of UsdPreviewSurface, which is twofold:

  1. To facilitate material interchange, especially from DCC to game engine, circa 2017. Game engine shading systems have gotten alot more powerful and general since 2017, so clearly UsdPreviewSurface is no longer fulfilling this role as it once did.
  2. To strike a balance between representational power and computational complexity such that it can provide reasonable "preview shading" on potentially hundreds of thousands of objects in a rasterizing, glsl-based renderer, as USD scenes frequently scale to that size and well beyond. Our proximal inspiration for UsdPreviewSurface was an internal PBR shader that we'd used for some time as a target to bake down our fully-featured RenderMan PxrSurface-based materials for viewport rendering in OpenGL.

Given that PBR Next is a quite fully-featured uber-surface, its featureset is likely not a good candidate for being a successor or new version of UsdPreviewSurface, as there's nothing really "preview" about it! Instead, I think we'd be looking at adding an additional surface to the set of "Universal shading nodes" that ship with USD core. Earlier this year we had discussed, in the ASWF USD/MaterialX working group, adding a native implementation of standard_surface to USD, but decided that for our community's needs, in which the MaterialX/USD/Hydra integration is easily and often available, the additional complexity of adding a USD version over just dropping in a MaterialX material that uses standard_surface was not worth it. But as you point out, as USD's reach and role grow, there may be a mid-term need for a more powerful PBR that is native and universal in USD, as a sibling to UsdPreviewSurface.

So, let's say we adopted PBR Next... what's the story for scalability? If it becomes as easy as we believe it will be to construct complex USD scenes with many many referenced glTF (or native USD) models, and all of those are using PBR Next, we have a likely challenging interactivity problem in our Storm renderer, and even for smaller scenes, some people expressed concern about mobile-device compute/battery demands. USD's shading model actually incorporates the notion of having multiple material bindings for the same objects, to fulfill different purposes, and the two builtin purposes are full and preview. That means it would be possible to additionally bind guaranteed-cheaper materials that use UsdPreviewSurface, as "preview" materials that Storm and other viewport renderers would consume.

But one problem with that story is that doing a reasonable job going from Next to Preview may require up-front shader baking, which we can't count on happening at asset creation time, and which is too expensive to do JIT.

Even acknowledging that Autodesk is an active participant in the development of PBR Next, many people I talked to were puzzled why Autodesk's Standard Surface was not selected for this purpose, since:

The last concern I'll mention is related, a confusion about too many standards, as, in addition to Autodesk's Standard Surface, there is also Adobe's... have Autodesk and Adobe signalled they are willing to deprecate their models in favor of PBR Next? If not, how do we guide content creators?

Thanks again, and looking forward to a good discussion in January!

bhouston commented 1 year ago

Thank you @spiffmon for your time and it is a pleasure to meet you! I should preface this with the disclaimer that I am speaking for myself in all of these comments and not Khronos or glTF, but I figure I should respond now in order to move the conversation forward in preparation for the chat later this month.

Summary

You mention that USD Preview Surface is intended for fast previewing at scale in glsl-based rasterizers. You then mention that because the Khronos PBR Next material is a full featured Uber-material, it isn't really well suited for that. Then you say that at the high end, it conflicts with Autodesk Standard Surface, which has more support in the VFX space.

Coming from Different Perspectives

I think you are coming from a high end VFX perspective with goals for optimizing high end VFX workflows. I used to be a VFX guy myself (e.g. https://scholar.google.ca/citations?user=--UmWDUAAAAJ&hl=en), but these days I am more concerned about real-time interoperability now, especially between game engines and on the web for AR.

Designed for Real-Time Rasterization via GLSL

It can meet your real-time rasterizing needs, and this is probably the main place where we currently disagree directly. There is convergence between real-time and non-real-time engines for sure, and Khronos PBR Next purposely uses formulations and terminology that are compatible with path tracer materials. But don’t let that fool you, the Khronos PBR Next standard has been designed to be realized in glsl-based rasterizers and to be incredibly fast. Many of the engines that already implement this specification are glsl-based real-time engines, such as Three.js, Babylon.js, etc. Three.js has almost 90K stars on GitHub now, one of the most starred projects on all of Github now, and is incredibly widely used because it works everywhere. There is also a reference implementations of it in real-time here: https://github.khronos.org/glTF-Sample-Viewer-Release/ https://github.com/KhronosGroup/glTF-Sample-Viewer

Our Goals Align with Updating USD Preview Surface

Khronos PBR Next material aligns well with how the USD Preview Surface is being used. From my perspective, it would be a shame if we didn’t update USD Preview Surface to be what is considered to be real-time best practices now, keeping with its original goals but just updating it with the times. Right now USD Preview Surface is being used with baked/flat textures and not have material graphs associated with each parameter, or at least many implementations utilizing USD Preview Surface, that is what they do (looking at Apple USDZ here.) While MaterialX will eventually become widespread (I've been a long time supporter of it, and even pushed to get it into Three.js), that is many years away from being that standard. In the meantime, we will be passing around Uber shaders with single maps connected to each input along with modulators/factors. This is what we need for the next couple years until MaterialX wins and the low-end GPUs get faster. Khronos PBR Next is designed for this use case.

A Possible Golden Age of Real-Time Interoperability

If USD adopted Khronos PBR Next as its USD Preview Surface I see USD replacing the need for FBX data imports into Unreal Engine and Unity. At least for a while, we'll have a golden age of tool interoperability around PBR materials, animation data and meshes all facilitated by USD. Honestly, that would be such an amazing achievement and we are so close to it. (We technically have this with meshes and animations and skinning right now, why cannot we have this with flat PBR Uber materials?)

High End VFX Standards are Separate

I think you should feel free to figure out what material standards you want on the very high end VFX side of things along with how those should be previewed in Hydra. To me this is a separate question. High end VFX does not have the constraints that real-time engines have and going for a single solution across both can be suboptimal at both ends of the spectrum, or one side has to compromise. So I am not arguing at all against Autodesk Standard Surface for VFX usage. If that is what the majority of VFX studio users want, then it is probably the right thing. And on the real-time side of things, I think it is clear that people have generally been coalescing around the Khronos PBR Next material standard when it comes to interoperability given its wide adoption and industry stakeholders.

Standards Should Not Be Innovators

The Khronos PBR Next standard is generally a subset of the Autodesk Standard Surface as well as the Adobe Standard Material and the Dassault Enterprise PBR. These company-specific materials can be converted to the Khronos PBR Next material via baking and converting or discarding some of their material parameters. I would not ask Adobe, Autodesk and Dassault to stop innovating on their materials. Industry standards are best if they are lagging the innovators because the standards act as consolidators of best practices, rather than innovators of new techniques which may not turn out (and thus become prematurely obsolete, wasting everyone's time who has been trying to implement the "standard.")

Idea: A Way of Aligning Autodesk Standard Surface and Khronos PBR Next

I wonder, and I haven’t brought this up with Autodesk, if it is possible to align the Autodesk Standard Surface preview simplification with the Khronos PBR Next standard? I suspect it is already very close. This may be a way of hitting all goals without much compromise? This may be something that other complex non-real-time materials can do as well -- align their preview simplifications with Khronos PBR Next. This also aligns well with updating USD Preview Surface to the Khronos PBR Next material standard as it servers the same role as it has.

meshula commented 1 year ago

Hi Ben, I'm very curious to see interactive versions of those images you posted ~ are they from they conformance suite? I'm hoping you can save me digging around on the web to find them :)

bhouston commented 1 year ago

Hi @meshula:

They are available in a few different flavours but one way is to use the glTF SampleViewer on the web:

https://github.khronos.org/glTF-Sample-Viewer-Release/

The model names you want to try are "ToyCar" and "IridescentDishWithOlives." There are other interesting ones as well, such as "DragonAttenuation", "GlamVelvetSofa", "MosquitoInAmber", "StainedGlassLamp" and "TransmissionTest."

My only concern is that the default IBL in this sample viewer is not great, and it can make things look a little wonky.

Here are two other fairly capable glTF viewers that has better default lighting and are slightly more polished:

https://sandbox.babylonjs.com (based on Babylon.js) https://gltf-viewer.donmccurdy.com (based on Three.js)

The glTF files can all be found in this repository using the same names: https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/2.0

emackey commented 1 year ago

Some direct viewing links are also available:

https://sandbox.babylonjs.com/?assetUrl=https://raw.githubusercontent.com/KhronosGroup/glTF-Sample-Models/master/2.0/IridescentDishWithOlives/glTF/IridescentDishWithOlives.gltf

https://sandbox.babylonjs.com/?assetUrl=https://raw.githubusercontent.com/KhronosGroup/glTF-Sample-Models/master/2.0/ToyCar/glTF/ToyCar.gltf

meshula commented 1 year ago

Thanks for the links! It's helpful to see them in action.

pjcozzi commented 1 year ago

@bhouston @emackey thanks for sharing these, it is excellent to seeing them running so well in a web browser on my old MacBook Air. Do you know of any performance stress tests using Khronos PBR? For example, lots of models and lots of different materials. That could be helpful to show potential performance on different hardware.

emackey commented 1 year ago

Do you know of any performance stress tests using Khronos PBR?

Different implementations of Khronos PBR have differing performance needs, optimizations and trade-offs. The PBR model is intended to scale gracefully from high-end path tracing to lightweight mobile GPUs. Individual vendors may performance test their own and their competitors' apps, but Khronos does not offer any performance tests of its own. Our sample viewer, @pjcozzi may recall, was intentionally renamed "sample" from "reference" to highlight that it is not the one "true" reference implementation, merely one sample implementation. By gearing our PBR parameters to represent physical attributes of real-world materials, we encourage adopters to treat real life as reference, and aim for that goal.

bhouston commented 1 year ago

Exactly what @emackey says! I will add that the main fidelity tradeoff to make Three.js/Babylon.js fast is that non-opaque objects generally do not interact properly with other non-opaque objects -- you can not look through a non-opaque object and see other non-opaque objects. This is because the non-opaque objects are rendered generally as a secondary pass after opaque objects and are layered on top. This means that they are order-dependent and all of the issues that come with that. Solutions would be to implement either order-independent transparent (OIT) or depth peeling or both, but these come at higher run-time costs. And of course all lighting is local + shadow maps, rather than GI.

But as @emackey said, it scales upwards. For example here is a very interesting project here that uses the same Khronos PBR material but uses it to produce a path traced result: https://github.com/gkjohnson/three-gpu-pathtracer. Example render: https://gkjohnson.github.io/three-gpu-pathtracer/example/bundle/index.html

jstone-lucasfilm commented 1 year ago

@bhouston @spiffmon @meshula This is likely just a preview of ideas we'll discuss in person, but I wanted to provide a few initial thoughts on Ben's proposal:

There's clearly great value in supporting the glTF PBR model in USD/Hydra rendering, and this will be critical in order to support glTF assets as USD leaf nodes, as noted by Spiff earlier in the thread. I'm not sure that the right approach, though, is to add a second, hardcoded shading model to USD, especially one based on an active, continuously-evolving standard like glTF PBR.

Instead, I would recommend that we take advantage of the existing MaterialX graph for glTF PBR, without requiring that clients fully support MaterialX in order to leverage it. For clients whose MaterialX support is too early, they can pre-generate GLSL/OSL/MDL for their own rendering environment, and then provide this as a monolithic implementation of the shading model in their environment. This is similar to the approach taken by Foundry in Mari, where their Autodesk Standard Surface implementation was pre-generated from the MaterialX graph and integrated into the application, without needing to implement all of the BSDF details on their own.

To my mind, this offers the best of both worlds, where DCC environments with full MaterialX support (e.g. Houdini, Clarisse, Maya) can continue to use the native MaterialX graph for glTF PBR, and those who haven't yet added this support can pre-generate the appropriate shading code from the MaterialX graph, achieving very high-quality results without requiring a deep dive into the specification and implementations of the model. Both full-MaterialX and early-MaterialX applications can share the same USD assets based on the glTF PBR shading model, expecting consistent, accurate rendering in both contexts.

As the MaterialX graph for glTF PBR evolves (and its version number advances), applications without full MaterialX support can automatically regenerate their monolithic shaders and mark them with the new node version, while applications with MaterialX support can continue to access all versions of the shading model through the native graph. As more applications achieve full support for MaterialX, they can abandon their pre-generated shading code and simply leverage these native graphs.

spiffmon commented 1 year ago

Thanks, @bhouston - all very compelling arguments. I am more of a "shading object model" guy than a rendering/PBR guy, so I'll leave it to others to debate the tradeoffs and levels of computational complexity of various PBR models. But I do think there's a misunderstanding of the performance concern I raised, and it isn't so much about live procedural networks in "high end VFX" assets vs flat, baked textures - many studios carry baked textures through their entire pipeline.

USD's "Storm" rasterizing renderer does not claim to be realtime in the same sense as game engine renderers, which typically require preconditioning of assets to fit known performance characteristics for a given engine and scene. Rather, one of the things that attracted VFX studios to USD and Hydra/Storm is that you could throw a ton of production assets at it and get "pretty low latency" and "interactive animation and tumble". @pjcozzi is on the track of concern: we'll typically have hundreds to thousands of unique assets instanced (possibly with overrides that can make instances unique) tens to hundreds of thousands of times, and some workflows benefit tremendously from being able to interact (and visually debug) with the scene using the Storm renderer with low latency.

If the collective bag of geometry contained in such a scene winds up having thousands of (from a glsl perspective) unique materials, that can not only lead to high latency while we compile all those glsl programs, but also run up against the hardware's fill-rate and shading limitations, which is exacerbated the more computation a shader requires. I believe we can, to some degree, trade those two things off against each other, reducing the number of unique materials to compile by using more highly parameterized shaders, but that can impact per-sample shading cost. This is more the balance that I was referring to when suggesting we might want to distinguish UsdPreviewSurface from a more fully featured uber surface, but I'll happily defer to more expert opinions, there.

USD itself provides mechanisms for assets to provide lightweight "proxy" versions of themselves for Storm/interactive rendering, though that can be tricky for animated assets, and makes the pipeline more complicated. And with enough time and resources, there are lots of things we would like to do for on-the-fly LOD in Storm, but as it stands right now, the only widespread simplification we are comfortable using for our workflow needs is eliding subdivision of subdivision surfaces.

Which is another wall of text that can be summarized as: we need to consider Storm as a demanding implementation with particular performance needs, so the more we can demonstrate scalability of PBR Next along the axes described above, the more comfortable we would be in promoting its use as an alternative and/or enhancement to UsdPreviewSurface!

bhouston commented 1 year ago

@jstone-lucasfilm I strongly believe that MaterialX is the future of material graph interoperability, although I am not as convinced that it will take over all BSDF glsl code in the near term. But maybe I am wrong about the BDSF glsl code? Maybe we should aim to see if we can convert Three.js, Babylon.js, the glTF Sample Viewer or the Three-GPU-Pathtracer project I linked to above to a MaterialX-generated BSDF and also see if it is possible to get them to move over to it? I think we will either learn we should go down this route, or what are the blockers. Maybe we could get funding for such an experiment? The goals is for the download to be small and for it to be as performant as their existing code while significantly reducing the complexity of maintaining their custom shaders.

In many ways, I would want to explore this irrespective of this Khronos PBR material in USD.

jstone-lucasfilm commented 1 year ago

@bhouston Agreed that this would be a valuable proof of concept, and we'd be happy to help with this. From our own experience at Lucasfilm, it's much easier to generate GLSL code from a canonical MaterialX graph, and then to customize and simplify it for the needs of a specific renderer, than it is to implement the complete uber-shader from scratch (including all of the subtle details of energy conservation and compensation). We've used this approach in GLSL implementations of our own studio shading model (which is more complex than either Autodesk Standard Surface or glTF PBR), and it allows us to get closely-matching behavior across applications very quickly and accurately.

pjcozzi commented 1 year ago

Great to see the creativity and collaboration in this issue!

RE: https://github.com/PixarAnimationStudios/USD/issues/2119#issuecomment-1370295391

The [Khronos] PBR model is intended to scale gracefully from high-end path tracing to lightweight mobile GPUs.

@emackey @bhouston I agree this is a nice asset (no pun intended) of Khronos PBR.

In the spirit of what @spiffmon mentioned in https://github.com/PixarAnimationStudios/USD/issues/2119#issuecomment-1370362735:

so the more we can demonstrate scalability of PBR Next along the axes described above, the more comfortable we would be in promoting its use as an alternative and/or enhancement to UsdPreviewSurface!

It seems like any education material (whitepapers, best practices, etc.) and tunable demos (number of models & materials; complexity of shaders / shortcuts used) that help the community understand the performance characteristics of Khronos PBR seems like they would be beneficial for our collective understanding. If there's anything in the realm that already exists, it would be great to share in this thread. If not, perhaps there could be a discussion as to what may be valuable and meaningful.

bhouston commented 1 year ago

@spiffmon wrote:

we need to consider Storm as a demanding implementation with particular performance needs, so the more we can demonstrate scalability of PBR Next along the axes described above, the more comfortable we would be in promoting its use as an alternative and/or enhancement to UsdPreviewSurface!

This makes a lot of sense and I understand your concern.

At this point I think it is just rendering engine best practices matching with your specific domain needs. The Khronos PBR material is nothing special both in terms of computation complexity or texture bandwidth requirements for modern GPUs, especially if you use the local illumination and limited non-opaque interaction model that Three.js and Babylon.js use for rendering. Modern games are basically running an incredibly similar BRDF anyhow at 4K at 60fps with a bunch of additional things on top like AO, GI, Reflections, and an antialiasing method.

Do something similar to the following and it will meet your performance needs:

Khronos PBR is a super set of the existing USD Preview Surface there is a lot of ways you can incrementally adopt it across a pipeline's tool set and you can adaptively fall back as well.

If I had some time, which I currently do not, I would mock up a test scenario that has a single shader running all these permutations on 100,000 objects using my personal render engine threeify. Maybe someone else has an engine they could try this in?

jstone-lucasfilm commented 1 year ago

Following up on this thread, I wanted to include thoughts from a parallel discussion on this subject, which has been actively moving forward on the USD/MaterialX channel of the ASWF Slack.

Here's one alternative approach discussed in the parallel thread that seems relevant here:

... I do wonder whether we'd create a better path for the industry by leaning into building the USD package with MaterialX enabled, rather than engineering workarounds for applications that delay this choice. In communications with vendors whose USD builds still lack MaterialX, it sounds like nearly all of them plan to take this step over the next year, and I wonder if we might bring the industry forward more quickly by encouraging this directly and making this the default selection in future USD builds.

Historically there were a few important USD integrations that didn't yet support MaterialX, specifically Unreal Engine, NVIDIA Omniverse, and Apple USD. Over the past year, however, both Unreal Engine and NVIDIA Omniverse have released early MaterialX support, and it's reasonable to expect that Apple USD will follow suit in the future.

From this vantage point, since MaterialX already ships with a high-quality graph for the glTF PBR shading model, it seems worthwhile to consider this as the standard mechanism for accessing glTF PBR content in USD/Hydra, using an approach that naturally extends to present and future shading models such as Autodesk Standard Surface and Adobe Standard Material.

spitzak commented 1 year ago

For the preview shader I don't think the USD file should contain a MaterialX description of the shader. It should only contain an indication that the preview shader is being used, similar to how UsdPreviewShader is used currently. It is perfectly fine to document a MaterialX shader to describe how the preview shader is supposed to work, and certainly some (all?) hydra shaders might implement it by just running MaterialX on a built-in copy of this shader.

This may be what everybody is talking about but I just wanted to make sure. Some of the discussion sounded like the MaterialX description of the shader has to be in the usd file.

On Mon, Jan 9, 2023 at 1:13 PM Jonathan Stone @.***> wrote:

Following up on this thread, I wanted to include thoughts from a parallel discussion on this subject, which has been actively moving forward on the USD/MaterialX channel of the ASWF Slack https://academysoftwarefdn.slack.com/archives/C02HJH53RN3/p1672772456371949 .

Here's one alternative approach discussed in the parallel thread that seems relevant here:

... I do wonder whether we'd create a better path for the industry by leaning into building the USD package with MaterialX enabled, rather than engineering workarounds for applications that delay this choice. In communications with vendors whose USD builds still lack MaterialX, it sounds like nearly all of them plan to take this step over the next year, and I wonder if we might bring the industry forward more quickly by encouraging this directly and making this the default selection in future USD builds.

Historically there were a few important USD integrations that didn't yet support MaterialX, specifically Unreal Engine, NVIDIA Omniverse, and Apple USD. Over the past year, however, both Unreal Engine https://docs.unrealengine.com/5.1/en-US/unreal-engine-5.1-release-notes/ and NVIDIA Omniverse https://twitter.com/theCGchannel/status/1590337399195836416 have released early MaterialX support, and it's reasonable to expect that Apple USD will follow suit in the future.

From this vantage point, since MaterialX already ships with a high-quality graph for the glTF PBR shading model https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/libraries/bxdf/gltf_pbr.mtlx, it seems worthwhile to consider this as the standard mechanism for accessing glTF PBR content in USD/Hydra, using an approach that naturally extends to present and future shading models such as Autodesk Standard Surface https://github.com/AcademySoftwareFoundation/MaterialX/blob/main/libraries/bxdf/standard_surface.mtlx and Adobe Standard Material.

— Reply to this email directly, view it on GitHub https://github.com/PixarAnimationStudios/USD/issues/2119#issuecomment-1376324340, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJMCEK7OXNY63AIEPQ54E3WRR5QPANCNFSM6AAAAAASQ3QX7A . You are receiving this because you are subscribed to this thread.Message ID: @.***>

jstone-lucasfilm commented 1 year ago

That's completely accurate, @spitzak, and the MaterialX graph for glTF PBR only needs to be stated once in the USD distribution, and can then be leveraged freely by any USD asset. Additionally, applications are free to use their own optimized or modified implementation instead of the built-in MaterialX definition, and this more flexible approach is currently used by the MaterialX implementation in Unreal Engine, among others.

spiffmon commented 1 year ago

I believe @spitzak is stating there should not need to be any "...MaterialX graph for glTF PBR only needs to be stated once in the USD distribution" that requires the MaterialX runtime to consume or even parse in USD. Is that accurate, @spitzak ? @jstone-lucasfilm I believe you are making the case that it should be represented as an actual MaterialX graph in a) the .usd file(s) b) the USD/Hydra runtime that generates a shader implementation (that, with additional work, we could pregenerate and consume without MaterialX codegen, in which case USD would only use MaterialX runtime to translate the MaterialX graph to USD. Is that accurate?

spitzak commented 1 year ago

I meant that the .usd(abc) file should not contain declarations of the individual MaterialX nodes that make up the shader. It is fine if the implementation is done by making a lot of MaterialX nodes. It sounds like what I requested is in fact what is happening.

On Mon, Jan 9, 2023 at 9:23 PM F. Sebastian (spiff) Grassia < @.***> wrote:

I believe @spitzak https://github.com/spitzak is stating there should not need to be any "...MaterialX graph for glTF PBR only needs to be stated once in the USD distribution" that requires the MaterialX runtime to consume or even parse in USD. Is that accurate, @spitzak https://github.com/spitzak ? @jstone-lucasfilm https://github.com/jstone-lucasfilm I believe you are making the case that it should be represented as an actual MaterialX graph in a) the .usd file(s) b) the USD/Hydra runtime that generates a shader implementation (that, with additional work, we could pregenerate and consume without MaterialX codegen, in which case USD would only use MaterialX runtime to translate the MaterialX graph to USD. Is that accurate?

— Reply to this email directly, view it on GitHub https://github.com/PixarAnimationStudios/USD/issues/2119#issuecomment-1376752557, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJMCEK3HBENF7R7ZS2RARLWRTW6BANCNFSM6AAAAAASQ3QX7A . You are receiving this because you were mentioned.Message ID: @.***>

jstone-lucasfilm commented 1 year ago

There are some new ideas on this topic in the USD/MaterialX channel of the ASWF Slack, and they seem worthwhile to reference here.

Here's a top-line summary of the new ideas, for those without access to the ASWF Slack:

One topic that has been brought up in a number of recent meetings is enabling MaterialX by default in USD, which would provide access to shading models such as Autodesk Standard Surface and glTF PBR in off-the-shelf builds of UsdView and HdStorm, not to mention providing access to procedural material nodes. After discussing this further with a colleague, I wanted to propose one potential path for achieving this, with the focus on keeping the USD build process as simple and efficient as possible.

From a high level, the idea would be to directly include MaterialX modules inside the USD libraries that reference them, so that default builds never require additional modules to be downloaded.

The UsdMtlx module, for example, would directly include the folders for MaterialXCore and MaterialXFormat, upon which it depends:

Clients would still need an option to reference an external build of MaterialX when required, but this would become an option that's needed only more rarely, and by default UsdMtlx would have all of the code that it needs to build with MaterialX support.

Similarly, the HdMtlx module would directly include the folders for MaterialXGenShader and MaterialXGenGlsl, upon which it depends:

Importantly, none of these MaterialX modules have any external dependencies at all, so they could be directly included without any effect on USD dependencies.

There are additional details to be considered here, but I wanted to put out this initial idea for discussion, to see if it's a direction that might be aligned with the goals of teams working with USD and MaterialX.

bhouston commented 1 year ago

In order to turn this from a proposal into something concerete, I've started work on the PR here: https://github.com/PixarAnimationStudios/USD/pull/2421. I've started with the specific parameters I am adding and their formal definitions. I just need to figure out how to properly weave this through the USD library now.

bhouston commented 1 year ago

I was told by @jstone-lucasfilm that this whole idea is a non-starter and I should stop work on it. He also told me that multiple VFX companies are going to push back on it if I pursue it. I am a busy person and was just seeing a gap that I should could be addressed. I was not intending to cause issues nor do I have time to fight against half the VFX industry. Closing this and moving on.

spiffmon commented 1 year ago

Thanks so much for starting this engagement, @bhouston ! I agree with Jonathan that in the long term (and possibly short, also), we'll be better served by more directly mirroring PBRNext in USD (likely through MaterialX, though possibly also as a standalone shader like UsdPreviewSurface if the community feels we need accelerated conformance, as you suggest) rather than trying to translate and tether UsdPreviewSurface to it. But your arguments here lit a lightbulb for me, and started positive movement in the right directions for glTF/USD interop.