Add support for USD and USDZ formats

[HenryB96]

It would be great to get Three.js support for the Universal Scene Description (USD) file format and the upcoming USDZ format announced at WWDC today.

USDZ specification document: https://graphics.pixar.com/usd/files/USDZFileFormatSpecification.pdf (I assume that more details will come out in the coming months for this format)

[Mugen87]

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD/USDZ does better than glTF?

[Usnul]

USD documentation: http://graphics.pixar.com/usd/docs/index.html

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD/USDZ does better than glTF?

I agree, part of it is politics me thinks. However, that's a separate discussion :)

[donmccurdy]

USD has been around for a couple years — it's commonly used in film, and not an obvious choice for runtime transmission. As far as I can tell USDZ is a wrapper around that. Apple hasn't given much detail at this point; maybe they have some plan to make it more runtime-appropriate, or maybe the messaging at WWDC was just unclear with respect to how they're actually using it. In any case I think we probably need to wait for more information.

If someone is interested in contributing a THREE.USDZLoader, then by all means. 🙂 But all of the documentation I've found so far suggests the format is very closely tied to the official (C++/Python) libraries for parsing it, similar to FBX and the FBX SDK. So unless those can be made web-friendly with WASM, implementing USDZ support in three.js and keeping up with changes in the format over time looks like a very time-consuming process, without the benefit of those libraries.

[richardmonette]

Out of curiosity, I tried opening the "Retro TV" file from https://developer.apple.com/arkit/gallery/ as if it was a simple zip. I 'unarchived' the .usdz by renaming it's extension to .zip, and then having my OS do its usual open procedure. Presumably, something similar could be achieved with https://gildas-lormeau.github.io/zip.js/

This is what I see when I 'unarchive' the usdz:

So, the textures are just PNGs, which are probably referenced from within that .usdc, which is the next issue. According to https://graphics.pixar.com/usd/docs/Converting-Between-Layer-Formats.html, ".usdc files – binary format" which is in Crate (see Crate File Format in https://graphics.pixar.com/usd/docs/USD-Glossary.html) format.

There is also .usda, which is ASCII encoded, which would presumably be easier to write a loader around.

As a rough outline of some to-dos for this, minimally a Loader would be looking at extracting a Zip file in JS, and then also understanding the .usda (and potentially binary Crate .usdc as well.)

[TimvanScherpenzeel]

Hi @richardmonette, I've done some research on USDZ and the USDA format and have written a proof-of-concept glTF to USDZ converter. Perhaps it is useful for your research: https://github.com/timvanscherpenzeel/gltf-to-usdz. You can see a demo here: https://twitter.com/domenicopanacea/status/1011292393801420800 or try it out yourself here: https://timvanscherpenzeel.github.io/gltf-to-usdz/ (requires iOS 12).

In short: geometry gets converted to OBJ, the metallic-roughness texture is split by channel and textures are Y-flipped. From this a USDA file is constructed dynamically and fed into usdz_converter to construct the final usdz.

I know some people are looking into writing a direct converter to USDZ but not a whole lot is known about the USDC format at this point in time (also in terms of decoding).

[richardmonette]

Hey @TimvanScherpenzeel, thanks for your message and work on this!

Have you considered using the USD Python API (https://graphics.pixar.com/usd/docs/Hello-World---Creating-Your-First-USD-Stage.html)? I wondered if it might be possible to skip the .obj stage and, using Python, read the glTF and go straight into .usda/c that way?

[TimvanScherpenzeel]

I've had a lot of issues setting up USD and have not yet taken the time to fix the issue (I think it has something to do with having multiple versions of Python installed and during compilation it links to the wrong Python).

You could indeed write the OBJ file directly into the USDA file (that could actually be a good next step in my proof of concept). I know the file structure of the AR Gallery examples because a contributor to my project send me the USDA files after having the USDC files decoded it with usdcat.

[PixelPartner]

The whole point of Apple providing a USDZ QuickLook is to totally bypass using WebGL, DOM and Javascript and directly use the scene with ARkit2.0. It's just a milestone to an rOS based on Apple/Pixar tech. When they later add interactivity/scriptability, even less Web-UI will be needed.

Pixar USD, using highly efficient OpenSubDiv Patches for geometry, ported to mobile Metal will soon outperform HighEnd PC-VR WebVR. Not as a file format, but as a whole Experience-Making/Presenting/(-Interaction) pipeline.

So IMHO it's more important to export USDZ from AR/VR Editors and for Android and Windows to catch the USD train, than to add an importer to the WebGL2 pipeline.

An important issue is that WebXR would have to also support SubDiv patches instead of legacy polymeshes to fully support USD as used by Apple.

[FreakTheMighty]

@pixelpartner you sorta lost me.

It seems to me that unless a loader is built natively into browsers, it would be helpful to build one in JavaScript..

I’m sure quick look is highly optimized, but what is the path forward for creating interactive content using USDZ assets in a browser?

[looeee]

It's a valid feature request but I personally don't understand Apple/Pixar. Why are they introducing a new file format and not just using glTF? Can somebody explain what USD /USDZ does better than glTF?

I had a chat with someone working on product display software recently, they export to glTF and USDZ so I asked him what he thought about this. His take in a nutshell: it's so that Apple can continue to claim to be the high-end luxury choice, with an exclusive 3D format that only works on their platform via ARKit2.0/SceneKit. Adopting the standard glTF format would run contrary to their goals here, they want to be able to claim that their format is better and allows for higher quality results due to PBR material properties that glTF doesn't (yet?) support.

16977, and perhaps #14051, are likely needed to get three.js up to the same rendering standards as Apple's offerings, but I'm not sure how the current state or future plans for glTF relate to this.

In any case, if for no other reasons than to subvert Apple's nefarious goals here, I think we'll need to support USDZ at some point.

[donmccurdy]

it's so that Apple can continue to claim to be the high-end luxury choice, with an exclusive 3D format that only works on their platform...

I'm willing to give more benefit of the doubt here – they have different goals for a "runtime" format than we (threejs + web devs) do, and are able to make different choices as a result. In particular, bundling the ~15MB USD runtime into iOS means they don't care that loading USDZ without a native runtime is prohibitively difficult, and they seem to be OK with not providing the web application any access to the model, but just opening the model as-is in a native app instead.

On "high-end luxury", Apple's iOS QuickLook viewer supports metallic/roughness PBR materials, and the material model is largely equivalent to what glTF 2.0 supports today. The additional material features planned for glTF 2.0 in the future, and described in https://github.com/mrdoob/three.js/issues/16977, would go beyond what Apple's USDZ viewers support now.

I mention "Apple's USDZ viewers" because that is, de facto, what people seem to mean by USDZ. But the USDZ format, technically, is just a zipped USD file, and could contain pretty much anything from a common material model to a completely custom shader. Apple's viewers support a small subset of USD, and that subset isn't well defined, but the best summary you're likely to find is here: https://github.com/google/usd_from_gltf#compatibility.

In terms of rendering fidelity, I'd point out that model-viewer displays glTF models with threejs and provides the means to open a USDZ version in iOS Quick Look, and the visual results are quite similar. This page provides some direct comparisons, although unfortunately it is down right now.

In any case, if for no other reasons than to subvert Apple's nefarious goals here, I think we'll need to support USDZ at some point.

See https://github.com/mrdoob/three.js/issues/14219#issuecomment-395107896.

[garyo]

I just want to mention here that there are reasons for supporting USD beyond Apple "claiming to be a luxury choice". @Mugen87 asks "Can somebody explain what USD/USDZ does better than glTF?" A few things that I care about:

• HDR/float textures
• Environment maps (UsdLuxDomeLight)
• Area and sphere lights (UsdLuxRectLight etc.)
• Shader materials (OSL/MDL) with "preview surface" for WebGL

My use case is loading the same scene description into WebGL and into a high-end rendering pipeline - understanding the looks will differ of course and some substitution will take place.

[themightyatom]

@garyo lighting and environment are handled by the AR viewer, in an attempt to simulate the actual environment. This video gives you an idea of what maps they support (albedo, metallic, roughness, normal, ambient occlusion, emissive)
https://developer.apple.com/videos/play/wwdc2018/603/

What I'd like to see is an USDA exporter from three.js, so you can save models you have created/ configurered in a three project, for viewing in AR. "model-view" includes a tag for specifying a USDZ model alternative, for when viewed on iOS. https://modelviewer.dev/

It really is a shame Apple went for a format no-one else uses, and isn't natively supported anywhere (beta exporter is available for Blender). So you have no choice but manually convert all your existing models. The fact that you have to include your textures is also a pain if you use the same texture over an entire range of models. Surely they could have external references? It's really a bit of an hodgepodge format, that only utilises geometry and mapping from the Pixar specs, and zips them together with your image files.
glTF/glb would have been much better and we would only need ONE model for everywhere... In the realm of e-commerce, which is what they are aiming at, no customer will want a web experience that ONLY works for iOS. As usual we'll have to do all the monkey work and maintenance.

[makc]

@garyo

My use case is loading the same scene description into WebGL and into a high-end rendering pipeline

wait, is there a way to load usd(z) in webgl?

[PixelPartner]

The feature set of USDZ recently drastically increased with iOS 13.4 (and RealityComposer 1.4 (133+))

Apple added a lot of USD schemes that currently no other software can interpret and might simply ignore. Almost the whole interactive functionality of RealityComposer is now not only saved to .reality scene files, but can also be exported in this new USD flavour. Some new USD features of the top of my head: Trigger bounding, tap or enter area grouped Animation actions, in sequence and/or parallel Text Geometrie is created on the fly by a new type of Prim that takes the string, font and size

This opens a new era. Any developer can now use the Pixar use tools and libs (C++/python 2.6/python 3.x) on any platform to write USD files that can either simply show content or create interactive experiences. Obviously you still need an iDevice for testing.

- Best wishes, (just ending quarantine, staying in home office) Excited Thomas

Am 14.04.2020 um 20:25 schrieb themightyatom notifications@github.com: @garyo https://github.com/garyo lighting and environment are handled by the AR viewer, in an attempt to simulate the actual environment. This video gives you an idea of what maps they support (albedo, metallic, roughness, normal, ambient occlusion, emissive) https://developer.apple.com/videos/play/wwdc2018/603/ https://developer.apple.com/videos/play/wwdc2018/603/ What I'd like to see is an USDA exporter from three.js, so you can save models you have created/ configurered in a three project, for viewing in AR. includes a tag for specifying a USDZ model alternative, for when viewed on iOS. https://modelviewer.dev/ https://modelviewer.dev/ It really is a shame Apple went for a format no-one else uses, and isn't natively supported anywhere (beta exporter is available for Blender). So you have no choice but manually convert all your existing models. The fact that you have to include your textures is also a pain if you use the same texture over an entire range of models. Surely they could have external references? It's really a bit of an hodgepodge format, that only utilises geometry and mapping from the Pixar specs, and zips them together with your image files. glTF/glb would have been much better and we would only need ONE model for everywhere...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/14219#issuecomment-613604756, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAR6X5SXAY32JAPYL5V44TRMSTBTANCNFSM4FDHSXWQ.

[donmccurdy]

I'm not aware of any web-compatible parser for USD or USDZ. Rendering in WebGL or another API isn't a problem, it's parsing the file's content that is complex — that task is best handled by the official USD runtime, which has dependencies including OpenSubdiv and MaterialX.

Any developer can now use the Pixar use tools and libs (C++/python 2.6/python 3.x) on any platform to write USD files..

Being able to read USD files on any platform would be nice. 🙂 For technical reasons I do not think a functional three.js implementation is likely right now. If someone is blessed with the spare time, my suggestion would be to work on implementing OpenSubdiv (w/ WASM?) or MaterialX (w/ THREE.NodeMaterial?), as those would be useful in their own right, even without a USD parser.

[garyo]

@garyo

My use case is loading the same scene description into WebGL and into a high-end rendering pipeline

wait, is there a way to load usd(z) in webgl?

No, that's what I'm looking for. I have scenes (object, lights, materials, camera, animation, etc. - the whole thing) and I want to be able to load the same scene, in whatever format, into WebGL (via three.js) for preview, and something like Blender for final renders. glTF is my best option for now, but it doesn't support the things I mention above (area lights etc.) I understand the glTF folks are reluctant to add more features because they're targeting last-mile to devices rather than true scene transfer. Just looking for options.

[themightyatom]

@garyo Sounds like Collada might fit the bill? I use an edited version of the collada exporter from three.js to take assets into Blender for renders.

[garyo]

@themightyatom I looked into Collada a couple of years ago -- may have to revisit it. Thanks. (At the time I was convinced by arguments such as the Godot one, no PBR support, etc -- that may have changed.)

[themightyatom]

@garyo Collada was designed for transferring assets around between game developers all working on different platforms. It's not a viable delivery format, but it does what it's meant for very well :) I was lucky enough to spend a week in Italy with one of it's creators, Rene Arnaud. It's written "from the GPU's perspective" and is generally the first format to support changes in spec. (OpenGL, DirectX, etc.).

[ronpadz]

I think what would be super valuable is to integrate https://github.com/google/usd_from_gltf as an exporter. Much less important to me to be able to open USD.

[donmccurdy]

To do anything useful with USDZ (load or export) you are going to need the USD SDK. USD from glTF requires the USD SDK as a dependency, but the SDK isn't supported on the web. Apple gets around this by leaving the web browser and taking you into their native AR Quick Look application, but that's hardly a solution for the web in general.

[c4b4d4]

Has anyone figured out something yet? Or it's literally impossible at the moment? I'm looking forward to use for the following pipeline:

1. Create a scene with Three.JS
2. Export it to GLTF for non-iOS and export it to USDZ to enable it for iOS devices (this is where I see no possibility)
3. Load it to <model-viewer>

It works great on desktop and non-iOS devices. And I cannot rely on servers converting the files, because it's all done client-side. User generates the scene on the web that needs to be displayed in AR.

[mrdoob]

It works great on desktop and non-iOS devices.

What desktop?

[c4b4d4]

It works great on desktop and non-iOS devices.

What desktop?

MacOS+Chrome (Obvs AR not being used) Talking about <model-viewer>, which forces USDZ for iOS devices.

[mrdoob]

@kzka90: #21096

[ajmas]

One other project of reference could be https://github.com/syoyo/tinyusdz , which states:

TinyUSDZ is dependency-free(depends only on C++ STL. Yes, you don't need USD library!) USDZ loader library written in C++11.

I came across the pxr-usdz format by means of https://size.link/ . I had hoped it was generating everything front-end, but it turns out it is doing this server side. On Android it is using gltf for use with ARCore, while on iOS it is usdz for ARKit.

[mrdoob]

https://github.com/PixarAnimationStudios/USD/issues/1492 👀

[mrdoob]

Also...

According to https://graphics.pixar.com/usd/docs/Converting-Between-Layer-Formats.html, ".usdc files – binary format" which is in Crate (see Crate File Format in https://graphics.pixar.com/usd/docs/USD-Glossary.html) format.

Here's the source code for the crate file format: https://github.com/PixarAnimationStudios/USD/blob/release/pxr/usd/usd/crateFile.cpp

[donmccurdy]

Pretty large binary at the moment (35 MB minzipped) but will be interested to see how that evolves!

[mrdoob]

@donmccurdy I think the use case for the time being would be to be able to load usd files (ie. in https://threejs.org/editor/) and then export in whatever format is appropriate.

[netgfx]

With this https://developer.apple.com/videos/play/wwdc2021/10076/# I believe USD models will be in more demand than ever and they will spill over web as well

[mrdoob]

The good news is that they seem to be creating usda files, which are "easy" to parse:

https://twitter.com/MikkoH/status/1402114796175204352

[jankais3r]

If anyone wants to have a closer look, here is a sample USDZ file created with the ObjectCapture API. ObjectCapture_sample.zip

[disjukr]

here is another sample file (2.6mb zipped, created with object capture api): saeukkang.usdz.zip

[jeffscottward]

Any updates on a USD(a/z) loader?

[rfcj]

in the website cadforum.cz there is a online usd viewer:

https://www.cadforum.cz/cz/USDview/?file=cz/USDview/knife.usdz&cameraY=150&cameraZ=100

does anyone know what was used?

[donmccurdy]

Autodesk released an open-source, experimental WASM build for USD, with a three.js render delegate. See:

• https://twitter.com/KaiSchroeder/status/1491167208529367043
• https://twitter.com/hybridherbst/status/1500266804471603205

[jeffscottward]

I asked the <model-viewer> people about integration and it didn’t get a lot of love.

Also it crashes on mobile right now

On Thu, Mar 24, 2022 at 11:46 PM Don McCurdy @.***> wrote:

Autodesk released an open-source WASM build for USD, with a three.js render delegate. See:

• https://twitter.com/KaiSchroeder/status/1491167208529367043
• https://twitter.com/hybridherbst/status/1500266804471603205

— Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/14219#issuecomment-1078624879, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJUO7T54PJ4TQ6CHEOY6K3VBUZKFANCNFSM4FDHSXWQ . You are receiving this because you commented.Message ID: <mrdoob/three. @.***>

-- Thanks, Jeff Ward Front-end Developer Tel: 516-551-8624 @. @.>* @.**** https://twitter.com/#!/jeffscottward

[ponahoum]

I did a quick integration (with Vue/Typescript/ThreeJs) for tests purposes with the ObjectCapture API generated USDZ: USDZ Web Viewer, it's based on the Autodesk implementation which works pretty well and works without backend. I was so tired of those USDZ viewers that require you to upload your USDZ file to then convert it to GLTF on a server and then bring it back to ThreeJs. @jankais3r @jeffscottward I tested it with your models and it seems to work well.

[jankais3r]

Nice, works well. Any plans to share the code on GitHub?

[ponahoum]

@jankais3r I'm working on a library for everyone to easily use on NPM and hopefully will release it very soon. In the meantime, Autodesk published an implementation as an example on their GitHub: Autodesk USD GitHub. The code looks pretty strange sometimes, but it does the job!

[ponahoum]

As promised I did a quick typescript wrapper around the Autodesk implementation for an easier use with NPM : three-usdz-loader Tell me if that works for you guys!

[lin72h]

OMG, it's works on my Mac(OS: 12.4, Safari: 15.5) Thanks

[PixelPartner]

@ponahoum Who did the magic trick to make it run with Safari? That's very helpful!

[jeffscottward]

Is there a reason why it doesn’t work on mobile safari? Is there no wasm on mobile?

On Thu, Jun 23, 2022 at 11:51 AM Thomas Kumlehn @.***> wrote:

Who did the magic trick to make it run with Safari? That's very helpful!

— Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/14219#issuecomment-1164585247, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJUO7X6ODF7UGJSSAAMN2LVQSBWLANCNFSM4FDHSXWQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Thanks, Jeff Ward Front-end Developer Tel: 516-551-8624 @. @.>* @.**** https://twitter.com/#!/jeffscottward

[ajmas]

@jeffscottward what version of iOS are you using? Asking in case this depends on something only available in the most recent versions of Safari?

[darkvertex]

WASM should work in iOS Mobile Safari since iOS 11 (says caniuse.com.)

[ponahoum]

@PixelPartner it runs on Mac Safari because latest Safari versions now support SharedArrayBuffer / Atomics objects, which this implementation uses.

@jeffscottward @darkvertex Doesn't seem to work on my iphone with iOS 15.5 either and it's pretty hard to debug as i don't have a mac. I'll take a look when i have time. WASM is of course supported on iOS alongside with WebGL2 in latests versions so this is not the cause of the issue.

[ajmas]

On iOS there are some experimental settings that might need enabling, but I also would need to investigate. I don’t currently have access to my computer or a USD file. If anyone has one, I could experiment while I am travelling.