Add support for USD and USDZ formats

[HenryB96]

It would be great to get Three.js support for the Universal Scene Description (USD) file format and the upcoming USDZ format announced at WWDC today.

USDZ specification document: https://graphics.pixar.com/usd/files/USDZFileFormatSpecification.pdf (I assume that more details will come out in the coming months for this format)

[netgfx]

@ajmas I have tried this model: Match_ChairWhite-_100102.zip Direct file: https://qsxfdqhsuyovskknxkaj.supabase.co/storage/v1/object/public/threed/models/Match_Chair_White_-_100102.usdz?t=2022-06-24T08%3A07%3A32.406Z

And on web, it seems to be working fine. Also, https://sketchfab.com/ offers auto-generated USDZ files with each model (free and paid ones), so that is a good source of models too.

[MAG-AdrianMeredith]

iOs is throwing a RangeError: Out of memory. We've seen this error before, its a bug in safari where the new thread support requires a little bit extra memory on startup than it thinks it does. It should load fine if you increase the initial memory to the wasm module

[MAG-AdrianMeredith]

[MAG-AdrianMeredith]

We fixed it by tweaking the compile flags in emscripten like this (your values may vary!) "-sINITIAL_MEMORY=33554432", "-sMAXIMUM_MEMORY=1073741824",

[ponahoum]

@MAG-AdrianMeredith I can confirm that on iOS the error is RangeError: Out of memory. For some reason the WASM module was trying to assign more memory that it could. I did put a memory limit for iOS devices on three-usdz-loader latest version and it now works like a charm on my iPhone 12 Pro with iOS 15.5.

@netgfx i tried with your model and it works pretty well now !

[MAG-AdrianMeredith]

I believe there's an actual bug ticket in safari / WebKit somewhere. Not on my computer right now but can dig it out on Monday if needed.

On Sat, 25 Jun 2022, 05:22 Jeff Scott Ward, @.***> wrote:

Great work @ponahaum! So that's the problem, how do we describe a solution in a PR to AutoCAD?

I'm unfamiliar with emacripten workflows. I imagine there is no way to dynamically detect what that range should be to stop the error, hmmm.

— Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/14219#issuecomment-1166189275, or unsubscribe https://github.com/notifications/unsubscribe-auth/AX35EVNKIVWIKGNVI2QLLWLVQ2CP3ANCNFSM4FDHSXWQ . You are receiving this because you were mentioned.Message ID: @.***>

[expenses]

here is another sample file (2.6mb zipped, created with object capture api): saeukkang.usdz.zip

I looked into this file and the format is very strange. The points and texture coord arrays have different lengths and have individual index buffers.

Edit: same with this one:

If anyone wants to have a closer look, here is a sample USDZ file created with the ObjectCapture API. ObjectCapture_sample.zip

[ponahoum]

@disjukr @expenses both open well it seems with three-usdz-loader

[mrdoob]

Here's a very naive first version of the USDZLoader: https://github.com/mrdoob/three.js/pull/24568

I've only tried it with the models shared in this thread. It only supports usdz files that use usda internally.

The next steps are to gather as many sample files as we can (using usd/usda) and add as many features as possible.

[jeffscottward]

I just console.log'd against your sample file and it loaded great 👏 .

I generated my own .usdz file from the "Kaedim" AI service that turns 2D photos into 3D, which I THEN took into Reality Composer to create a facial target to lock onto and added some textures and normals.

No dice as expected with this loader. Just got silent fail of an empty object.

After getting USDTools working on Apple Silicon (https://developer.apple.com/forums/thread/702189?answerId=711711022#711711022) - I unzipped the .usdz file to get the texture files and a .usdc, then used usdcat to convert to .usda.

The .usda was successfully output but did not help in rendering a model when usdzconverting to .usdz from that. Same empty object.

I don't know if this at all helps but thought I would run it through. Here is the .usdz file in question. https://cloudflare-ipfs.com/ipfs/QmcgmRXJZmeY51mvLikhEm3yo8oxrz15WJhBmQUyU8yTmi/ssj2-face.usdz (Github wouldn't allow .usdz file type - had to go with slow IPFS option, give it a minute)

PS: The model is supposed to be like wearing a yellow Super Saiyan wig. iOS QuickLook should automatically lock it onto your face.

[PixelPartner]

This animated flag pole uses SkelAnim to move the flag. If you need an example for that https://usdzshare.com/?ug-gallery=photo-detail&photo_id=6557

[PixelPartner]

This seems to be the new way Apple accepts Object placement since the latest update of RealityConverter. I'd like to point out how fine tuned pivot placement is now possible and how this differs from vanilla USD. IMHO this was introduced to handle all kind of CAD export needs.

float3 xformOp:rotateXYZ:geometricRotation = (0, 0, 0)
float3 xformop:rotateXYZ:postRotation = (0,0,0)
float3 xform0p:rotateXYZ:preRotation = (0,0,0)
float3 xformOp:rotateXYZ:rotateXYZ = (0,0,0)
float3 xformOp:scale:geometricScaling = (1,1,1)
float3 xformOp:scale:scale =(1,1,1)
float3 xformOp:translate:geometricTranslation=(0,0,0)
float3 xformOp:translate:rotationOffset=(0,0,0)
float3 xformOp:translate:rotationPivot = (0,0,0)
float3 xform0p:translate:scalingOffset = (0,0,0)
float3 xformOp:translate:scalingpivot=(0,0,0)
float3 xformOp:translate:translate=(0,0,0)
uniform token[] xformOpOrder=[ "xformOp:translate:translate", "xformOp:translate:rotationoffset", "xformOp:translate:rotationpivot", "xformOp:rotateXYZ:preRotation", "xformOp:rotateXYZ:rotateXYZ", "xformOp:rotateXYZ:postRotation", "!invert!xformOp:translate:rotationPivot", "xformOp:translate:scalingOffset", '"xformOp:translate:scalingpivot", "xformOp:scale:scale", "!invert!xformOp:translate:scalingPivot", "xformOp: translate:geometricTranslation",
"xformOp:rotateXYZ:geometricRotation",
"xformOp:scale:geometricScaling" ]

[mrdoob]

@jeffscottward

I don't know if this at all helps but thought I would run it through. Here is the .usdz file in question. https://cloudflare-ipfs.com/ipfs/QmcgmRXJZmeY51mvLikhEm3yo8oxrz15WJhBmQUyU8yTmi/ssj2-face.usdz (Github wouldn't allow .usdz file type - had to go with slow IPFS option, give it a minute)

That link doesn't work for me... I think you should be able to zip the usdz file and attach the zip to your post here.

[mrdoob]

Attaching the file here in case those links die: ssj2-face.usdz.zip

[mrdoob]

That file uses a usdc internally. We can't parse these yet.

[jeffscottward]

Yea I figured, it was just a test to see if converting to USDA would help at all for parsing it.

On Thu, Sep 1, 2022 at 7:14 PM mrdoob @.***> wrote:

That file uses a usdc internally. We can't parse these yet.

[image: Screen Shot 2022-09-01 at 4 14 12 PM] https://user-images.githubusercontent.com/97088/188027885-48393709-8136-4b21-a564-5ebd42b7b0f0.png

— Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/14219#issuecomment-1234886768, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJUO7TEVR2SSDEP6AQ26H3V4E2GLANCNFSM4FDHSXWQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Thanks, Jeff Ward Front-end Developer Tel: 516-551-8624 @. @.>* @.**** https://twitter.com/#!/jeffscottward

[mrdoob]

But it doesn't seem like it was actually converted to usda? 🤔

[jeffscottward]

I did that on my machine. This is just the original starting point USDz. Going back to USDz but from USDA had no effect.

On Thu, Sep 1, 2022 at 9:17 PM mrdoob @.***> wrote:

But it doesn't seem like it was actually converted to usda? 🤔

— Reply to this email directly, view it on GitHub https://github.com/mrdoob/three.js/issues/14219#issuecomment-1234960008, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJUO7RRZ573YPNFT73A6BLV4FITFANCNFSM4FDHSXWQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Thanks, Jeff Ward Front-end Developer Tel: 516-551-8624 @. @.>* @.**** https://twitter.com/#!/jeffscottward

[SC36-11-1]

it's so that Apple can continue to claim to be the high-end luxury choice, with an exclusive 3D format that only works on their platform...

I'm willing to give more benefit of the doubt here – they have different goals for a "runtime" format than we (threejs + web devs) do, and are able to make different choices as a result. In particular, bundling the ~15MB USD runtime into iOS means they don't care that loading USDZ without a native runtime is prohibitively difficult, and they seem to be OK with not providing the web application any access to the model, but just opening the model as-is in a native app instead.

On "high-end luxury", Apple's iOS QuickLook viewer supports metallic/roughness PBR materials, and the material model is largely equivalent to what glTF 2.0 supports today. The additional material features planned for glTF 2.0 in the future, and described in #16977, would go beyond what Apple's USDZ viewers support now.

I mention "Apple's USDZ viewers" because that is, de facto, what people seem to mean by USDZ. But the USDZ format, technically, is just a zipped USD file, and could contain pretty much anything from a common material model to a completely custom shader. Apple's viewers support a small subset of USD, and that subset isn't well defined, but the best summary you're likely to find is here: https://github.com/google/usd_from_gltf#compatibility.

In terms of rendering fidelity, I'd point out that model-viewer displays glTF models with threejs and provides the means to open a USDZ version in iOS Quick Look, and the visual results are quite similar. This page provides some direct comparisons, although unfortunately it is down right now.

In any case, if for no other reasons than to subvert Apple's nefarious goals here, I think we'll need to support USDZ at some point.

See #14219 (comment).

[jeffscottward]

Worth mentioning a new joint effort to further the ecosystem with this new alliance.

https://aousd.org/

Pixar Adobe Apple Autodesk Nvidia Epic Unity Cesium and others

[ShaddyDC]

I recently came across tinyusdz which claims to support compilation to WASM. I should note that I've used neither three.js nor the library, but maybe this is useful for people looking into this.

[Mugen87]

Since three.js offers USDZLoader and USDZExporter for quite a while now, this issue can be closed.

Please file module specific feature requests or bug reports in new issues.

[yosun]

Since three.js offers USDZLoader and USDZExporter for quite a while now, this issue can be closed.

Please file module specific feature requests or bug reports in new issues.

yep modelviewer.dev has always had usdz support ("free quicklook" no iOS/VIsionOS) via three.js even though they are advocating for glb/gltf