USDZ Loader - Feature mismatch with USDZExporter

[GitHubDragonFly]

Description

This is intended as an FYI bug report related to currently available features.

The file exported as USDZ, by using the link provided by @Mugen87 in https://github.com/mrdoob/three.js/pull/26607 does not work in the current editor (an error is thrown). USDZ Loader is currently missing the code / features required to open this file.

If it does help, and for comparison, this exported model can be tested as working here with the customized non-module version of the USDZ Loader.

Customized module version can be seen here and also further below in the Code section - probably something that @hybridherbst might take a look at.

All the custom code is what I assume to be correct and MeshStandardMaterial was replaced with MeshPhysicalMaterial.

Reproduction steps

1. Use https://rawcdn.githack.com/Mugen87/three.js/dev75/examples/misc_exporter_usdz.html to export the model to USDZ
2. Try to open this exported file in the editor https://threejs.org/editor/

Code

import {
    BufferAttribute,
    BufferGeometry,
    ClampToEdgeWrapping,
    FileLoader,
    Group,
    Loader,
    Mesh,
    MeshPhysicalMaterial,
    MirroredRepeatWrapping,
    RepeatWrapping,
    SRGBColorSpace,
    TextureLoader,
    Object3D,
} from 'three';

import * as fflate from '../libs/fflate.module.js';

class USDAParser {

    parse( text ) {

        const data = {};

        const lines = text.split( '\n' );
        const length = lines.length;

        let current = 0;
        let string = null;
        let target = data;

        const stack = [ data ];

        // debugger;

        function parseNextLine() {

            const line = lines[ current ];

            // console.log( line );

            if ( line.includes( '=' ) ) {

                const assignment = line.split( '=' );

                const lhs = assignment[ 0 ].trim();
                const rhs = assignment[ 1 ].trim();

                if ( rhs.endsWith( '{' ) ) {

                    const group = {};
                    stack.push( group );

                    target[ lhs ] = group;
                    target = group;

                } else {

                    target[ lhs ] = rhs;

                }

            } else if ( line.endsWith( '{' ) ) {

                const group = target[ string ] || {};
                stack.push( group );

                target[ string ] = group;
                target = group;

            } else if ( line.endsWith( '}' ) ) {

                stack.pop();

                if ( stack.length === 0 ) return;

                target = stack[ stack.length - 1 ];

            } else if ( line.endsWith( '(' ) ) {

                const meta = {};
                stack.push( meta );

                string = line.split( '(' )[ 0 ].trim() || string;

                target[ string ] = meta;
                target = meta;

            } else if ( line.endsWith( ')' ) ) {

                stack.pop();

                target = stack[ stack.length - 1 ];

            } else {

                string = line.trim();

            }

            current ++;

            if ( current < length ) {

                parseNextLine();

            }

        }

        parseNextLine();

        return data;

    }

}

class USDZLoader extends Loader {

    constructor( manager ) {

        super( manager );

    }

    load( url, onLoad, onProgress, onError ) {

        const scope = this;

        const loader = new FileLoader( scope.manager );
        loader.setPath( scope.path );
        loader.setResponseType( 'arraybuffer' );
        loader.setRequestHeader( scope.requestHeader );
        loader.setWithCredentials( scope.withCredentials );
        loader.load( url, function ( text ) {

            try {

                onLoad( scope.parse( text ) );

            } catch ( e ) {

                if ( onError ) {

                    onError( e );

                } else {

                    console.error( e );

                }

                scope.manager.itemError( url );

            }

        }, onProgress, onError );

    }

    parse( buffer ) {

        const parser = new USDAParser();

        function parseAssets( zip ) {

            const data = {};
            const loader = new FileLoader();
            loader.setResponseType( 'arraybuffer' );

            for ( const filename in zip ) {

                if ( filename.endsWith( 'png' ) ) {

                    const blob = new Blob( [ zip[ filename ] ], { type: { type: 'image/png' } } );
                    data[ filename ] = URL.createObjectURL( blob );

                }

                if ( filename.endsWith( 'usd' ) || filename.endsWith( 'usda' ) ) {

                    const text = fflate.strFromU8( zip[ filename ] );
                    data[ filename ] = parser.parse( text );

                }

            }

            return data;

        }

        function findUSD( zip ) {

            for ( const filename in zip ) {

                if ( filename.endsWith( 'usda' ) ) {

                    return zip[ filename ];

                }

            }

        }

        const zip = fflate.unzipSync( new Uint8Array( buffer ) );

        // console.log( zip );

        const assets = parseAssets( zip );

        // console.log( assets )

        const file = findUSD( zip );

        if ( file === undefined ) {

            console.warn( 'THREE.USDZLoader: No usda file found.' );

            return new Group();

        }

        // Parse file

        const text = fflate.strFromU8( file );
        const root = parser.parse( text );

        // Build scene

        function findMeshGeometry( data ) {

            if ( ! data ) return undefined;

            if ( 'prepend references' in data ) {

                const reference = data[ 'prepend references' ];
                const parts = reference.split( '@' );
                const path = parts[ 1 ].replace( /^.\//, '' );
                const id = parts[ 2 ].replace( /^<\//, '' ).replace( />$/, '' );

                return findGeometry( assets[ path ], id );

            }

            return findGeometry( data );

        }

        function findGeometry( data, id ) {

            if ( ! data ) return undefined;

            if ( id !== undefined ) {

                const def = `def Mesh "${id}"`;

                if ( def in data ) {

                    return data[ def ];

                }

            }

            for ( const name in data ) {

                const object = data[ name ];

                if ( name.startsWith( 'def Mesh' ) ) {

                    // Move points to Mesh

                    if ( 'point3f[] points' in data ) {

                        object[ 'point3f[] points' ] = data[ 'point3f[] points' ];

                    }

                    // Move st to Mesh

                    if ( 'float2[] primvars:st' in data ) {

                        object[ 'float2[] primvars:st' ] = data[ 'float2[] primvars:st' ];

                    }

                    if ( 'texCoord2f[] primvars:st' in data ) {

                        object[ 'texCoord2f[] primvars:st' ] = data[ 'texCoord2f[] primvars:st' ];

                    }

                    // Move st indices to Mesh

                    if ( 'int[] primvars:st:indices' in data ) {

                        object[ 'int[] primvars:st:indices' ] = data[ 'int[] primvars:st:indices' ];

                    }

                    return object;

                }

                if ( typeof object === 'object' ) {

                    const geometry = findGeometry( object );

                    if ( geometry ) return geometry;

                }

            }

        }

        function buildGeometry( data ) {

            if ( ! data ) return undefined;

            let geometry = new BufferGeometry();

            if ( 'int[] faceVertexIndices' in data ) {

                const indices = JSON.parse( data[ 'int[] faceVertexIndices' ] );
                geometry.setIndex( new BufferAttribute( new Uint16Array( indices ), 1 ) );

            }

            if ( 'point3f[] points' in data ) {

                const positions = JSON.parse( data[ 'point3f[] points' ].replace( /[()]*/g, '' ) );
                const attribute = new BufferAttribute( new Float32Array( positions ), 3 );
                geometry.setAttribute( 'position', attribute );

            }

            if ( 'normal3f[] normals' in data ) {

                const normals = JSON.parse( data[ 'normal3f[] normals' ].replace( /[()]*/g, '' ) );
                const attribute = new BufferAttribute( new Float32Array( normals ), 3 );
                geometry.setAttribute( 'normal', attribute );

            } else {

                geometry.computeVertexNormals();

            }

            if ( 'float2[] primvars:st' in data ) {

                data[ 'texCoord2f[] primvars:st' ] = data[ 'float2[] primvars:st' ];

            }

            if ( 'texCoord2f[] primvars:st' in data ) {

                const uvs = JSON.parse( data[ 'texCoord2f[] primvars:st' ].replace( /[()]*/g, '' ) );
                const attribute = new BufferAttribute( new Float32Array( uvs ), 2 );

                if ( 'int[] primvars:st:indices' in data ) {

                    geometry = geometry.toNonIndexed();

                    const indices = JSON.parse( data[ 'int[] primvars:st:indices' ] );
                    geometry.setAttribute( 'uv', toFlatBufferAttribute( attribute, indices ) );

                } else {

                    geometry.setAttribute( 'uv', attribute );

                }

            }

            return geometry;

        }

        function toFlatBufferAttribute( attribute, indices ) {

            const array = attribute.array;
            const itemSize = attribute.itemSize;

            const array2 = new array.constructor( indices.length * itemSize );

            let index = 0, index2 = 0;

            for ( let i = 0, l = indices.length; i < l; i ++ ) {

                index = indices[ i ] * itemSize;

                for ( let j = 0; j < itemSize; j ++ ) {

                    array2[ index2 ++ ] = array[ index ++ ];

                }

            }

            return new BufferAttribute( array2, itemSize );

        }

        function findMeshMaterial( data ) {

            if ( ! data ) return undefined;

            if ( 'rel material:binding' in data ) {

                const reference = data[ 'rel material:binding' ];
                const id = reference.replace( /^<\//, '' ).replace( />$/, '' );
                const parts = id.split( '/' );

                return findMaterial( root, ` "${ parts[ 1 ] }"` );

            }

            return findMaterial( data );

        }

        function findMaterial( data, id = '' ) {

            for ( const name in data ) {

                const object = data[ name ];

                if ( name.startsWith( 'def Material' + id ) ) {

                    return object;

                }

                if ( typeof object === 'object' ) {

                    const material = findMaterial( object, id );

                    if ( material ) return material;

                }

            }

        }

        function buildMaterial( data ) {

            const material = new MeshPhysicalMaterial();

            if ( data !== undefined ) {

                if ( 'def Shader "PreviewSurface"' in data ) {

                    const surface = data[ 'def Shader "PreviewSurface"' ];

                    if ( 'color3f inputs:diffuseColor.connect' in surface ) {

                        const path = surface[ 'color3f inputs:diffuseColor.connect' ];
                        const sampler = findTexture( root, /(\w+).output/.exec( path )[ 1 ] );

                        material.map = buildTexture( sampler );
                        material.map.colorSpace = SRGBColorSpace;

                    } else if ( 'color3f inputs:diffuseColor' in surface ) {

                        const color = surface[ 'color3f inputs:diffuseColor' ].replace( /[()]*/g, '' );
                        material.color.fromArray( JSON.parse( '[' + color + ']' ) );

                    }

                    if ( 'color3f inputs:emissiveColor.connect' in surface ) {

                        const path = surface[ 'color3f inputs:emissiveColor.connect' ];
                        const sampler = findTexture( root, /(\w+).output/.exec( path )[ 1 ] );

                        material.emissiveMap = buildTexture( sampler );
                        material.emissiveMap.colorSpace = SRGBColorSpace;
                        material.emissive.fromArray( JSON.parse( '[ 1, 1, 1 ]' ) );

                    } else if ( 'color3f inputs:emissiveColor' in surface ) {

                        const color = surface[ 'color3f inputs:emissiveColor' ].replace( /[()]*/g, '' );
                        material.emissive.fromArray( JSON.parse( '[' + color + ']' ) );

                    }

                    if ( 'normal3f inputs:normal.connect' in surface ) {

                        const path = surface[ 'normal3f inputs:normal.connect' ];
                        const sampler = findTexture( root, /(\w+).output/.exec( path )[ 1 ] );

                        material.normalMap = buildTexture( sampler );
                        material.normalMap.colorSpace = SRGBColorSpace;

                    }

                    if ( 'float inputs:roughness' in surface ) {

                        material.roughness = parseFloat( surface[ 'float inputs:roughness' ] );

                    } else if ( 'float inputs:roughness.connect' in surface ) {

                        const path = surface[ 'float inputs:roughness.connect' ];
                        const sampler = findTexture( root, /(\w+).output/.exec( path )[ 1 ] );

                        material.roughnessMap = buildTexture( sampler );
                        material.roughnessMap.colorSpace = SRGBColorSpace;

                    }

                    if ( 'float inputs:occlusion.connect' in surface ) {

                        const path = surface[ 'float inputs:occlusion.connect' ];
                        const sampler = findTexture( root, /(\w+).output/.exec( path )[ 1 ] );

                        material.aoMap = buildTexture( sampler );
                        material.aoMap.colorSpace = SRGBColorSpace;

                    }

                    if ( 'float inputs:metallic' in surface ) {

                        material.metalness = parseFloat( surface[ 'float inputs:metallic' ] );

                    } else if ( 'float inputs:metallic.connect' in surface ) {

                        const path = surface[ 'float inputs:metallic.connect' ];
                        const sampler = findTexture( root, /(\w+).output/.exec( path )[ 1 ] );

                        material.metalnessMap = buildTexture( sampler );
                        material.metalnessMap.colorSpace = SRGBColorSpace;

                    }

                    if ( 'float inputs:opacity' in surface ) {

                        material.opacity = parseFloat( surface[ 'float inputs:opacity' ] );

                    } else if ( 'float inputs:opacity.connect' in surface ) {

                        const path = surface[ 'float inputs:opacity.connect' ];
                        const sampler = findTexture( root, /(\w+).output/.exec( path )[ 1 ] );

                        material.alphaMap = buildTexture( sampler );
                        material.alphaMap.colorSpace = SRGBColorSpace;

                    }

                    if ( 'float inputs:clearcoat' in surface ) {

                        material.clearcoat = parseFloat( surface[ 'float inputs:clearcoat' ] );

                    }

                    if ( 'float inputs:clearcoatRoughness' in surface ) {

                        material.clearcoatRoughness = parseFloat( surface[ 'float inputs:clearcoatRoughness' ] );

                    }

                    if ( 'float inputs:ior' in surface ) {

                        material.ior = parseFloat( surface[ 'float inputs:ior' ] );

                    }

                }

                if ( 'def Shader "diffuseColor_texture"' in data ) {

                    const sampler = data[ 'def Shader "diffuseColor_texture"' ];

                    material.map = buildTexture( sampler );
                    material.map.colorSpace = SRGBColorSpace;

                }

                if ( 'def Shader "normal_texture"' in data ) {

                    const sampler = data[ 'def Shader "normal_texture"' ];

                    material.normalMap = buildTexture( sampler );
                    material.normalMap.colorSpace = SRGBColorSpace;

                }

            }

            return material;

        }

        function findTexture( data, id ) {

            for ( const name in data ) {

                const object = data[ name ];

                if ( name.startsWith( `def Shader "${ id }"` ) ) {

                    return object;

                }

                if ( typeof object === 'object' ) {

                    const texture = findTexture( object, id );

                    if ( texture ) return texture;

                }

            }

        }

        function buildTexture( data ) {

            if ( 'asset inputs:file' in data ) {

                const path = data[ 'asset inputs:file' ].replace( /@*/g, '' );

                const loader = new TextureLoader();

                const texture = loader.load( assets[ path ] );

                const map = {
                    '"clamp"': ClampToEdgeWrapping,
                    '"mirror"': MirroredRepeatWrapping,
                    '"repeat"': RepeatWrapping
                };

                if ( 'token inputs:wrapS' in data ) {

                    texture.wrapS = map[ data[ 'token inputs:wrapS' ] ];

                }

                if ( 'token inputs:wrapT' in data ) {

                    texture.wrapT = map[ data[ 'token inputs:wrapT' ] ];

                }

                return texture;

            }

            return null;

        }

        function buildObject( data ) {

            const geometry = buildGeometry( findMeshGeometry( data ) );
            const material = buildMaterial( findMeshMaterial( data ) );

            const mesh = geometry ? new Mesh( geometry, material ) : new Object3D();

            if ( 'matrix4d xformOp:transform' in data ) {

                const array = JSON.parse( '[' + data[ 'matrix4d xformOp:transform' ].replace( /[()]*/g, '' ) + ']' );

                mesh.matrix.fromArray( array );
                mesh.matrix.decompose( mesh.position, mesh.quaternion, mesh.scale );

            }

            return mesh;

        }

        function buildHierarchy( data, group ) {

            for ( const name in data ) {

                if ( name.startsWith( 'def Scope' ) ) {

                    buildHierarchy( data[ name ], group );

                } else if ( name.startsWith( 'def Xform' ) ) {

                    const mesh = buildObject( data[ name ] );

                    if ( /def Xform "(\w+)"/.test( name ) ) {

                        mesh.name = /def Xform "(\w+)"/.exec( name )[ 1 ];

                    }

                    group.add( mesh );

                    buildHierarchy( data[ name ], mesh );

                }

            }

        }

        const group = new Group();

        buildHierarchy( root, group );

        return group;

    }

}

export { USDZLoader };

Live example

• jsfiddle-latest-release
• jsfiddle-dev

Screenshots

No response

Version

r155

Device

Desktop

Browser

Firefox

OS

Windows

[Mugen87]

Thanks for reporting! The project develops loaders and exporters independently so there is no feature parity policy in place.

The file exported as USDZ, by using the link provided by @Mugen87 in https://github.com/mrdoob/three.js/pull/26607 does not work in the current editor (an error is thrown).

The actual runtime error is:

Uncaught (in promise) TypeError: Cannot use 'in' operator to search for 'def "Geometry"' in undefined at findGeometry (USDZLoader.js:259:14) at findMeshGeometry (USDZLoader.js:245:12) at buildObject (USDZLoader.js:577:36) at buildHierarchy (USDZLoader.js:605:19) at buildHierarchy (USDZLoader.js:615:6) at buildHierarchy (USDZLoader.js:601:6) at buildHierarchy (USDZLoader.js:615:6) at USDZLoader.parse (USDZLoader.js:625:3)

I think it's best to use this issue to fix this particular error.

[GitHubDragonFly]

I will leave this issue with you but just to make a suggestion:

In order to resolve that particular error it appears that the following lines need to be changed in order to see the model without properly applied texture:

// Line 187 from
                if ( filename.endsWith( 'usd' ) ) {
// to
                if ( filename.endsWith( 'usda' ) ) {
// or (what I am using)
                if ( filename.endsWith( 'usd' ) || filename.endsWith( 'usda' ) ) {

// Line 257 from
                const def = `def "${id}"`;
// to
                const def = `def Mesh "${id}"`;

[GitHubDragonFly]

This post is just an FYI for those who might be interested in it.

I have further modified both USDZ Loader an Exporter on my end, by separating the material entries in each and adding a few new entries. All the code can be seen in my repositories and tested as well.

My GLTF Viewer is the only one currently employing this new USDZ Exporter but it can also be found in the Localized_3js repository which has customized three.js editor.

Here is a picture of the Damaged Helmet as exported and loaded with the newly modified files:

[Mugen87]

In order to resolve that particular error it appears that the following lines need to be changed in order to see the model without properly applied texture:

Since you already figured out the necessary changes, would you like filing a PR?

[hybridherbst]

Would also welcome to see your changes as a PR or at least as a link to a branch from you! I can reproduce the issue and can look at fixing it but would be great to see the other changes you did in a structured form.

[GitHubDragonFly]

@hybridherbst feel free to use the code from PR that was closed in order to fix the editor error caused by the USDZ Loader.

It should be sufficient to make the changes as they are on lines 181, 198, 257 and 291-296.

Changing material from MeshStandardMaterial to MeshPhysicalMaterial can probably wait for any later changes.

[Mugen87]

Let's use this issue to just fix the round-trip. Meaning importing files exported from USDZExporter should not result in a runtime error.

I've checked the source code of importer and exporter and it seems the issuecan be fixed with a single line in USDZLoader. The problem is that the loader looks for a single .usda model file. All other asset definitions (like geometry data) have to use the .usd extension. Since the exporter exports geometry data with the .usda extension, there is a mismatch now. The loader can't find the geometry data.

Since QuickLook can display the exported USDZ, the issue has to be in the loader. However, using if ( filename.endsWith( 'usd' ) || filename.endsWith( 'usda' ) ) { as suggested above results in an unexpected behavior since the model data are now saved two times (in the assets and file object). Yes, the model can be imported but the code seems confusing and not right.

I've tried to find more information about the USDZ spec but only found this: https://openusd.org/release/spec_usdz.html

TBH, I have expected a glTF similar document that actually explains how the files are organized. Do we have to solve this issue in a trial and error fashion or can we refer to some sort of documentation?

[GitHubDragonFly]

@Mugen87 that is a good explanation since I was wondering about those usd and usda checks on both 181 and 198 lines.

I might not be much of a help with resolving this issue properly.

At least my trial and error attempts currently export and show USDZ models, which I think is a good start.

[hybridherbst]

@Mugen87 I am planning to fix the issue, just didn't get to it yet. USD doesn't have a formal specification yet (it's in the works) but the "usd" file extension is always valid, with usda/usdc being hints at the file being text (a) vs binary/crate (c). See https://openusd.org/release/usdfaq.html#what-file-format-is-my-usd-file.

USDZs are just zip files with a 64-byte alignment that contain pretty much arbitrary data (e.g. an USDZ can contain folders and subfolders, any number of additional USDZ files, any number of usda/usdc/etc files, ...). An USDZ is "presentable" (something can be displayed from it) if the first file in the zip (no matter the path, just the first file) is USD/USDA/USDC/USDZ; that file will then be used as entry point.

As I'm sure @mrdoob can confirm the scope of the USDLoader right now is NOT loading arbitrary USD files. It's a very small supported subset, and an explicit goal is indeed that the files exported by three can be loaded :)

In the general case USD files can have any number of interdependencies, nesting, composition arcs with overrides and overs/unders, variants that are applied while loading... which is a bigger topic.

[Mugen87]

Thanks for the additional explanation! It's good to hear that a formal specification is on the way.

Since you are planning to fix the issue, I'm not filing a PR by myself but reviewing yours 😇 .