[Feature request]: Extract all vertices, normals and materials as javascript const, and use them to create meshes and react-three-fiber as gltjfx code

[JustFly1984]

Currently I'm developing 3d game with multiple gtlf files loaded locally with Gatsby.js and Typescript. I'm using Blender to export models, one per file. I need to reload the page a lot to see my changes, so often, there is some network glitch, which prevents one of gltf file from loading, and crashes the app. I can setupErrorBoundary, but that is leaving a user with network issues with fallback, but I want to escape to load the model in first place.

I've tried to load models with import { useGLTF } from '@react-three/drei/useGLTF', but this method does not allow to change materials, and there is a lot of cloning happening to reuse same model in multiple positions in same react-three-fiber scene.

I've tried gltfjsx, and it does allow to provide materials to the <mesh/> as I need, but <mesh/> getting prop geometry as instance of BufferGeometry type, and I can't simply extract vertices and faces to const variables, to use in new BufferGeometry() or `.

I do not want to use materials from the model at all.

I understand that JS bundle size will grow, but if I we get rid of <Suspense/>, I guess we could pre-render stuff offline.

import * as React from 'react'
import { useGLTF } from '@react-three/drei/useGLTF'

import { GLTF } from 'three/examples/jsm/loaders/GLTFLoader'

type GLTFResult = GLTF & {
  nodes: {
    model: THREE.Mesh
  }
  materials: {
    ['Material.001']: THREE.MeshStandardMaterial
  }
}

function Model(props: JSX.IntrinsicElements['group']) {
  const group = React.useRef<THREE.Group>()

  const { nodes } = useGLTF('/pieces/model.gltf') as GLTFResult

  const model = nodes.model

  console.info(model.geometry)

  return (
    <group ref={group} {...props} dispose={null}>
      <mesh geometry={nodes.model.geometry} rotation={[0, 0, -Math.PI]} >
        <meshBasicMaterial color="peachpuff" opacity={0.5} transparent />
      </mesh>
    </group>
  )
}

export default React.memo(Model)

useGLTF.preload('/pieces/model.gltf')

Would be great if model could be pre-parsed with gltfjsx

Thank you for great work, I think my attempt with 3d in React would stuck again today, if I did not found your library, It did miracle, and now I can dynamically change materials of my 3d models!

[JustFly1984]

I've printed to the console the model.geometry, and copied arrays of position, normal, and index, wrapped arrays in according Floating32Array and Uint16Array

The code looks like this:

import * as React from 'react'
import { Vector3, Object3D } from 'three'

import { vertices, normals, indices } from './model-vertices'

function toggleBoolean(bool: boolean): boolean {
  return !bool
}

import type { Vertex, VertexId } from './types'

interface Props {
  id: VertexId
  cube: Vertex
  position: Vector3
  color: string
}

const scale: [x: number, y: number, z: number] = [0.25, 0.25, 0.25]
const rotation: [x: number, y: number, z: number] = [0, 0, -Math.PI]

function Model({ position, cube, color, id  }: Props) {
  const ref = React.useRef<Object3D>()

  const [hovered, setHover] = React.useState(false)

  const onClick = React.useCallback(function callback(e) {
    e.stopPropagation()

    setHover(toggleBoolean)

    console.info('cube', cube)
  }, [])

  const onPointerOver = React.useCallback(function callback() {
    setHover(true)
  }, [])

  const onPointerOut = React.useCallback(function callback() {
    setHover(false)
  }, [])

  return (
    <mesh
      castShadow
      receiveShadow
      position={position}
      rotation={rotation}
      scale={scale}
      ref={ref}
      onClick={onClick}
      onPointerOver={onPointerOver}
      onPointerOut={onPointerOut}
      userData={{ id, cube }}
    >
      <bufferGeometry>
        <bufferAttribute
          attachObject={['attributes', 'position']}
          count={vertices.length / 3}
          array={vertices}
          itemSize={3}
        />
        <bufferAttribute
          attachObject={['attributes', 'normal']}
          count={normals.length / 3}
          array={normals}
          itemSize={3}
        />
        <bufferAttribute
          attachObject={['attributes', 'index']}
          count={indices.length}
          array={indices}
          itemSize={1}
        />
      </bufferGeometry>
      <meshBasicMaterial
        color={hovered ? 'hotpink' : color}
        // opacity={0.5}
        // transparent
      />
    </mesh>
  )
}

export default React.memo(Model)

but resulting render is broken:

Can somebody please explain why rendering is broken, and how to fix it? I can still correctly load gltf file and render the model in recommended way, but I still want to escape to load and parse gltf for each page load.

[drcmda]

you dont need to fetch, all threejs loaders have a parse function. just import your glb as an arraybuffer (hence it will be part of your bunlde) and do new GLTFLoader().parse(model) you still can get the virtual graph if you use the useGraph hook. as to why why your vertices look like that, i think you'll have better luck on threes discourse forum.

[JustFly1984]

@drcmda I have managed to fix issue with rendering by changing passing indices from <bufferAttribute/> to <bufferGeometry/>

This is the code that eventually work out:

import * as React from 'react'
import { Vector3, Object3D, BufferAttribute } from 'three'

import { vertices, normals, indices } from './model-vertices'

function toggleBoolean(bool: boolean): boolean {
  return !bool
}

import type { Vertex, VertexId } from './types'

interface Props {
  id: VertexId
  cube: Vertex
  position: Vector3
  color: string
}

const scale: [x: number, y: number, z: number] = [0.25, 0.25, 0.25]
const rotation: [x: number, y: number, z: number] = [0, 0, -Math.PI]
const index: BufferAttribute = new BufferAttribute(indices, 1)

function Model({ position, cube, color, id  }: Props) {
  const ref = React.useRef<Object3D>()

  const [hovered, setHover] = React.useState(false)

  const onClick = React.useCallback(function callback(e) {
    e.stopPropagation()

    setHover(toggleBoolean)

    console.info('pawn', cube)
  }, [])

  const onPointerOver = React.useCallback(function callback() {
    setHover(true)
  }, [])

  const onPointerOut = React.useCallback(function callback() {
    setHover(false)
  }, [])

  return (
    <mesh
      castShadow
      receiveShadow
      position={position}
      rotation={rotation}
      scale={scale}
      ref={ref}
      onClick={onClick}
      onPointerOver={onPointerOver}
      onPointerOut={onPointerOut}
      userData={{ id, cube }}
    >
      <bufferGeometry index={index}>
        <bufferAttribute
          attachObject={['attributes', 'position']}
          count={vertices.length / 3}
          array={vertices}
          itemSize={3}
          normalized={false}
        />
        <bufferAttribute
          attachObject={['attributes', 'normal']}
          count={normals.length / 3}
          array={normals}
          itemSize={3}
          normalized={false}
        />
      </bufferGeometry>
      <meshBasicMaterial
        color={hovered ? 'hotpink' : color}
      />
    </mesh>
  )
}

export default React.memo(Model)

[drcmda]

i think it could be easier

import { useAsset } from "use-asset"
// eslint-disable-next-line import/no-webpack-loader-syntax
import model from '!arraybuffer-loader!./assets/model.gltf'

function Foo({ buffer }) {
  const { scene } = useAsset(
    (buffer) => new Promise(res => new GLTFLoader().parse(buffer, "", res)),
    [buffer]
  )
  const { nodes, materials }= useGraph(scene)
  return (
    <mesh geometry={nodes.name.geometry}>
      <meshStandardMaterial />
    </mesh>
  )
}

<Foo buffer={model} />

this will suspend the loading process, but the model-data is part of your bundle.

[JustFly1984]

@drcmda in my case I do not need to use <React.Suspend /> at all!!! I can reuse models with SSR. If it will be too much for the bundle size, I can use React.lazy() and dynamic import() to optimize the bundle size.

The goal is not to load the scene, but reduce network request failures.

Extracting vertices, normals and indexes from the modal, reduces the total bundle size in general.

Now I have a single ts file per model with exported typed arrays from model geometry, without over-complicating the build process(which runs on each build/hmr)

Notice - Development experience improved, cos there is no network calls for models, hence almost instant page reloads.

[drcmda]

alright happy it worked out for you!

[JustFly1984]

@drcmda Thank you, but can you reopen this issue as feature request?

Is it possible to add some flag, so it will create more granular gtlfjsx component, which does not use <React.Suspend/> nor use @react-three/drei/useGLTF ?

Would be awesome to have this option, as I see no other tool which could provide this functionality.

[drcmda]

i am thinking about about the arraybuffer thing: https://codesandbox.io/s/r3f-basic-demo-forked-1sjyd?file=/src/App.js

but vertices and normals would to too much imo and very ineffective since the data overhead will be too large. with arraybuffer you still just ship a tiny draco compressed thing, it's loaded without xhr request and is part of the bundle. the other thing seems too specific to me and would still be easy to implement in your project.

[JustFly1984]

@drcmda Wow nice lego block like scene!

In my case I really do not want to load whole scene, I need only geometry and materials.

In my case I see that bufferAttributes has uv index position and normal typed arrays. May be I'm missing something.

With experience, I know that less dependencies === smaller bundle size === better performance.

This feature will allow more granular dependency import, and smaller bundle size/performance.

Please reconsider about reopening an issue.

[drcmda]

the thing is if you do it like that these arrays will take lots of space. whereas the gltf is draco compressed. draco is a major improvement, you can compress 100mb files into a few kilobyte with that. but if you take raw vertex data, it will cost a lot, this is what usually makes mesh files so big. as for the scene data in the gltf, it's almost nothing, a few bytes. but again, it may just be your specific usecase and perhaps you do get real benefits from it - the arraybuffer thing is something i was planing to find a decent abstraction for a long time.

[JustFly1984]

I'd argue that gltf file is parsed to js anyway, so this typed arrays are created in memory in both cases, in my case it is created immediately, without spending computation resources on loading, parsing and other work with gltf file.

[drcmda]

Of course, but it's shipped compressed that's what I meant. Given you have a regular gltf that's 15mb, your draco compressed bundle will be, say, 200kb, but your vertices js bundle will probably be larger than the uncompressed file even since at least the 15mb are binary data, whereas the vertices are a comma separated list with long floats, you'll likely ship 20-30mb over the wire that way.