Geometry instancing?

[nvaytet]

I don't know if this is the right place to ask such questions, but I couldn't really find a Q/A space or forum for pythreejs.

Could anyone point me to how one can achieve geometry instancing with pythreejs?

I would basically want to reproduce the example described here using pythreejs.

Or to put it more simply, I would like to have 10,000 boxes, each with a different color and orientation, with maximum performance by minimizing the number of draw calls. I don't even need any lighting, boxes can just be made of BasicMaterial.

Any help would be greatly appreciated!

[vidartf]

Which step(s) are you finding hard to "translate"? If you share how far you've gotten, and where you are stuck, it will be a lot quicker to help hopefully.

[nvaytet]

Thanks for the quick reply. I'm basically already stuck at the first hurdle where he says to

create an instance of InstancedBufferGeometry and copy the geometry contents in it:

var geometry = new THREE.BoxBufferGeometry( 20, 20, 20 );
var instancedGeometry = new THREE.InstancedBufferGeometry() //this is going to wrap both geometry and a bit of the scene graph
//we have to copy the meat - geometry into this wrapper
Object.keys(geometry.attributes).forEach(attributeName=>{
  instancedGeometry.attributes[attributeName] = geometry.attributes[attributeName]
})
//along with the index
instancedGeometry.index = geometry.index

I've tried

import pythreejs as p3
N = 50
geometry = p3.BoxBufferGeometry( 20, 20, 20 )
instancedGeometry = p3.InstancedBufferGeometry()
for key, attr in geometry.attributes.items():
    instancedGeometry.attributes[key] = attr

but apparently

AttributeError: 'BoxBufferGeometry' object has no attribute 'attributes'

The I found instancedGeometry has something called from_geometry() and thought maybe i could use that, but when I do

instancedGeometry = p3.InstancedBufferGeometry().from_geometry(geometry)
print(instancedGeometry.attributes)

i get an empty dict.

[vidartf]

For InstancedBufferGeometry, it is better if you supply the attributes in one go, and pass during init. E.g. in the ThickLines example notebook

posInstBuffer = InstancedInterleavedBuffer( np.array([
    [0, 0, 0, 1, 1, 1],
    [2, 2, 2, 4, 4, 4]
], dtype=np.float32))
colInstBuffer = InstancedInterleavedBuffer( np.array([
    [1, 0, 0, 1, 0, 0],
    [0, 1, 0, 0, 0, 1]
], dtype=np.float32))

# This uses InstancedBufferGeometry, so that the geometry is reused for each line segment
lineGeo = InstancedBufferGeometry(attributes={
    # Helper line geometry (2x4 grid), that is instanced
    'position': BufferAttribute(np.array([
        [ 1,  2, 0], [1,  2, 0],
        [-1,  1, 0], [1,  1, 0],
        [-1,  0, 0], [1,  0, 0],
        [-1, -1, 0], [1, -1, 0]
    ], dtype=np.float32)),
    'uv': BufferAttribute(np.array([
        [-1,  2], [1,  2],
        [-1,  1], [1,  1],
        [-1, -1], [1, -1],
        [-1, -2], [1, -2]
    ], dtype=np.float32)),
    'index': BufferAttribute(np.array([
        0, 2, 1,
        2, 3, 1,
        2, 4, 3,
        4, 5, 3,
        4, 6, 5,
        6, 7, 5
    ], dtype=np.uint8)),
    # The line segments are split into start/end for each instance:
    'instanceStart': InterleavedBufferAttribute(posInstBuffer, 3, 0),
    'instanceEnd': InterleavedBufferAttribute(posInstBuffer, 3, 3),
    'instanceColorStart': InterleavedBufferAttribute(colInstBuffer, 3, 0),
    'instanceColorEnd': InterleavedBufferAttribute(colInstBuffer, 3, 3),
})

[nvaytet]

Thanks for the reply. I had seen the ThickLines example, and tried to work from it, but never managed to use it for me needs.

Nevertheless, I think I made some progress. I managed to copy the attributes by using the from_geometry. The trick is to copy the attributes from a BufferGeometry:

import pythreejs as p3
import numpy as np

geometry = p3.BoxBufferGeometry(
    width=5,
    height=10,
    depth=15)

morph = p3.BufferGeometry.from_geometry(geometry)
print(morph.attributes)

instancedGeometry = p3.InstancedBufferGeometry()
for key, attr in morph.attributes.items():
    instancedGeometry.attributes[key] = attr
print(instancedGeometry.attributes)

but I still cannot manage to see anything in the renderer. I tried to then do:

N = 5

colors = p3.InstancedBufferAttribute(array=np.random.random([N, 4]))
offsets = p3.InstancedBufferAttribute(array=np.random.random([N, 3]) * 10.0)
instancedGeometry.maxInstancedCount = N
instancedGeometry.attributes['offset'] = offsets
instancedGeometry.attributes['color'] = colors
print(instancedGeometry.attributes)

mesh = p3.Mesh( instancedGeometry, p3.MeshBasicMaterial(vertexColors='VertexColors') )
# mesh = p3.Mesh( instancedGeometry, p3.MeshBasicMaterial(color='red') )

view_width = 600
view_height = 400
camera = p3.PerspectiveCamera(position=[20, 0, 0], aspect=view_width/view_height)
key_light = p3.DirectionalLight(position=[0, 10, 10])
ambient_light = p3.AmbientLight()
scene = p3.Scene(children=[mesh, camera, key_light, ambient_light], background="#000000")
controller = p3.OrbitControls(controlling=camera)
renderer = p3.Renderer(camera=camera, scene=scene, controls=[controller],
                    width=view_width, height=view_height)
renderer

but nothing appears. I tried to just put a plain red color for the material but it didn't help. My guess is that i'm doing something wrong with the material, maybe? Thanks!

[nvaytet]

I've also noticed something strange in Jupyter. If I do

import pythreejs as p3
import numpy as np

geometry = p3.BoxBufferGeometry(
    width=5,
    height=10,
    depth=15)

morph = p3.BufferGeometry.from_geometry(geometry)
print(morph.attributes)

inside one cell, it prints an empty dict.

However, if I put the print statement in a cell below, it prints

{'position': BufferAttribute(array=array([[ 2.5,  5. ,  7.5],
       [ 2.5,  5. , -7.5],
       [ 2.5, -5. ,  7.5],
       [ 2.5, -5. , -7.5],
       [-2.5,  5. , -7.5],
       [-2.5,  5. ,  7.5],
       [-2.5, -5. , -7.5],
       [-2.5, -5. ,  7.5],
       [-2.5,  5. , -7.5],
       [ 2.5,  5. , -7.5],
       [-2.5,  5. ,  7.5],
       [ 2.5,  5. ,  7.5],
       [-2.5, -5. ,  7.5],
       [ 2.5, -5. ,  7.5],
       [-2.5, -5. , -7.5],
       [ 2.5, -5. , -7.5],
       [-2.5,  5. ,  7.5],
       [ 2.5,  5. ,  7.5],
       [-2.5, -5. ,  7.5],
       [ 2.5, -5. ,  7.5],
       [ 2.5,  5. , -7.5],
       [-2.5,  5. , -7.5],
       [ 2.5, -5. , -7.5],
       [-2.5, -5. , -7.5]], dtype=float32), normalized=True, version=1), 'normal': BufferAttribute(array=array([[ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [ 1.,  0.,  0.],
       [-1.,  0.,  0.],
       [-1.,  0.,  0.],
       [-1.,  0.,  0.],
       [-1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  1.,  0.],
       [ 0., -1.,  0.],
       [ 0., -1.,  0.],
       [ 0., -1.,  0.],
       [ 0., -1.,  0.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0.,  1.],
       [ 0.,  0., -1.],
       [ 0.,  0., -1.],
       [ 0.,  0., -1.],
       [ 0.,  0., -1.]], dtype=float32), normalized=True, version=1), 'uv': BufferAttribute(array=array([[0., 1.],
       [1., 1.],
       [0., 0.],
       [1., 0.],
       [0., 1.],
       [1., 1.],
       [0., 0.],
       [1., 0.],
       [0., 1.],
       [1., 1.],
       [0., 0.],
       [1., 0.],
       [0., 1.],
       [1., 1.],
       [0., 0.],
       [1., 0.],
       [0., 1.],
       [1., 1.],
       [0., 0.],
       [1., 0.],
       [0., 1.],
       [1., 1.],
       [0., 0.],
       [1., 0.]], dtype=float32), normalized=True, version=1)}

It seems something gets updated only after the cell is executed.

I have the same problem from inside a Python script.

I created a function in a file:

import pythreejs as p3
import numpy as np

def make_geom():

    geometry = p3.BoxBufferGeometry(
        width=5,
        height=10,
        depth=15)

    morph = p3.BufferGeometry.from_geometry(geometry)
    print(morph.attributes)

    instancedGeometry = p3.InstancedBufferGeometry()
    for key, attr in morph.attributes.items():
        instancedGeometry.attributes[key] = attr
    print(instancedGeometry.attributes)

    return morph, instancedGeometry

and then from inside Jupyter:

from test_instancing import make_geom
morph, inst = make_geom()

outputs

{}
{}

However, if I then do in a separate cell

print(morph.attributes)

I get the populated dict, but doing

print(inst.attributes)

prints an empty dict. So it seems after the cell is executed, the attributes of morph are updated, but the attributes of inst never get populated, because at the time the for loop over morphs atributes is executed, the attributes don't exist and so inst never gets any attributes.

Any ideas? Do I have to manually run some update/synchronization function?

[nvaytet]

Ah, this seems related to #256

[vidartf]

I don't have time to read all of the above, but I can at least say that you should avoid using from_geometry. That function should really come with a big warning.

[vidartf]

Also, I'll repeat this bit: For InstancedBufferGeometry, it is better if you supply the attributes in one go, and pass during init. In general with widgets, modifying lists or dictionaries in-place are likely to not work as you hope (it will not create change events, and might actually prevent them from firing later on as well).

[nvaytet]

Hi @vidartf , Once again, thanks for your input. I've made some further progress, but I am struggling to finish the final touches.

I've dropped from_geometry in favour of specifying the vertices of the box by hand. The example below is giving me 50 boxes, but they are all distributed along the x axis instead of being randomly placed in 3D space, and the colours are all shades of red instead of being random colors. It's as if the shader is only seeing one dimension/column in my arrays.

import pythreejs as p3
import numpy as np

vertices = np.array(
    [[-0.5, -0.5, -0.5], [0.5, -0.5, -0.5], [0.5, 0.5, -0.5],
     [-0.5, 0.5, -0.5], [-0.5, -0.5, 0.5], [0.5, -0.5, 0.5],
     [0.5, 0.5, 0.5], [-0.5, 0.5, 0.5]],
    dtype=np.float32)
faces = np.array([[0, 4, 3], [3, 4, 7], [1, 2, 6], [1, 6, 5],
                  [0, 1, 5], [0, 5, 4], [2, 3, 7], [2, 7, 6],
                  [0, 2, 1], [0, 3, 2], [4, 5, 7], [5, 6, 7]],
                 dtype=np.uint32)

N = 50
colors = p3.InstancedBufferAttribute(array=np.random.random([N, 3]))
offsets = p3.InstancedBufferAttribute(array=(np.random.random([N, 3]) - 0.5) * 100.0)

instancedGeometry = p3.InstancedBufferGeometry(
    maxInstancedCount=N,
    attributes={
        "position": p3.BufferAttribute(array=vertices),
        "index": p3.BufferAttribute(array=faces.ravel()),
        "offset": offsets,
        "color": colors        
    })

material = p3.ShaderMaterial(
    vertexShader='''
precision highp float;
attribute vec3 offset;
varying vec3 mypos;
varying vec4 vColor;
void main(){

    mypos = position;
    mypos.x += offset.x;
    mypos.y += offset.y;
    mypos.z += offset.z;

    vColor = vec4( color, 1.0);
    gl_Position = projectionMatrix * modelViewMatrix * vec4( mypos, 1.0 );
}
''',
    fragmentShader='''
precision highp float;
varying vec4 vColor;
void main() {
    gl_FragColor = vec4( vColor );
}
''',
    vertexColors='VertexColors'
)

mesh = p3.Mesh( instancedGeometry,  material)

view_width = 600
view_height = 400
camera = p3.PerspectiveCamera(position=[20, 0, 0], aspect=view_width/view_height)
key_light = p3.DirectionalLight(position=[0, 10, 10])
ambient_light = p3.AmbientLight()
scene = p3.Scene(children=[mesh, camera, key_light, ambient_light], background="#000000")
controller = p3.OrbitControls(controlling=camera)
renderer = p3.Renderer(camera=camera, scene=scene, controls=[controller],
                    width=view_width, height=view_height)
renderer

Any help is once again much appreciated!

[vidartf]

The InstancedBufferAttribute is missing the meshPerAttribute argument (which defaults to 1). So you should ravel the position/color and add the missing argument:

colors = p3.InstancedBufferAttribute(array=np.random.random([N * 3]), meshPerAttribute=3)
offsets = p3.InstancedBufferAttribute(array=(np.random.random([N * 3]) - 0.5) * 100.0, meshPerAttribute=3)

[vidartf]

Actually, you can keep the array in the shape that it is, just specify meshPerAttribute=3.

[vidartf]

Relevant threejs example line for reference: https://github.com/mrdoob/three.js/blob/7c1424c5819ab622a346dd630ee4e6431388021e/examples/webgl_buffergeometry_instancing.html#L152

[nvaytet]

Ah thanks so much! It finally worked :-) You made my day

In case you are interested, i've been using this for representing neutron physics instruments in the jupyter notebook (and you can also play with in the docs): https://scipp.github.io/scipp-neutron/instrument-view.html So far I was creating the entire mesh by hand without instancing, but I will shortly change to implement the solution you just gave me.

[vidartf]

Cool :) You might benefit from passing antialias=True to the Renderer constructor 👍

[nvaytet]

:+1: How much does antialias=True affect performance? I would potentially have 1-2 million boxes to represent the detector pixels...

[nvaytet]

@vidartf I still think this is not entirely solved, as using meshPerAttribute=3 in the code above does not draw 50 boxes but only 17 (which coincidently is ~50/3).

It seems to me that in the line you are pointing to in the webgl_buffergeometry_instancing.html#L152 example, the numbers 3 (and 4 in the line below) in the InstancedBufferAttribute constructor are actually setting the itemSize and not the meshPerAttribute. https://threejs.org/docs/#api/en/core/InstancedBufferAttribute

But using meshPerAttribute=3 is giving me almost what I need, just not enough boxes. Do I have to count the triangles (=12) instead of the boxes?

[nvaytet]

I apologise in advance for the long message.

Trying to use a more predictable example, I figured I could simply multiply the maxInstanceCount by 3:

import pythreejs as p3
import numpy as np

vertices = np.array(
[[-2.5, -1.5, -0.5], [2.5, -1.5, -0.5], [2.5, 1.5, -0.5],
 [-2.5, 1.5, -0.5], [-2.5, -1.5, 0.5], [2.5, -1.5, 0.5],
 [2.5, 1.5, 0.5], [-2.5, 1.5, 0.5]],
dtype=np.float32)

faces = np.array([[0, 4, 3], [3, 4, 7], [1, 2, 6], [1, 6, 5],
                  [0, 1, 5], [0, 5, 4], [2, 3, 7], [2, 7, 6],
                  [0, 2, 1], [0, 3, 2], [4, 5, 7], [5, 6, 7]],
                 dtype=np.uint32)

colors = p3.InstancedBufferAttribute(
    array=np.array([[1, 1, 1], [1, 0, 0], [0, 1, 0], [0, 0, 1]],
                   dtype=np.float32),
    meshPerAttribute=3)
offsets = p3.InstancedBufferAttribute(
    array=np.array([[0, 0, 0], [1, 0, 0], [0, 1, 0], [0, 0, 1]],
                   dtype=np.float32) * 7.0,
    meshPerAttribute=3)

instancedGeometry = p3.InstancedBufferGeometry(
    maxInstancedCount=4*3,
    attributes={
        "position": p3.BufferAttribute(array=vertices),
        "index": p3.BufferAttribute(array=faces.ravel()),
        "offset": offsets,
        "color": colors
    })

material = p3.ShaderMaterial(
    vertexShader='''
precision highp float;
attribute vec3 offset;
varying vec3 vPosition;
varying vec4 vColor;
void main(){

    vPosition = position;
    vPosition += offset;

    vColor = vec4( color, 1.0);
    gl_Position = projectionMatrix * modelViewMatrix * vec4( vPosition, 1.0 );
}
''',
    fragmentShader='''
precision highp float;
varying vec4 vColor;
void main() {
    gl_FragColor = vec4( vColor );
}
''',
    vertexColors='VertexColors'
)

mesh = p3.Mesh( instancedGeometry,  material)

view_width = 600
view_height = 400
camera = p3.PerspectiveCamera(position=[20, 0, 0], aspect=view_width/view_height)
scene = p3.Scene(children=[mesh, camera, p3.AxesHelper()], background="#000000")
controller = p3.OrbitControls(controlling=camera)
renderer = p3.Renderer(camera=camera, scene=scene, controls=[controller],
                    width=view_width, height=view_height)
renderer

which gives 4 rectangular boxes, with white, red, green and blue colors, located at x=y=z=0, x=7, y=7, z=7, respectively.

However, I then tried adding some rotations and it would seem I am rendering too many boxes, and the rotations are not working out as expected

import pythreejs as p3
import numpy as np

vertices = np.array(
    [[-2.5, -1.5, -0.5], [2.5, -1.5, -0.5], [2.5, 1.5, -0.5],
     [-2.5, 1.5, -0.5], [-2.5, -1.5, 0.5], [2.5, -1.5, 0.5],
     [2.5, 1.5, 0.5], [-2.5, 1.5, 0.5]],
    dtype=np.float32)

faces = np.array([[0, 4, 3], [3, 4, 7], [1, 2, 6], [1, 6, 5],
                  [0, 1, 5], [0, 5, 4], [2, 3, 7], [2, 7, 6],
                  [0, 2, 1], [0, 3, 2], [4, 5, 7], [5, 6, 7]],
                 dtype=np.uint32)

colors = p3.InstancedBufferAttribute(
    array=np.array([[1, 1, 1], [1, 0, 0], [0, 1, 0], [0, 0, 1]],
                   dtype=np.float32),
    meshPerAttribute=3)
offsets = p3.InstancedBufferAttribute(
    array=np.array([[0, 0, 0], [1, 0, 0], [0, 1, 0], [0, 0, 1]],
                   dtype=np.float32) * 7.0,
    meshPerAttribute=3)
rotations = p3.InstancedBufferAttribute(
    array=np.array([[0., 0., 0., 0.],        # no rotation
                    [0.7071, 0, 0, 0.7071],  # rotate 90deg around x axis
                    [0, 0.7071, 0, 0.7071],  # rotate 90deg around y axis
                    [0, 0, 0.7071, 0.7071]], # rotate 90deg around z axis
                   dtype=np.float32),
    meshPerAttribute=4)

instancedGeometry = p3.InstancedBufferGeometry(
    maxInstancedCount=12,
    attributes={
        "position": p3.BufferAttribute(array=vertices),
        "index": p3.BufferAttribute(array=faces.ravel()),
        "offset": offsets,
        "color": colors,
        "rotation": rotations,
    })

material = p3.ShaderMaterial(
    vertexShader='''
precision highp float;
attribute vec3 offset;
attribute vec4 rotation;
varying vec3 vPosition;
varying vec4 vColor;
void main(){

    vPosition = position + 2.0 * cross( rotation.xyz, cross( rotation.xyz, position ) + rotation.w * position );
    vPosition += offset;

    vColor = vec4( color, 0.5);
    gl_Position = projectionMatrix * modelViewMatrix * vec4( vPosition, 1.0 );
}
''',
    fragmentShader='''
precision highp float;
varying vec4 vColor;
void main() {
    gl_FragColor = vec4( vColor );
}
''',
    vertexColors='VertexColors',
    transparent=True
)

mesh = p3.Mesh( instancedGeometry,  material)

view_width = 600
view_height = 400
camera = p3.PerspectiveCamera(position=[20, 0, 0], aspect=view_width/view_height)
scene = p3.Scene(children=[mesh, camera, p3.AxesHelper()], background="#000000")
controller = p3.OrbitControls(controlling=camera)
renderer = p3.Renderer(camera=camera, scene=scene, controls=[controller],
                    width=view_width, height=view_height)
renderer

There should be one red box rotated 90deg around the x axis, a green box rotated 90deg around the y axis, and one blue box rotated 90deg around the z axis.

[vidartf]

I'm sorry to say, but my time to help on this project is rather limited, so I don't have the opportunity to do in-depth troubleshooting. Of course if you find anything you believe is a bug in how pythreejs passes the information to threejs, I'm happy to fix that.

[nvaytet]

Sure, I definitely know what it’s like to try and help out on (or keep up to date) a project which is not your main task/job.

I’ll first try to recreate my example in native threejs and then do the equivalent in pythreejs to see if there is indeed a translation bug somewhere.

Cheers