render point clouds

[gabrielalexandrelopes]

Let me start by saying that pixi3d is quite impressive. It is the simplest way to develop 3D apps using webGL that I have found. Thank you!

Regarding my question, is it possible to render large point clouds with good performance? or add a new point cloud class? I'm trying to find a good way to render point clouds from lidars at 20Hz.

Best regards

[jnsmalm]

Hey! Thank you, glad you like it!

Do you mean this type of thing? https://www.youtube.com/watch?v=h8XbxnCGvCQ

A bit busy right now, but will try to make an example of this.

[jnsmalm]

Roughly, how many points are we talking about?

[gabrielalexandrelopes]

Yes, similar to the video, but loading the data from a lidar sensor, for example: this video

For some of these sensors you can easily get 20000 points but in practice displaying 2000 points is sufficient. Thanks for taking a look at this!!

[jnsmalm]

Here is a full example how to render point cloud, let me know if you have any questions. Hope it's ok with TypeScript.

import { Application, Assets, DRAW_MODES, Geometry, Program, Renderer, Shader, TYPES } from "pixi.js"
import { CameraOrbitControl, LightingEnvironment, ImageBasedLighting, Model, Mesh3D, Light, LightType, ShadowCastingLight, ShadowQuality, Material, Camera, MeshShader, Point3D, MeshGeometry3D, Color } from "pixi3d/pixi7"

let app = new Application({
  backgroundColor: 0xdddddd, resizeTo: window, antialias: true
})
document.body.appendChild(app.view as HTMLCanvasElement)

let vert = `
  attribute vec3 a_Position;
  attribute vec3 a_Color;
  varying vec3 v_Color;
  uniform mat4 u_Model;
  uniform mat4 u_ViewProjection;
  uniform float u_Size;
  void main() {
    v_Color = a_Color;
    gl_Position = u_ViewProjection * u_Model * vec4(a_Position, 1.0);
    gl_PointSize = u_Size;
  }`

let frag = `
  varying vec3 v_Color; 
  void main() {
    gl_FragColor = vec4(v_Color, 1.0);
  }`

class CustomMaterial extends Material {
  constructor() {
    super()
    this.drawMode = DRAW_MODES.POINTS
  }
  updateUniforms(mesh: Mesh3D, shader: Shader) {
    shader.uniforms.u_Model = mesh.worldTransform.array
    shader.uniforms.u_ViewProjection = Camera.main.viewProjection.array
    shader.uniforms.u_Size = 3.0
  }
  createShader() {
    return new MeshShader(Program.from(vert, frag))
  }
}

function createPointCloudMesh(points: Point3D[], colors: Color[]) {
  let geometry = new MeshGeometry3D()
  geometry.positions = {
    buffer: new Float32Array(points.length * 3)
  }
  for (let i = 0; i < points.length; i++) {
    geometry.positions.buffer[i * 3 + 0] = points[i].x
    geometry.positions.buffer[i * 3 + 1] = points[i].y
    geometry.positions.buffer[i * 3 + 2] = points[i].z
  }
  geometry.colors = {
    buffer: new Float32Array(colors.length * 3)
  }
  for (let i = 0; i < colors.length; i++) {
    let rgb = colors[i].rgb
    geometry.colors.buffer[i * 3 + 0] = rgb[0]
    geometry.colors.buffer[i * 3 + 1] = rgb[1]
    geometry.colors.buffer[i * 3 + 2] = rgb[2]
  }
  return new Mesh3D(geometry, new CustomMaterial())
}

let numberOfPoints = 20000

let points: Point3D[] = []
for (let i = 0; i < numberOfPoints; i++) {
  points.push(new Point3D(-1 + Math.random() * 2, -1 + Math.random() * 2, -1 + Math.random() * 2))
}

let colors: Color[] = []
for (let i = 0; i < numberOfPoints; i++) {
  colors.push(new Color(1, Math.random(), 1))
}

let pointCloud = app.stage.addChild(createPointCloudMesh(points, colors))

let rotation = 0
app.ticker.add(() => {
  pointCloud.rotationQuaternion.setEulerAngles(0, rotation++, 0)
})

let control = new CameraOrbitControl(app.view as HTMLCanvasElement);

[gabrielalexandrelopes]

This works great, thank you for responding so fast!

For our robotics applications we need to update the point cloud very often, so I'm trying to add this function:

function updatePointCloud(points: Point3D[], pointCloud: Mesh3D)
{
  for (let i = 0; i < points.length; i++) {
    pointCloud.geometry.positions.buffer[i * 3 + 0] = points[i].x
    pointCloud.geometry.positions.buffer[i * 3 + 1] = points[i].y
    pointCloud.geometry.positions.buffer[i * 3 + 2] = points[i].z
  }
}

And then update the ticker with that function:

app.ticker.add(() => {
  pointCloud.rotationQuaternion.setEulerAngles(0, rotation++, 0)

  let points: Point3D[] = []
  for (let i = 0; i < numberOfPoints; i++) {
    points.push(new Point3D(-1 + Math.random() * 2, -1 + Math.random() * 2, -1 + Math.random() * 2))
  }

  updatePointCloud(points, pointCloud)
})

It's not working yet because of an error: 'pointCloud.geometry.positions' is possibly 'undefined'. I'm going to try to solve it, I'm new to typescript.

If this works well with point clouds then we are on a path to build an ultra lightweight replacement for the visualisation part of gazebo. Unfortunately ROS and Gazebo have grown to be quite large, which makes them hard to use in some industry environments. With pixi3D we can do it with a few lines of code. Thanks again!

[jnsmalm]

Cool use case for Pixi3D! Will you always have the same amount of points or will it be different each time?

TypeScript is giving you a warning that that 'positions' might be undefined. You can remove that warning by checking if the property exists:

if (pointCloud.geometry.positions) {
    pointCloud.geometry.positions.buffer[i * 3 + 0] = points[i].x
    pointCloud.geometry.positions.buffer[i * 3 + 1] = points[i].y
    pointCloud.geometry.positions.buffer[i * 3 + 2] = points[i].z
}

You can also just use JavaScript instead, just remove all the types. Might also want to do some optimisations, might be some performance issues with creating many Point3D's each frame - but it depends.

[gabrielalexandrelopes]

yes, we use the same number of points all the times. Indeed I need better optimisations too 😅

[gabrielalexandrelopes]

so after updating the points in the buffer, do we need to issue some refresh command to redraw the openGL with the new points?

[jnsmalm]

I can add some improvements to your update-function tomorrow.

[jnsmalm]

Created a slightly modified example where the points are updated each frame. This example recreates all the Point3D objects which is very ineffective. I don't know what the lidar data looks like, but I would try to directly transfer that data to the geometry without creating Point3D's in-between.

import { Application, DRAW_MODES, Program, Shader } from "pixi.js"
import { CameraOrbitControl, Mesh3D, Material, Camera, MeshShader, Point3D, MeshGeometry3D, Color, Container3D } from "pixi3d/pixi7"

let app = new Application({
  backgroundColor: 0xdddddd, resizeTo: window, antialias: true
})
document.body.appendChild(app.view as HTMLCanvasElement)

let vert = `
  attribute vec3 a_Position;
  attribute vec3 a_Color;
  varying vec3 v_Color;
  uniform mat4 u_Model;
  uniform mat4 u_ViewProjection;
  uniform float u_Size;
  void main() {
    v_Color = a_Color;
    gl_Position = u_ViewProjection * u_Model * vec4(a_Position, 1.0);
    gl_PointSize = u_Size;
  }`

let frag = `
  varying vec3 v_Color; 
  void main() {
    gl_FragColor = vec4(v_Color, 1.0);
  }`

class CustomMaterial extends Material {
  constructor() {
    super()
    this.drawMode = DRAW_MODES.POINTS
  }
  get shader() {
    return this._shader
  }
  updateUniforms(mesh: Mesh3D, shader: Shader) {
    shader.uniforms.u_Model = mesh.worldTransform.array
    shader.uniforms.u_ViewProjection = Camera.main.viewProjection.array
    shader.uniforms.u_Size = 3.0
  }
  createShader() {
    return new MeshShader(Program.from(vert, frag))
  }
}

class PointCloud extends Container3D {
  mesh: Mesh3D
  material = new CustomMaterial()

  constructor(private numberOfPoints: number) {
    super()

    let geometry = new MeshGeometry3D()
    geometry.positions = {
      buffer: new Float32Array(numberOfPoints * 3)
    }
    geometry.colors = {
      buffer: new Float32Array(numberOfPoints * 3)
    }
    this.mesh = this.addChild(new Mesh3D(geometry, this.material))
  }

  update(points: Point3D[], colors: Color[]) {
    if (!this.material.shader) {
      // The shader hasn't been created yet
      return
    }
    let geometry = this.mesh.geometry.getShaderGeometry(this.material.shader)
    if (!geometry) {
      // The geometry hasn't been created yet
      return
    }
    let pointBuffer = geometry.getBuffer("a_Position")
    let colorBuffer = geometry.getBuffer("a_Color")

    for (let i = 0; i < this.numberOfPoints; i++) {
      (pointBuffer.data as Float32Array).set(points[i].array, i * 3);
      (colorBuffer.data as Float32Array).set(colors[i].rgb, i * 3);
    }
    pointBuffer.update()
    colorBuffer.update()
  }
}

function generatePoints(numberOfPoints: number) {
  let points: Point3D[] = []
  for (let i = 0; i < numberOfPoints; i++) {
    points.push(new Point3D(-1 + Math.random() * 2, -1 + Math.random() * 2, -1 + Math.random() * 2))
  }
  let colors: Color[] = []
  for (let i = 0; i < numberOfPoints; i++) {
    colors.push(new Color(1, Math.random(), 1))
  }
  return { points, colors }
}

let numberOfPoints = 20000
let pointCloud = app.stage.addChild(new PointCloud(20000))
let rotation = 0

app.ticker.add(() => {
  const { points, colors } = generatePoints(numberOfPoints)
  pointCloud.update(points, colors)
  pointCloud.rotationQuaternion.setEulerAngles(0, rotation++, 0)
})

let control = new CameraOrbitControl(app.view as HTMLCanvasElement);

[gabrielalexandrelopes]

That works great, thanks! It uses a bit of CPU but the memory footprint is small. The data from the lidars is usually an array of bytes with [x,y,z,intensity]. We can reshape it to any byte structure or make a JSON. I will try to prepare tomorrow a sample scan.

[gabrielalexandrelopes]

Hi! I created an example by modifying slightly your code:

https://github.com/gabrielalexandrelopes/pixi3d-lidar

At the moment I'm using a synthetic lidar of a sphere moving around just for testing purposes. It works quite well, this proves the concept! The next steps are to optimise the code and create some communication channel to interface with an actually lidar.

Thanks a lot for the help!

[jnsmalm]

Awesome, let me know how the project goes!