[Feature]: Visualization of scene of point clouds, meshes, and other 3d shapes over time

[etaoxing]

Description

Based on the latest docs, wandb currently supports 3D visualizations for the these filetypes: ['obj', 'gltf', 'glb', 'babylon', 'stl', 'pts.json'], as well as point clouds given as nx3, nx4, nx6.

It'd be nice to have a convenient way to do 3D scene visualizations over time (rather than a point cloud or mesh at a single instance in time).

Suggested Solution

Would like an API similar to:

scene = wandb.3DScene()

timesteps = ...

# sequence of point clouds
pcd_sequence = ...

# sequence of meshes
vertices_sequence = ...
faces_sequence = ...

for i, t in enumerate(timesteps):
    # probably need another parameter for length, assuming inputs are padded
    scene.add_pointcloud(t=t, pcd_sequence[i,...])
    scene.add_mesh(t=t, v=vertices_sequence[i,...], f=faces_sequence[i,...])

Would also be nice to be able to log batches of 3d object sequences.

Alternatives

I've found that meshplot is the easiest for HTML visualization of meshes / point clouds.

However, kaolin functionality is closer to the feature request, as they use USD format to store the underlying scene over time.

Additional Context

Related issue https://github.com/wandb/client/issues/3023#issue-1077225817 on optimizing 3d visualizations.

[nate-wandb]

Hi @etaoxing, thank you for the feature request along with a few possible solutions.

The request seems pretty straight forward but could you explain "Would also be nice to be able to log batches of 3d object sequences" a little? Are you asking for the ability to pass in several 3d object sequences at once rather than having to iterate over your data and pass in each individual sequence into wandb.3DScene()?

Thank you, Nate

[exalate-issue-sync[bot]]

Internal GitHub User commented:

Description

Based on the latest docs, wandb currently supports 3D visualizations for the these filetypes: ['obj', 'gltf', 'glb', 'babylon', 'stl', 'pts.json'], as well as point clouds given as nx3, nx4, nx6.

It'd be nice to have a convenient way to do 3D scene visualizations over time (rather than a point cloud or mesh at a single instance in time).

Suggested Solution

Would like an API similar to:

scene = wandb.3DScene()

timesteps = ...

# sequence of point clouds
pcd_sequence = ...

# sequence of meshes
vertices_sequence = ...
faces_sequence = ...

for i, t in enumerate(timesteps):
    # probably need another parameter for length, assuming inputs are padded
    scene.add_pointcloud(t=t, pcd_sequence[i,...])
    scene.add_mesh(t=t, v=vertices_sequence[i,...], f=faces_sequence[i,...])

Would also be nice to be able to log batches of 3d object sequences.

Alternatives

I've found that meshplot is the easiest for HTML visualization of meshes / point clouds.

However, kaolin functionality is closer to the feature request, as they use USD format to store the underlying scene over time.

Additional Context

Related issue https://github.com/wandb/client/issues/3023#issue-1077225817 on optimizing 3d visualizations.

[etaoxing]

Are you asking for the ability to pass in several 3d object sequences at once rather than having to iterate over your data and pass in each individual sequence into wandb.3DScene()?

Yes, if I'm generating multiple sequences of 3d objects, it'd be nice to add them to the scene together, and have a way to track which object is which. Possibly:

scene.add_object("object_name", timesteps, object_data_seq)
# or
scene.add_objects(["object1", "object2"], timesteps, [object1_data_seq, object2_data_seq])

[nate-wandb]

@etaoxing Thank you for the clarifications! I'll pass this on to our engineering team and follow up if they have any additional questions or if they are able to implement these asks in the product.

[nate-wandb]

Hi @etaoxing, I wanted to see if this solution would work for you in this case:

If you upload a 3D object multiple times to the same name it will actually show up in the UI with a step slider where you could scroll through each step.

So in this example you could iterate through each "frame" in your 3D scene and log it to the same name with wandb.log() and then use the step slider to visualize it over time in the UI.

[exalate-issue-sync[bot]]

WandB Internal User commented: nate-wandb commented: Hi @etaoxing, I wanted to see if this solution would work for you in this case:

If you upload a 3D object multiple times to the same name it will actually show up in the UI with a step slider where you could scroll through each step.

So in this example you could iterate through each "frame" in your 3D scene and log it to the same name with wandb.log() and then use the step slider to visualize it over time in the UI.

[etaoxing]

Hi @etaoxing, I wanted to see if this solution would work for you in this case:

If you upload a 3D object multiple times to the same name it will actually show up in the UI with a step slider where you could scroll through each step.

So in this example you could iterate through each "frame" in your 3D scene and log it to the same name with wandb.log() and then use the step slider to visualize it over time in the UI.

Yes, this can work for now, but latency (ref: Issue 3023 in the OP) is still a factor, and this will also clutter the media dashboard.

Ie. If I have a point cloud sequence of 60 steps, then I would need to move the slider to each step, to cache the point cloud files.

Maybe this could be resolved by adding an additional slider, so that the original slider corresponds to optimization steps, and a new slider corresponds to the point cloud timestep index? This way, all the point clouds at the current optimization step is cached. Would also be nice to cycle through the timestep slider automatically like a video.

[nate-wandb]

Hi @etaoxing, thank you for the additional context. I'll put in this feature request with our engineering team and follow up with you if this gets selected for development.