skywolf829 / GSTK

Gaussian Splatting toolkit application. One stop shop for preprocessing your dataset, training your model with human-in-the-loop training, and editing saved GSplat PLY files.
MIT License
4 stars 1 forks source link

Design "selector" paradigm in backend #14

Closed skywolf829 closed 6 months ago

skywolf829 commented 7 months ago

In the frontend, we want to allow the user to select groups of points. This selector should be viewable in the frontend render window, and in the backend there should be some representation for a selector to be used within the renderer or other code that edits the gaussians. This issue is only related to the backend's representation of the selectors, nothing with the frontend.

Types of selectors

At the moment, there are 2 simple selectors that should be considered, but also consider expanding this:

The selectors should be adjustable (scale, position, possibly rotation), and allow some way to see if the points are selected or not for use in rendering/point removal/other operations.

Extra

skywolf829 commented 6 months ago

Using new WebGPU (pygfx) backend renderer, we can use the gizmo attachment for transforming any mesh. Next is to find an efficient way to query if a point is inside of the object.

skywolf829 commented 6 months ago

Created selector object, uses a mesh + a gizmo. Only a box selector now, but others can be implemented as well.

Updated CUDA backend to accept a tensor for booleans for whether the primitive is inside or outside the selector. More convenient and adaptable to future selectors. Custom CUDA code for each selector would be more efficient FPS wise, but would require a not-as-flexible support for different selectors, and will be required to be re-ran per frame, vs in python we can update the selector buffer as necessary.

skywolf829 commented 6 months ago

Selector complete. Only uses WebGPU renderer when the selector is active in the frontend via an edit tool. Depending on tool, the selection mask is enabled/disabled (need to see for point removal, but not very useful for point adding). Properly scales alpha values for points outside the selection mask.