skywolf829 / GSTK

Gaussian Splatting toolkit application. One stop shop for preprocessing your dataset, training your model with human-in-the-loop training, and editing saved GSplat PLY files.

MIT License

4 stars 1 forks source link

Rendering and sending image to frontend #10

Closed skywolf829 closed 7 months ago

skywolf829 commented 7 months ago

Given the current viewing camera and gaussian model, the scene should be rendered and returned to the frontend as frequently as possible both during training and while it is paused.

Requirements

May need some render thread that will continue rendering while a camera and gaussian model exist
Each rendered frame should be sent to the frontend (compressed/encoded first?)

Extra

If rendering speed is slow, adjust render resolution adaptively.
If training while rendering, allow a slider for how much time to spend rendering vs training.

skywolf829 commented 7 months ago

Images rendering and sent to the frontend for display. With DEBUG mode, sends noise each frame essentially.

TODO:

See about further encoding the data instead of the default pickle protocol multiprocessing.connection uses.
Test with real model
See if the 20 fps I get on my laptop is okay or if there is a problem. Seems to just be torch is slow at instanciating 800x800x3 every frame on my laptop

skywolf829 commented 7 months ago

So far, default pickle protocol seems okay, only small overhead from the actual send call. Tested with real model and everything is working smoothly.