Training speed slowed from threads

[skywolf829]

Obeserved

Training speed in train.py for the mic test dataset reaches a max around ~220 updates per second. Meanwhile, training through the frontend app gives training speeds around ~120 fps.

I've noticed that when some threads are stopped, performance increases.

• Disabling the render thread gives +40 fps (mostly due to less GPU work, frontend is rendering at ~100 fps)
• Disabling communication thread gives + 80 fps
• Closing app and letting only backend run the training gives full fps (~220 fps)

Desired

The communication thread should not cause a slow down for backend training speed. If the renderer is completely disabled, backend training should run at full speed.

[skywolf829]

Multiprocessing doesn't seem like a solution here because too much data is shared. May need to use different communication backend if the multiprocessing.connection is the big slowdown.

[skywolf829]

Not exactly closed, but I re-worked the backend to not use as many threads so we have more direct control over the main thread and computation. Threads are only created for the initial socket setup (since its blocking and stops control+c) and for dataset loading since it's I/O bound.

Now, we have 1 main loop that will

1. read and process all messages available from the client
2. render the model from the render camera's viewpoint
3. perform train step

Some steps may be skipped if the model isn't initialized, training isn't turned on, etc.

With this implementation, our frontend has approx the same framerates as the official SIBR implementation during training on the mic dataset. 100-120 FPS while training with rendering at the same speed. Disabling rendering or training helps.

TODO

• JPEG encoding was updated as well to a faster library (from 5-6ms -> 2-3ms) with more control over quality now as well. However, 2-3ms when the entire train/render step is 10ms is significant. Faster encoding can bring the total train and render times down.