search

pierotofy / OpenSplat

Production-grade 3D gaussian splatting with CPU/GPU support for Windows, Mac and Linux 🚀
https://antimatter15.com/splat/?url=https://splat.uav4geo.com/banana.splat
GNU Affero General Public License v3.0
887 stars 85 forks source link

A way to continue training #83

Open Garshishka opened 6 months ago

Garshishka commented 6 months ago

Considering that this program can use CPU and low VRam cards to train, how about adding a way or parameter to continue training from saved splat.ply? Is this even feasible?

pierotofy commented 6 months ago

I don't see why not.

  1. Modify savePly to store the current step count (in a comment PLY header value, maybe)
  2. Read PLY back into the tensors (reverse of savePly), read step count.
  3. Resume from the previous step count.

For a numerically correct resume, one should also dump the optimizer state but I don't think that would actually matter too much for the end result.

We'd welcome a pull request for this. Interested?

Garshishka commented 6 months ago

I would if I could :( But cpp and ML are an unknown to me

stefvfx commented 6 months ago

I think it would be very useful.

Itox001 commented 3 months ago

+1 for this feature. Currently I can only reasonably train ~3000 iterations before RAM consumption exhausts my resources because of the memory leak on MPS devices. I am hoping that stopping and resuming the training would reset this, allowing me to train for longer.

eloquentarduino commented 1 month ago

+1. I'm not a C++ guy so I can't help here.

AsherJingkongChen commented 2 weeks ago

+1 for this feature. Currently I can only reasonably train ~3000 iterations before RAM consumption exhausts my resources because of the memory leak on MPS devices. I am hoping that stopping and resuming the training would reset this, allowing me to train for longer.

So sad. I hope there would be a solution for you 😊.