Bugs when doing parallel training

nyu-systems / Grendel-GS

Ongoing research training gaussian splatting at scale by distributed system

Apache License 2.0

389 stars 20 forks source link

Open LaFeuilleMorte opened 2 months ago

LaFeuilleMorte commented 2 months ago

Assertion Error: Each part between division_pos must be large enough

TarzanZhao commented 2 months ago

Hi, you can try enabling --no_heuristics_update and run it again. I think this should solve your issue

EhabWilson commented 2 months ago

I also found this problem when training on the dataset rubble. Does that mean workload balance is not necessary when training on the large scene?