no point cloud result was generated

[oym050922021]

hi, Thank you for the open source code. ###questions####### I would like to ask you some questions. During model training, a point_cloud.ply file was generated at iteration 7000, but no point cloud results were saved at iteration 30000, and there were no error messages either. What could be the reason for this? Thank you very much.

####Training log###### (base) root@VM-0-13-ubuntu:/home/ubuntu/projects/gaussian-splatting# cat train-2-21.log nohup: ignoring input Optimizing Output folder: ./output/52735481-0 [21/02 18:13:22] Tensorboard not available: not logging progress [21/02 18:13:22] Reading camera 624/624 [21/02 18:13:26] Loading Training Cameras [21/02 18:13:26] Loading Test Cameras [21/02 18:14:40] Number of points at initialisation : 526103 [21/02 18:14:40] Training progress: 100%|██████████| 30000/30000 [44:55:29<00:00, 5.39s/it, Loss=0.0217425]

####Training parameters configuration###### the parameters configuration of the train.py ： if name == "main":

Set up command line argument parser

parser = ArgumentParser(description="Training script parameters")
lp = ModelParams(parser)
op = OptimizationParams(parser)
pp = PipelineParams(parser)
parser.add_argument('--ip', type=str, default="127.0.0.1")
parser.add_argument('--port', type=int, default=6009)
parser.add_argument('--debug_from', type=int, default=-1)
parser.add_argument('--detect_anomaly', action='store_true', default=False)
parser.add_argument("--test_iterations", nargs="+", type=int, default=[7_000, 30_000])
parser.add_argument("--save_iterations", nargs="+", type=int, default=[7_000, 30_000])
parser.add_argument("--quiet", action="store_true")
parser.add_argument("--checkpoint_iterations", nargs="+", type=int, default=[])
parser.add_argument("--start_checkpoint", type=str, default = None)
args = parser.parse_args(sys.argv[1:])
args.save_iterations.append(args.iterations)

print("Optimizing " + args.model_path)

...

the parameters configuration of the gaussian-splatting/arguments/init.py ： class ModelParams(ParamGroup): def init(self, parser, sentinel=False): self.sh_degree = 3 self._source_path = "/home/ubuntu/projects/gaussian-splatting/datasets/dji_0843_ds15"
self._model_path = "" self._images = "images" self._resolution = 2 self._white_background = False
self.data_device = "cpu" self.eval = False #####False super().init(parser, "Loading Parameters", sentinel) ...

class OptimizationParams(ParamGroup): def init(self, parser): self.iterations = 30_000 self.position_lr_init = 0.000160.1
self.position_lr_final = 0.00000160.1
self.position_lr_delay_mult = 0.01 self.position_lr_max_steps = 30_000 self.feature_lr = 0.0025 self.opacity_lr = 0.05 self.scaling_lr = 0.001 self.rotation_lr = 0.001 self.percent_dense = 0.01 self.lambda_dssim = 0.2 self.densification_interval = 100 self.opacity_reset_interval = 3000 self.densify_from_iter = 500 self.densify_until_iter = 15_000 self.densify_grad_threshold = 0.0002 self.random_background = False super().init(parser, "Optimization Parameters") ...

[GaneshBannur]

Was the CPU memory almost full before the 30,000^th iteration? If yes then your issue could be due to an OOM occurring when it tries to save the point cloud (Gaussians). See https://github.com/graphdeco-inria/gaussian-splatting/issues/235#issuecomment-1956467415 for a solution.

[oym050922021]

@GaneshBannur Thank you for your reply. The GPU memory keeps increasing during model iterations. The command for checking is "watch -n 1 nvidia-smi".

[GaneshBannur]

The GPU memory increasing during training is normal. I was talking about the CPU memory usage, not GPU memory. Near the end of training, is the CPU memory almost fully used? If CPU memory is almost fully used then when it tries to save a point cloud at iteration 30,000 an Out-Of-Memory error will occur and the point cloud will not be saved.

[oym050922021]

@GaneshBannur Yes, the CPU is almost fully used before reaching 30,000 iterations.