ns-train complains of NaNs in PLY points 3d file

[edmondeng]

Describe the bug When I use ns-train splatfacto which includes converted data via ns-convert I get an error that is related to the .ply point cloud data file.

I processed my reality capture data and am able to train on the image/registration without the pointCloud, but when the point cloud is considered it throws the foillowing error

Traceback (most recent call last):
  File "/home/edmonde/miniconda3/envs/nerfstudio/bin/ns-train", line 8, in <module>
    sys.exit(entrypoint())
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 262, in entrypoint
    main(
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 247, in main
    launch(
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 189, in launch
    main_func(local_rank=0, world_size=world_size, config=config)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/scripts/train.py", line 99, in train_loop
    trainer.setup()
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/engine/trainer.py", line 153, in setup
    self.pipeline = self.config.pipeline.setup(
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/base_config.py", line 54, in setup
    return self._target(self, **kwargs)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/pipelines/base_pipeline.py", line 271, in __init__
    self._model = config.model.setup(
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/configs/base_config.py", line 54, in setup
    return self._target(self, **kwargs)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/models/splatfacto.py", line 205, in __init__
    super().__init__(*args, **kwargs)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/models/base_model.py", line 85, in __init__
    self.populate_modules()  # populate the modules
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/models/splatfacto.py", line 214, in populate_modules
    distances, _ = self.k_nearest_sklearn(means.data, 3)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/nerfstudio/models/splatfacto.py", line 345, in k_nearest_sklearn
    nn_model = NearestNeighbors(n_neighbors=k + 1, algorithm="auto", metric="euclidean").fit(x_np)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/sklearn/base.py", line 1152, in wrapper
    return fit_method(estimator, *args, **kwargs)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/sklearn/neighbors/_unsupervised.py", line 175, in fit
    return self._fit(X)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/sklearn/neighbors/_base.py", line 498, in _fit
    X = self._validate_data(X, accept_sparse="csr", order="C")
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/sklearn/base.py", line 605, in _validate_data
    out = check_array(X, input_name="X", **check_params)
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/sklearn/utils/validation.py", line 957, in check_array
    _assert_all_finite(
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/sklearn/utils/validation.py", line 122, in _assert_all_finite
    _assert_all_finite_element_wise(
  File "/home/edmonde/miniconda3/envs/nerfstudio/lib/python3.8/site-packages/sklearn/utils/validation.py", line 171, in _assert_all_finite_element_wise
    raise ValueError(msg_err)

ValueError: Input X contains NaN.
NearestNeighbors does not accept missing values encoded as NaN natively. For supervised learning, you might want to consider sklearn.ensemble.HistGradientBoostingClassifier and Regressor which accept missing values encoded as NaNs natively. Alternatively, it is possible to preprocess the data, for instance by using an imputer transformer in a pipeline or drop samples with missing values. See https://scikit-learn.org/stable/modules/impute.html You can find a list of all estimators that handle NaN values at the following page: https://scikit-learn.org/stable/modules/impute.html#estimators-that-handle-nan-values

Method for processing: ns-process-data realitycapture --data /home/edmonde/Documents/dataset/bloedel/images --output-dir /home/edmonde/Documents/dataset/bloedel/converted3d --csv /home/edmonde/Documents/dataset/bloedel/bloedel.csv --ply /home/edmonde/Documents/dataset/bloedel/blodedel_Model_3.ply

training cmd: ns-train splatfacto --max-num-iterations 1000000 --data /home/edmonde/Documents/dataset/bloedel/converted3d

Expected behavior Model trains and is faster and or better scene quality hopefully due to having used a point cloud instead of the random generatoin points.

I did a quicker style install via conda and its telling me I installed nerfstudio 1.1.3

[jb-ye]

It seems the point cloud exported from reality capture has NaN values. If you can share the RC data dump, I can try to reproduce your issue and potentially figure out a fix that filter out NaNs in the converted dataset.

[edmondeng]

Yes here it is. I was trying to clean it via meshlab etc. But good to know what you think. I have had success loading this into Postshot. But would prefer being using my linux machine (nerf studio). I had converted this via ns-process realitycapture and so this was the input file for that process. NaN_Model.zip

[edmondeng]

model.zip This is what ns-process created in the conversion dir which ns-train complains about.

[edmondeng]

were you able to reproduce this?

[edmondeng]

Closing, ended up this very specific data set I exported the point cloud model not registrations.