[visualize normal] Normals are displayed incorrectly

[QiuYilin]

operating system: Windows 10/Windows 11 PCL version: 1.14.0/1.14.1 https://github.com/PointCloudLibrary/pcl/commit/fc54abc12f34f902ed47c2ebec704e05601a9476 /8e2d2839fe73a955d49b9a72861de2becf2da compiler:MSVC 2022 17.8.34309/17.9.5

Details of the environment where I had the problem： vcpkg baseline: 326d8b43e365352ba3ccadf388d989082fe0f2a6 vcpkg install pcl[qt,vtk,cuda]

The pcl (1.13.1 without cuda) in my old version of vcpkg is correct.

I want to visualize the normals of a point cloud, but the display looks weird.

My code:

#define PCL_NO_PRECOMPILE

#include <pcl/features/normal_3d_omp.h>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/visualization/pcl_visualizer.h>

#include <iostream>

int main() {

  pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
  if (pcl::io::loadPCDFile<pcl::PointXYZ>(
          "D:/program_on_git/own/test/pcl_test/source/croped.pcd", *cloud) ==
      -1) {
    PCL_ERROR("Couldn't read file.\n");
    return -1;
  }

  pcl::NormalEstimationOMP<pcl::PointXYZ, pcl::Normal> ne;
  ne.setInputCloud(cloud);
  ne.setNumberOfThreads(8);
  pcl::search::KdTree<pcl::PointXYZ>::Ptr tree(
      new pcl::search::KdTree<pcl::PointXYZ>());
  ne.setSearchMethod(tree);
  pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
  ne.setKSearch(100); 
  ne.setViewPoint(100, 100, 1000);
  ne.compute(*normals);

  pcl::visualization::PCLVisualizer viewer("Point Cloud Viewer");
  viewer.setBackgroundColor(0.0, 0.0, 0.0);
  viewer.addPointCloud<pcl::PointXYZ>(cloud, "cloud");
  viewer.addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(
      cloud, normals, cloud->size() / 100, 20, "normals");
  viewer.setPointCloudRenderingProperties(
      pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 1, "cloud");
  viewer.spin();

  return 0;
}

The source cloud has 1280000 points. If I extract 432 keypoints from it ,the normals computed will be correct

[larshg]

Could you share the pointcloud? I have visualized quite a few pointclouds over time and it worked well - so I doubt its a bug in PCL.

[QiuYilin]

@larshg https://github.com/QiuYilin/test/blob/main/pcl_test/source/croped.pcd This is the pointcloud.

[mvieth]

• Which operating system do you use?
• Which PCL version do you use? How did you install it?
• Which compiler do you use?

[QiuYilin]

@mvieth Sorry I forget to say that.
operating system: Windows 11 PCL version: 1.14.0 fc54abc12f34f compiler:MSVC 2022

[larshg]

Tested it on ubuntu rq:

[QiuYilin]

@larshg I found that after removing the invalid points, it was displayed normally, but I did not see any instructions or errors that prohibit the use of invalid points for normal estimation.Why not process it into an algorithm that can filter out invalid points on the underlying mechanism, or are invalid points meaningful in some cases（It is true that invalid values are used when keeping organized, so why not let all classes accept this ordered structure?）? At present, I have a vague feeling that some classes are sensitive to invalid points and some are not. Is there any unified guiding principle?

[larshg]

Do you build in release or debug - I can verify that the normals is wrong when running in debug mode for some (probably memory indexing) reason.

Not sure if its due to invalid points though - I can try add that to verify.

[mvieth]

For normal estimation, invalid points should not be a problem. The normal for an invalid point will simply be (NaN, NaN, NaN). And addPointCloudNormals also checks whether point and normal are valid: https://github.com/PointCloudLibrary/pcl/blob/master/visualization/include/pcl/visualization/impl/pcl_visualizer.hpp#L909 You could also test what happens if you pretend that your point cloud is unorganized, by adding this after normal estimation:

cloud->width = cloud->width * cloud->height;
cloud->height = 1;
normals->width = normals->width * normals->height;
normals->height = 1;

[QiuYilin]

@mvieth Modified code pretending unorganized:

#define PCL_NO_PRECOMPILE

#include <pcl/features/normal_3d_omp.h>
#include <pcl/filters/filter_indices.h>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/visualization/pcl_visualizer.h>

#include <iostream>

int main() {
  pcl::PointCloud<pcl::PointXYZ>::Ptr cloud(new pcl::PointCloud<pcl::PointXYZ>);
  if (pcl::io::loadPCDFile<pcl::PointXYZ>(
          "D:/program_on_git/own/test/pcl_test/source/croped.pcd", *cloud) ==
      -1) {
    PCL_ERROR("Couldn't read file.\n");
    return -1;
  }

  //   std::vector<int> vec;
  //   pcl::removeNaNFromPointCloud(*cloud, *cloud, vec);

  pcl::NormalEstimationOMP<pcl::PointXYZ, pcl::Normal> ne;
  ne.setInputCloud(cloud);
  ne.setNumberOfThreads(8);
  pcl::search::KdTree<pcl::PointXYZ>::Ptr tree(
      new pcl::search::KdTree<pcl::PointXYZ>());
  ne.setSearchMethod(tree);
  pcl::PointCloud<pcl::Normal>::Ptr normals(new pcl::PointCloud<pcl::Normal>);
  ne.setKSearch(100);
  ne.setViewPoint(100, 100, 1000);
  ne.compute(*normals);

  cloud->width = cloud->width * cloud->height;
  cloud->height = 1;
  normals->width = normals->width * normals->height;
  normals->height = 1;

  pcl::visualization::PCLVisualizer viewer("Point Cloud Viewer");
  viewer.setBackgroundColor(0.0, 0.0, 0.0);
  viewer.addPointCloud<pcl::PointXYZ>(cloud, "cloud");
  viewer.addPointCloudNormals<pcl::PointXYZ, pcl::Normal>(
      cloud, normals, cloud->size() / 100, 20, "normals");
  viewer.setPointCloudRenderingProperties(
      pcl::visualization::PCL_VISUALIZER_POINT_SIZE, 1, "cloud");
  viewer.spin();

  return 0;
}

display: (no normals)

[QiuYilin]

@larshg I build in debug.

[QiuYilin]

I noticed that if I use BilateralFilter on this point cloud, if the invalid points are not removed, the program will report an error. Is this a feature of BilateralFilter or a related bug?

Mycode:

#include <pcl/filters/bilateral.h>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/search/kdtree.h>

#include <iostream>

int main() {
  pcl::PointCloud<pcl::PointXYZI>::Ptr cloud(
      new pcl::PointCloud<pcl::PointXYZI>);
  if (pcl::io::loadPCDFile<pcl::PointXYZI>(
          "D:/program_on_git/own/test/pcl_test/source/croped.pcd", *cloud) ==
      -1) {
    PCL_ERROR("Couldn't read input file.\n");
    return -1;
  }

  // if (cloud->is_dense == false) {
  //   std::cout << "Input cloud is not dense." << std::endl;
  // }

  // std::vector<int> vec;
  // pcl::removeNaNFromPointCloud(*cloud, *cloud, vec);

  pcl::BilateralFilter<pcl::PointXYZI> bilateral_filter;
  bilateral_filter.setInputCloud(cloud);
  auto tree = std::make_shared<pcl::search::KdTree<pcl::PointXYZI>>();
  bilateral_filter.setSearchMethod(tree);

  bilateral_filter.setHalfSize(0.05);
  bilateral_filter.setStdDev(0.1);

  pcl::PointCloud<pcl::PointXYZI>::Ptr filtered_cloud(
      new pcl::PointCloud<pcl::PointXYZI>);
  bilateral_filter.filter(*filtered_cloud);

  pcl::io::savePCDFileASCII("filtered_cloud.pcd", *filtered_cloud);
  std::cout << "Filtered cloud saved." << std::endl;

  return 0;
}

report

Assertion failed: point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to radiusSearch!", file C:\src\vcpkg\buildtrees\pcl\src\head\05601a9476-82ae3636f9.clean\kdtree\include\pcl/kdtree/impl/kdtree_flann.hpp, line 376
Debug Error!

Program: ...ram_on_git\own\test\pcl_test\build\Debug\bilateral_filter.exe

abort() has been called

(Press Retry to debug the application)

[mvieth]

I noticed that if I use BilateralFilter on this point cloud, if the invalid points are not removed, the program will report an error. Is this a feature of BilateralFilter or a related bug?

BilateralFilter should check for invalid points before calling radiusSearch. So yes, that is a bug in BilateralFilter.

[QiuYilin]

I tried updating the version of vcpkg and recompiling pcl, but the problem still didn't solve.

[QiuYilin]

The normals can be displayed correctly in pcl 1.13.1. However,new problems about "#define PCL_NO_PRECOMPILE" were generated:

[build] lld-link : error : undefined symbol: omp_get_num_procs [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:52
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::setNumberOfThreads(unsigned int))
[build]   
[build] lld-link : error : undefined symbol: __kmpc_global_thread_num [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:63
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:310
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::NNIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:229
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::LshIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced 1 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_push_num_threads [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:73
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:102
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:327
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::NNIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced 6 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_fork_call [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:73
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:102
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(private: virtual void __cdecl pcl::NormalEstimationOMP<struct pcl::PointXYZ, struct pcl::Normal>::computeFeature(class pcl::PointCloud<struct pcl::Normal> &))
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:327
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(public: virtual int __cdecl flann::NNIndex<struct flann::L2_Simple<float>>::knnSearch(class flann::Matrix<float> const &, class flann::Matrix<unsigned __int64> &, class flann::Matrix<float> &, unsigned __int64, struct flann::SearchParams const &) const)
[build]   >>> referenced 6 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_for_static_init_8 [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:79
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:108
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.1)
[build]   
[build] lld-link : error : undefined symbol: __kmpc_for_static_fini [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:73
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\pcl\features\impl\normal_3d_omp.hpp:102
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.1)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced 6 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_for_static_init_4 [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:241
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.8)
[build]   >>> referenced 4 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_reduce [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:241
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.8)
[build]   >>> referenced 4 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_end_reduce [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced 11 more times
[build]   
[build] lld-link : error : undefined symbol: __kmpc_barrier [C:\program_on_git\own\test\pcl_test\build\normal_estimation.vcxproj]
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:330
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.3)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\nn_index.h:345
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.5)
[build]   >>> referenced by C:\src\sfr-vcpkg\installed\x64-windows\include\flann\algorithms\lsh_index.h:241
[build]   >>>               normal_estimation.dir\Debug\normal_estimation.obj:(.omp_outlined._debug__.8)
[build]   >>> referenced 4 more times

[QiuYilin]

At present, I have reverted to a version that can display normally：

Windows 10(Version 22H2 19045.4291)

MSVC 2019(14.29.30133)

vcpkg.json

{
    "dependencies":[
        {
            "name": "pcl",
            "version>=": "1.12.0",
            "features": ["vtk"]
        }
    ],
    "builtin-baseline":"8eb57355a4ffb410a2e94c07b4dca2dffbee8e50",
    "overrides":[
        {
            "name": "pcl",
            "version": "1.13.1#1"
        }
    ]
}

I hope there is a hint that the 1.14.1 version of pcl on windows can run normally.

[mvieth]

Okay, I may have found the problem. As far as I can tell, the normals are only displayed incorrectly if Windows is used, and the debug configuration is used (release works fine), and the cloud or normals given to the visualizer contain invalid values (NaN). As I mentioned earlier, the addPointCloudNormals function does check for invalid values. However, the last few entries of pts are left uninitialized if invalid values are encountered, and when pts is later passed to vtk the full size is given instead of only the number of valid entries in pts. So I think if nr_normals is updated with cell_count or j respectively, it should work correctly. I am not sure why this is only a problem on Windows in debug configuration, but maybe memory allocated with new is filled with special values in that situation? I will create a pull request soon to fix this.

[QiuYilin]

If I switch to release, will there be any other problems besides the normal display? I switched to the release mode of 1.14.1 and encountered the following situation: SamplingSurfaceNormal will crash if the parameters are set to 0. Is this normal?

[mvieth]

SamplingSurfaceNormal will crash if the parameters are set to 0. Is this normal?

Which parameter? sample or ratio?

[QiuYilin]

SamplingSurfaceNormal will crash if the parameters are set to 0. Is this normal?

Which parameter? sample or ratio?

both