COLMAP pipeline gives faulty results on custom and vanilla data

Thanks for all the hard work on this!

Summary

The provided COLMAP pipeline is giving apparently faulty results, making it difficult to use my own custom data. I've confirmed this by running your given data through the pipeline with minimal changes to the code.

Overview

I'm attempting to run NeRF++ on my own custom dataset and I ran into very blurry and unusable results after running it through the COLMAP pipeline and training step. To isolate the issue, I ran the full dataset conversion on your dataset, specifically tat_training_Truck in your provided tanks and temples dataset.

I ran run_colmap.py on a new directory with just the rgb images of tat_training_Truck. Multiple issues arose when I visualized the results.

1. Focal point is out of frame in epipolar geometry visualization

I'm not terribly familiar with epipolar geometry, but I assume that the epipolar lines should converge within the view of the given frame (I assume this is the focal point? Please correct me if I'm wrong). This does not occur in the given dataset despite the camera pose pointing at the object of interest, which tells me that the outputted intrinsic matrix is incorrect. green camera is visible on left side of image, seemingly oriented and positioned correctly Screenshot from 2020-12-15 12-16-25 visualization of epipolar geometry of this pose

This tells me that there's some bug in the run_colmap.py pipeline that is causing a bad intrinsic matrix to result

2. Camera path not fully normalized to unit sphere

This was not an issue with my custom dataset, but it seems to be here. I visualized the automatic normalization that your script performed and the camera track did not get bound to the unit sphere. Additionally, there seems to be no built-in support for normalizing the kai_points.ply pointcloud. You seemed to have successfully normalized it in the example you gave, so I have two questions on this point:

How do you successfully normalize these camera poses within the unit sphere?
How do you normalize the kai_points.ply pointcloud and convert it to a mesh like you did in your example?

This comes straight out of the vanilla COLMAP pipeline, which is very different from the posted example

3. Blurry training results

I figure that this is a consequence of 1.. However, I can't demonstrate this for the vanilla data since its poses aren't successfully normalized according to 2.. Here's a sample of the blur experienced from training on a chair for many, many hours: Screenshot from 2020-12-15 12-55-58

I also wrote my own converter that takes this outputted COLMAP data and transforms it into NeRF++-readable format. I figure no bugs from there are present here since this is before that conversion even takes place. On that note, if you have official code for this process I'd also love to take a look.

End

Since I performed minimal modifications upon the code and I'm using vanilla data, I figure there's either a bug in the system or I'm doing this fundamentally improperly. Do you have any suggestions on how to fix this so that I can use my own custom data without running into these same issues?

Hi @dukeeagle , for the epipolar geometry visualization, if the camera parameters are correct, then for every colored keypoint in the left image, its colored corresponding epipolar line should go through the pixel in the right image corresponding to the colored left-image keypoint. This is apparently not the case in your example, which suggests that something went wrong with your running of colmap. Besides, it's perfectly fine for epipoles (intersection of all epipolar lines) to stay out of the image. Do you have epipolar visualizations for other datasets, e.g., your own dataset?

Thanks for the reply! I have epipolar visualizations of my own dataset and a similar issue seems to occur. Additionally, I made a fresh clone of nerfplusplus and re-ran COLMAP but the same output occurred, unfortunately. epipolar geometry_screenshot_18 12 2020 Here's an attempt to visualize epipolar geometry on my own data, the base of a chair in this case.

With regards to other questions I had - what's the approach you took to normalize the camera poses within the unit sphere for the truck example? And how'd you generate a mesh for the truck example rather than a pointcloud? Is this even an issue in your version of the pipeline?

Also curious to hear if you have any suggestions for solving this COLMAP issue, especially since it seems to be an issue on the provided, vanilla data. I think that my own results could be fixed if this issue is resolved

I also meet this problem,and how about your solution to this problem?

Thanks for the reply! I have epipolar visualizations of my own dataset and a similar issue seems to occur. Additionally, I made a fresh clone of nerfplusplus and re-ran COLMAP but the same output occurred, unfortunately. Here's an attempt to visualize epipolar geometry on my own data, the base of a chair in this case.

With regards to other questions I had - what's the approach you took to normalize the camera poses within the unit sphere for the truck example? And how'd you generate a mesh for the truck example rather than a pointcloud? Is this even an issue in your version of the pipeline?

Also curious to hear if you have any suggestions for solving this COLMAP issue, especially since it seems to be an issue on the provided, vanilla data. I think that my own results could be fixed if this issue is resolved

Thanks for all the hard work on this!

Summary

The provided COLMAP pipeline is giving apparently faulty results, making it difficult to use my own custom data. I've confirmed this by running your given data through the pipeline with minimal changes to the code.

Overview

I'm attempting to run NeRF++ on my own custom dataset and I ran into very blurry and unusable results after running it through the COLMAP pipeline and training step. To isolate the issue, I ran the full dataset conversion on your dataset, specifically tat_training_Truck in your provided tanks and temples dataset.

I ran run_colmap.py on a new directory with just the rgb images of tat_training_Truck. Multiple issues arose when I visualized the results.

1. Focal point is out of frame in epipolar geometry visualization

I'm not terribly familiar with epipolar geometry, but I assume that the epipolar lines should converge within the view of the given frame (I assume this is the focal point? Please correct me if I'm wrong). This does not occur in the given dataset despite the camera pose pointing at the object of interest, which tells me that the outputted intrinsic matrix is incorrect. green camera is visible on left side of image, seemingly oriented and positioned correctly visualization of epipolar geometry of this pose

This tells me that there's some bug in the run_colmap.py pipeline that is causing a bad intrinsic matrix to result

2. Camera path not fully normalized to unit sphere

This was not an issue with my custom dataset, but it seems to be here. I visualized the automatic normalization that your script performed and the camera track did not get bound to the unit sphere. Additionally, there seems to be no built-in support for normalizing the kai_points.ply pointcloud. You seemed to have successfully normalized it in the example you gave, so I have two questions on this point:

How do you successfully normalize these camera poses within the unit sphere?

How do you normalize the kai_points.ply pointcloud and convert it to a mesh like you did in your example?

This comes straight out of the vanilla COLMAP pipeline, which is very different from the posted example

3. Blurry training results

I figure that this is a consequence of 1.. However, I can't demonstrate this for the vanilla data since its poses aren't successfully normalized according to 2.. Here's a sample of the blur experienced from training on a chair for many, many hours:

I also wrote my own converter that takes this outputted COLMAP data and transforms it into NeRF++-readable format. I figure no bugs from there are present here since this is before that conversion even takes place. On that note, if you have official code for this process I'd also love to take a look.

End

Since I performed minimal modifications upon the code and I'm using vanilla data, I figure there's either a bug in the system or I'm doing this fundamentally improperly. Do you have any suggestions on how to fix this so that I can use my own custom data without running into these same issues?

I met the same problem. Have you deal with it?

I have no problem using the colmap scripts on my side. To help you debug this issue, here's the complete output of scripts on an example data: download from google drive. You might want to check if you can replicate the results on your side.

Regarding how to normalize a mesh, I added an example implementation using open3d, shown here.

Thanks for all the hard work on this!

Summary

The provided COLMAP pipeline is giving apparently faulty results, making it difficult to use my own custom data. I've confirmed this by running your given data through the pipeline with minimal changes to the code.

Overview

I'm attempting to run NeRF++ on my own custom dataset and I ran into very blurry and unusable results after running it through the COLMAP pipeline and training step. To isolate the issue, I ran the full dataset conversion on your dataset, specifically tat_training_Truck in your provided tanks and temples dataset. I ran run_colmap.py on a new directory with just the rgb images of tat_training_Truck. Multiple issues arose when I visualized the results.

1. Focal point is out of frame in epipolar geometry visualization

I'm not terribly familiar with epipolar geometry, but I assume that the epipolar lines should converge within the view of the given frame (I assume this is the focal point? Please correct me if I'm wrong). This does not occur in the given dataset despite the camera pose pointing at the object of interest, which tells me that the outputted intrinsic matrix is incorrect. green camera is visible on left side of image, seemingly oriented and positioned correctly visualization of epipolar geometry of this pose This tells me that there's some bug in the run_colmap.py pipeline that is causing a bad intrinsic matrix to result

2. Camera path not fully normalized to unit sphere

This was not an issue with my custom dataset, but it seems to be here. I visualized the automatic normalization that your script performed and the camera track did not get bound to the unit sphere. Additionally, there seems to be no built-in support for normalizing the kai_points.ply pointcloud. You seemed to have successfully normalized it in the example you gave, so I have two questions on this point:

How do you successfully normalize these camera poses within the unit sphere?

How do you normalize the kai_points.ply pointcloud and convert it to a mesh like you did in your example?

This comes straight out of the vanilla COLMAP pipeline, which is very different from the posted example

3. Blurry training results

I figure that this is a consequence of 1.. However, I can't demonstrate this for the vanilla data since its poses aren't successfully normalized according to 2.. Here's a sample of the blur experienced from training on a chair for many, many hours: I also wrote my own converter that takes this outputted COLMAP data and transforms it into NeRF++-readable format. I figure no bugs from there are present here since this is before that conversion even takes place. On that note, if you have official code for this process I'd also love to take a look.

End

Since I performed minimal modifications upon the code and I'm using vanilla data, I figure there's either a bug in the system or I'm doing this fundamentally improperly. Do you have any suggestions on how to fix this so that I can use my own custom data without running into these same issues?

I met the same problem. Have you deal with it?

Have you solved this problem? I have encountered the same problem and it is very bothering me. Thank you very much

Thanks for all the hard work on this!

Summary

The provided COLMAP pipeline is giving apparently faulty results, making it difficult to use my own custom data. I've confirmed this by running your given data through the pipeline with minimal changes to the code.

Overview

I'm attempting to run NeRF++ on my own custom dataset and I ran into very blurry and unusable results after running it through the COLMAP pipeline and training step. To isolate the issue, I ran the full dataset conversion on your dataset, specifically tat_training_Truck in your provided tanks and temples dataset.

I ran run_colmap.py on a new directory with just the rgb images of tat_training_Truck. Multiple issues arose when I visualized the results.

1. Focal point is out of frame in epipolar geometry visualization

I'm not terribly familiar with epipolar geometry, but I assume that the epipolar lines should converge within the view of the given frame (I assume this is the focal point? Please correct me if I'm wrong). This does not occur in the given dataset despite the camera pose pointing at the object of interest, which tells me that the outputted intrinsic matrix is incorrect. green camera is visible on left side of image, seemingly oriented and positioned correctly visualization of epipolar geometry of this pose

This tells me that there's some bug in the run_colmap.py pipeline that is causing a bad intrinsic matrix to result

2. Camera path not fully normalized to unit sphere

This was not an issue with my custom dataset, but it seems to be here. I visualized the automatic normalization that your script performed and the camera track did not get bound to the unit sphere. Additionally, there seems to be no built-in support for normalizing the kai_points.ply pointcloud. You seemed to have successfully normalized it in the example you gave, so I have two questions on this point:
1. How do you successfully normalize these camera poses within the unit sphere?

2. How do you normalize the `kai_points.ply` pointcloud and convert it to a mesh like you did in your example?
This comes straight out of the vanilla COLMAP pipeline, which is very different from the posted example

3. Blurry training results

I figure that this is a consequence of 1.. However, I can't demonstrate this for the vanilla data since its poses aren't successfully normalized according to 2.. Here's a sample of the blur experienced from training on a chair for many, many hours:

I also wrote my own converter that takes this outputted COLMAP data and transforms it into NeRF++-readable format. I figure no bugs from there are present here since this is before that conversion even takes place. On that note, if you have official code for this process I'd also love to take a look.

End

Since I performed minimal modifications upon the code and I'm using vanilla data, I figure there's either a bug in the system or I'm doing this fundamentally improperly. Do you have any suggestions on how to fix this so that I can use my own custom data without running into these same issues?

Have you solved this problem? I have encountered the same problem and it is very bothering me. Thank you very much

Kai-46 / nerfplusplus