Use my own camera position and point cloud data

[meriemjabri]

I already have camera positions/parameters and point cloud data, is there any way to use them instead of the ones generated by colmap? I already know how to use my own camera positions/parameters ( by creating camera.txt, points3D.txt and images.txt but points3D.txt is empty and I don't know how to add point cloud information to it ).

[Liquidmasl]

I came here with the same question.

I have high density laser scanned pointclouds with color information, and i have 360° panorama images every few meters. Do you see a way how i could use this directly? I guess it needs the images to train right? not only for making a pointcloud

Anyway, awesome project, I would absolutely love to get this run on laser scanned pointclouds

[meriemjabri]

check this to use your own camera settings https://colmap.github.io/faq.html. I think we need to adjust the point3d.txt file so that we can upload our own point cloud data, but it seems that here https://github.com/graphdeco-inria/gaussian-splatting/issues/118 it uploads them directly to the transforms.json file.

[lrwillis]

Me too. I am trying to use the panorama images from laser scanning data. Right now I'm trying to just use the images, no point cloud data. I've created the cameras, images and points3d text files, used colmap to convert this to binary representation, but training crashes with an error.

Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 216, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 90, in training loss.backward() File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\autograd__init__.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 2 - got [0, 0, 3] but expected shape compatible with [0, 16, 3] Training progress: 0%|

[lrwillis]

Here is a small example dataset that illustrates the crash.
garage.zip

After unzipping, run "python train.py -s data/garage"

[Liquidmasl]

I believe when laser scanning we are lacking enough images to derive a pointcloud from it. we will need the laser scanned point data. internally the scan has some relationship between panos and points, but i fear it will be a 1:1 relationship, one point to one pano :/

[lrwillis]

I can easily populate the point positions and color values, but I don't know what to put for the other fields (ERROR, TRACK)

https://colmap.github.io/format.html

POINT3D_ID, X, Y, Z, R, G, B, ERROR, TRACK[] as (IMAGE_ID, POINT2D_IDX)

Error as measured by what?
Track?

[grgkopanas]

Not important. Definitely ignored on our side.

On Tue, Sep 26, 2023, 19:15 lrwillis @.***> wrote:

I can easily populate the point positions and color values, but I don't know what to put for the other fields (ERROR, TRACK)

https://colmap.github.io/format.html POINT3D_ID, X, Y, Z, R, G, B, ERROR, TRACK[] as (IMAGE_ID, POINT2D_IDX)

Error as measured by what? Track?

— Reply to this email directly, view it on GitHub https://github.com/graphdeco-inria/gaussian-splatting/issues/205#issuecomment-1735961577, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACGXXYMLBOJDS6T4RHJO6E3X4MESBANCNFSM6AAAAAA424K7UM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Liquidmasl]

I can easily populate the point positions and color values, but I don't know what to put for the other fields (ERROR, TRACK)

https://colmap.github.io/format.html

POINT3D_ID, X, Y, Z, R, G, B, ERROR, TRACK[] as (IMAGE_ID, POINT2D_IDX)

Error as measured by what? Track?

i think track is a list of ids of images which where used to triangulate its position and error is the reprojection error from different viewpoints. but i might be very off.

Not important. Definitely ignored on our side.

yeah get it, but can you maybe give us a few tips? do you think its even feasable to adapt this to use a existing high density laser scanned pointcloud with panorama images (and their position and rotation)? Scanning very large scale scenes (we have scans above 100,000 m², clouds with 200gb) is still WAY faster and more accurate with mobile laser scanners, adapting this project to be able to be used with those scans would be incredible. Whole buildings as walkable digital twins with high accuracy, construction sights, virtual stores, etc. The combination of the 2 technologies would be amazing.

Or do you think there is no way because some crucial step would be missed/skipped?

Me, as an outsider who has basically no idea, thinks that what we are really missing is the reference from a point to all images where it might be seen, so its size can be trained... ?

[chris-aeviator]

@Liquidmasl

internally the scan has some relationship between panos and points, but i fear it will be a 1:1 relationship, one point to one pano :/

how would we need the data to be present, do we need a one image pixel to one point relationship? How is that data then picked up inside the code of this repo?

[Liquidmasl]

@Liquidmasl

internally the scan has some relationship between panos and points, but i fear it will be a 1:1 relationship, one point to one pano :/

how would we need the data to be present, do we need a one image pixel to one point relationship? How is that data then picked up inside the code of this repo?

honestly i am just guessing, I have no clue what is needed really, i just try to extrapolate from the little info i have.

I think in their case they have a relationship from 1 point to A BUNCH of pictures. so they can train the slot size by comparing to a lot of images. but for lidar scanner and panos, we just have a picture every few meters, so there arent this many pictures where a given point is visible.

but damn i still think this would be an increcible usecase i think lidar pointclouds are a lot denser and more accurate, i really wanna use them for splotting :/

[ThePassedWind]

Has anyone solved this problem? I have a same task but have no idea to implement this until now.

[cywm39]

Me too. I am trying to use the panorama images from laser scanning data. Right now I'm trying to just use the images, no point cloud data. I've created the cameras, images and points3d text files, used colmap to convert this to binary representation, but training crashes with an error.

Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 216, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 90, in training loss.backward() File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\autogradinit.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 2 - got [0, 0, 3] but expected shape compatible with [0, 16, 3] Training progress: 0%|

I have the same problem. Have you solved it? @lrwillis

[lrwillis]

No, I haven’t worked on this since I last posted. I am still very interested in solving this issue, but it’s a spare-time project, other work priorities have preempted me.

Sent from Mail for Windows

From: cywm39 Sent: Thursday, November 2, 2023 8:04 AM To: graphdeco-inria/gaussian-splatting Cc: lrwillis; Mention Subject: Re: [graphdeco-inria/gaussian-splatting] Use my own camera positionand point cloud data (Issue #205)

Me too. I am trying to use the panorama images from laser scanning data. Right now I'm trying to just use the images, no point cloud data. I've created the cameras, images and points3d text files, used colmap to convert this to binary representation, but training crashes with an error. Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 216, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 90, in training loss.backward() File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\autogradinit.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 2 - got [0, 0, 3] but expected shape compatible with [0, 16, 3] Training progress: 0%| I have the same problem. Have you solved it? @lrwillis — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

[lincob]

Guys, let's forget about it; this is never going to work. First, only the perspective camera model is supported, so there's no chance to use equirectangular... Okay, you could reproject the panoramas into several perspective frames (or even use the original frames - before stitching - from the scan if available, though they usually are of the worst quality, even on very high-end scanners). BUT first, they won't have a very wide focal length, so orienting them properly is going to be a nightmare. They also won't have enough overlap. And second, it's just like a classic photogrammetric workflow, unless you have a scan done every meter or so (which I doubt).

No, the only question here is whether it's possible to use a laser scan as the starting sparse cloud, a step that is still pending investigation by me, but you'd still need proper pictures with good coverage and overlap just like in photogrammetry. I've tried using a theta Z1, for interiors the results so far have been pretty poor. For exteriors it may be different, but quite frankly, when converted to perspective frames the quality is quite subpar because at the end it's still two crappy fisheye lenses to start from. Good results could probably be achieved from a high end DSLR with ultra wide rectilinear lens like 14mm or 12mm (I think I've read somewhere that the fisheye camera model was not supported either), haven't got time to test either. Good luck counting the hours passing by, I already went too far down that rabbit hole 🤣

[Liquidmasl]

Guys, let's forget about it; this is never going to work. First, only the perspective camera model is supported, so there's no chance to use equirectangular... Okay, you could reproject the panoramas into several perspective frames (or even use the original frames - before stitching - from the scan if available, though they usually are of the worst quality, even on very high-end scanners). BUT first, they won't have a very wide focal length, so orienting them properly is going to be a nightmare. They also won't have enough overlap. And second, it's just like a classic photogrammetric workflow, unless you have a scan done every meter or so (which I doubt).

No, the only question here is whether it's possible to use a laser scan as the starting sparse cloud, a step that is still pending investigation by me, but you'd still need proper pictures with good coverage and overlap just like in photogrammetry. I've tried using a theta Z1, for interiors the results so far have been pretty poor. For exteriors it may be different, but quite frankly, when converted to perspective frames the quality is quite subpar because at the end it's still two crappy fisheye lenses to start from. Good results could probably be achieved from a high end DSLR with ultra wide rectilinear lens like 14mm or 12mm (I think I've read somewhere that the fisheye camera model was not supported either), haven't got time to test either. Good luck counting the hours passing by, I already went too far down that rabbit hole 🤣

well i mean eqirectangular is quite an easy format, i think the transformation is actually quite a bit easier to understand then perspective. so yeah, perspective is the only supported.. for now haha

and yes we use the VLX3, so we have scans every few centimeters, and a panorama image every few meters, so the cloud is not sparse at all, its super dense.

But sadly i just dont have the time to research this monster project far enough to even come close implementing somthing

[Lee-JaeWon]

@lincob @ThePassedWind @meriemjabri @lrwillis

I'd like to use KITTI dateset. Have you solved it? I don't understand how to convert KITTI dataset into COLMAP format, but I hope someone has solved it.

[markinruc]

Me too. I am trying to use the panorama images from laser scanning data. Right now I'm trying to just use the images, no point cloud data. I've created the cameras, images and points3d text files, used colmap to convert this to binary representation, but training crashes with an error. Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 216, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 90, in training loss.backward() File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\autogradinit.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 2 - got [0, 0, 3] but expected shape compatible with [0, 16, 3] Training progress: 0%|

I have the same problem. Have you solved it? @lrwillis

I also have the same problem. The training stops at 10%. Have you tried to solve it? Or we can talk about the possible solution? @cywm39 @lrwillis

[Li-colonel]

Me too. I am trying to use the panorama images from laser scanning data. Right now I'm trying to just use the images, no point cloud data. I've created the cameras, images and points3d text files, used colmap to convert this to binary representation, but training crashes with an error. Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 216, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 90, in training loss.backward() File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\autogradinit.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 2 - got [0, 0, 3] but expected shape compatible with [0, 16, 3] Training progress: 0%|

I have the same problem. Have you solved it? @lrwillis

I also have the same problem. The training stops at 10%. Have you tried to solve it? Or we can talk about the possible solution? @cywm39 @lrwillis

Hi, there. I have encountered the same issue with a single-view image as well. It seems like this was caused by the alpha-reset starting when iteration=3000, and something bad occurs when density for every 100 iterations.

Perhaps we could adjust the arguments to fit such single-view scene optimizing. But I haven't locate the specific problem yet. Thus I have no idea which parameters will directly affect this issue either.

Please let me konw if you make any new discoveries, thank you.

[JinJinSang]

Not important. Definitely ignored on our side. … On Tue, Sep 26, 2023, 19:15 lrwillis @.> wrote: I can easily populate the point positions and color values, but I don't know what to put for the other fields (ERROR, TRACK) https://colmap.github.io/format.html POINT3D_ID, X, Y, Z, R, G, B, ERROR, TRACK[] as (IMAGE_ID, POINT2D_IDX) Error as measured by what? Track? — Reply to this email directly, view it on GitHub <#205 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACGXXYMLBOJDS6T4RHJO6E3X4MESBANCNFSM6AAAAAA424K7UM . You are receiving this because you are subscribed to this thread.Message ID: @.>

What do you mean by 'ignored'?

I am also try to transform my own lidar generated point clouds and images into the colmap data structure. You mean for the image.txt and pointcloud.txt, we don't have to correctly write the point registration between 2d keypoints and 3d points infomation into the file. We can even wirte all zeros in these values but just list all the image parameters and the pose info of 3D points right?

[lincob]

@Liquidmasl do you have access to individual unstitched frames from the different lenses on the VLX3 ? Tomorrow I'm surveying with TLS and UAV, it's a small project, I'll try to figure out something about the pointcloud, but yeah, time, time, time... This rabbit from Alice in wonderland is always breathing down my neck !

[lincob]

@Lee-JaeWon you're looking to start from the stereo images ?

[Liquidmasl]

@Liquidmasl do you have access to individual unstitched frames from the different lenses on the VLX3 ? Tomorrow I'm surveying with TLS and UAV, it's a small project, I'll try to figure out something about the pointcloud, but yeah, time, time, time... This rabbit from Alice in wonderland is always breathing down my neck !

it least on some scans i do have the unstiched images. They are very distored though. generally equirectangular mapping is very easy in code, so i think the panos should be fine. just unsure if there are even enough images to make the splatting algo work.

[Liquidmasl]

Not important. Definitely ignored on our side.

i think yall might be sleeping on a huge usecase. laser scanning is booming! Would be awesome if our company would have the time to put some resources into this, maybe at some point

[LakeofRain]

Me too. I am trying to use the panorama images from laser scanning data. Right now I'm trying to just use the images, no point cloud data. I've created the cameras, images and points3d text files, used colmap to convert this to binary representation, but training crashes with an error. Training progress: 0%| | 0/30000 [00:00<?, ?it/s]Traceback (most recent call last): File "train.py", line 216, in training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from) File "train.py", line 90, in training loss.backward() File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "C:\Users\leerw\miniconda3\envs\gaussian_splatting\lib\site-packages\torch\autogradinit.py", line 175, in backward allow_unreachable=True, accumulate_grad=True) # Calls into the C++ engine to run the backward pass RuntimeError: Function _RasterizeGaussiansBackward returned an invalid gradient at index 2 - got [0, 0, 3] but expected shape compatible with [0, 16, 3] Training progress: 0%|

I have the same problem. Have you solved it? @lrwillis

I also have the same problem. The training stops at 10%. Have you tried to solve it? Or we can talk about the possible solution? @cywm39 @lrwillis

Hi, there. I have encountered the same issue with a single-view image as well. It seems like this was caused by the alpha-reset starting when iteration=3000, and something bad occurs when density for every 100 iterations.

Perhaps we could adjust the arguments to fit such single-view scene optimizing. But I haven't locate the specific problem yet. Thus I have no idea which parameters will directly affect this issue either.

Please let me konw if you make any new discoveries, thank you.

I have this problem too. It seems to be related to the gaussians.reset_opacity() function, but I have not solved this yet.