jdavidberger
commented
2 years ago To make it easier to test:
#include "open3d/Open3D.h"
#include "open3d/t/geometry/Image.h"
using namespace open3d;
using core::Tensor;
using t::geometry::PointCloud;
using t::geometry::RGBDImage;
int main(int argc, char *argv[]) {
auto voxel_grid = t::geometry::VoxelBlockGrid::Load(argv[1]);
auto intrinsic_t = Tensor::Init<double>(
{{1537.13, 0.0, 962.311},
{0, 1537.66, 585.374},
{0, 0, 1}});
auto extrinsics_t = Tensor::Init<double>(
{{-0.888227, -0.446662, 0.107447, 0.00176051,},
{-0.238145, 0.647672, 0.723746, 0.0136757},
{-0.39286, 0.617263, -0.68165, 0.865006},
{0, 0, 0, 1}});
auto input_depth_tensor = Tensor::Load(argv[2]);
core::Device device("CUDA:0");
auto empty_depth = std::make_shared<open3d::t::geometry::Image>(input_depth_tensor);
auto block_coords = voxel_grid.GetUniqueBlockCoordinates(empty_depth->To(device), intrinsic_t, extrinsics_t);
std::vector<std::string> attrs = {"depth", "color"};
auto rgbd_map = voxel_grid.RayCast(block_coords, intrinsic_t, extrinsics_t,
1920,
1080,
attrs,
1000.0,
.1,
3.0,
0, 8, 8);
auto depth = open3d::t::geometry::Image(rgbd_map["depth"]).ToLegacy();
auto color = open3d::t::geometry::Image(rgbd_map["color"]).ToLegacy();
auto rgbd = geometry::RGBDImage::CreateFromColorAndDepth(color, depth, 1000, 3, true);
visualization::DrawGeometries({rgbd}, "RGBD", 1920 * 2,
1080);
}Ran with the file path as args for: https://drive.google.com/file/d/1HGRqOTGA2bYpCVzTQFKZKI9hjTOhTG6X/view?usp=sharing https://drive.google.com/file/d/1xDjLWyWqV2IJpPUxOtZLXlSZohXMRBaG/view?usp=sharing
respecively produces this:
Since I had it now in C++ I hacked it abit to make sure it was running on CPU, and the python attempt was not. But when running (both frustrum and raycast) on CPU I get:
which is likely good enough for tracking although it does have a few obvious blocks missing.
glxinfo snippet:
Memory info (GL_NVX_gpu_memory_info):
Dedicated video memory: 16384 MB
Total available memory: 16384 MB
Currently available dedicated video memory: 10897 MB
OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: NVIDIA GeForce RTX 3080 Laptop GPU/PCIe/SSE2
OpenGL core profile version string: 4.6.0 NVIDIA 510.73.05
OpenGL core profile shading language version string: 4.60 NVIDIA
nfrankisrg
Buffyqsf
Checklist
masterbranch).Describe the issue
I'm working on code derived from the new SLAM reconstruction model and so far it works very well and is very fast, with the exception of situations in which the raycasting to generate synthetic frames produces bad output. This always seems to be the case for larger voxel grid sizes -- say, .010 but I have seen it even on .002 voxel grid sizes.
.01 output
The artifacting is worse the larger the voxel grid is.
Raycast:
Pointcloud:
Frustrum (plotted voxels from -.5 to .5 tsdf, same frustrum given to raycast)
Raycast superimposed on PC:
Raycast superimposed over frustrum
Note that my tooling regenerates everything to turn on frustrum visibility; each call to raycast generates different splotchy output seemingly at random
.003 output
Same artifacts at .003 on same dataset
Things I've tried:
This didnt change the behavior; which implies it's not a GPU issue although it is sorta hard to verify externally whether that is truly running on CPU or not.
Steps to reproduce the bug
Error message
No response
Expected behavior
I'd expect the raytracing to mostly look like the generated pointcloud from the given perspective assuming generation inputs -- weight threshold, voxelgrid, etc -- being the same.
Open3D, Python and System information
Additional information
No response