search

hustvl / 4DGaussians

[CVPR 2024] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering
https://guanjunwu.github.io/4dgs/
Apache License 2.0
2.25k stars 186 forks source link

Problem with custom stereo dataset #141

Open lolwarmaze opened 5 months ago

lolwarmaze commented 5 months ago

Hi,

I am trying to train a 4D Gaussian model using my own set of stereo video pairs using dynerf type dataset directory structure. I have two video files (.mp4) from my stereo camera setup and have generated a poses_bounds.npy file with LLFF format because I was unable to generate camera poses for a stereo camera setup using COLMAP. Now I run into a problem when trying to generate the point clouds from my input data. It seems like it is unable to do triangulation as the image sees 0 points:

['r_000', 'png']
['r_001', 'png']
[[[ 1.77708380e-02 -9.99836743e-01 -3.26325977e-03  0.00000000e+00]
  [ 9.83394206e-01  1.80679709e-02 -1.80580720e-01  0.00000000e+00]
  [ 1.80610195e-01 -2.32830644e-10  9.83554780e-01  0.00000000e+00]]

 [[ 1.77708380e-02 -9.99836743e-01 -3.26325977e-03  1.19952627e-01]
  [ 9.83394206e-01  1.80679709e-02 -1.80580720e-01  0.00000000e+00]
  [ 1.80610195e-01 -2.32830644e-10  9.83554780e-01  0.00000000e+00]]]
I20240610 15:52:57.562021 139730277408768 misc.cc:198] 
==============================================================================
Feature extraction
==============================================================================
I20240610 15:53:00.702751 139728917594112 feature_extraction.cc:254] Processed file [1/2]
I20240610 15:53:00.702776 139728917594112 feature_extraction.cc:257]   Name:            r_000.png
I20240610 15:53:00.702779 139728917594112 feature_extraction.cc:283]   Dimensions:      1352 x 1014
I20240610 15:53:00.702782 139728917594112 feature_extraction.cc:286]   Camera:          #1 - SIMPLE_RADIAL
I20240610 15:53:00.702784 139728917594112 feature_extraction.cc:289]   Focal Length:    1622.40px
I20240610 15:53:00.702790 139728917594112 feature_extraction.cc:302]   Features:        7461
I20240610 15:53:00.853336 139728917594112 feature_extraction.cc:254] Processed file [2/2]
I20240610 15:53:00.853363 139728917594112 feature_extraction.cc:257]   Name:            r_001.png
I20240610 15:53:00.853368 139728917594112 feature_extraction.cc:283]   Dimensions:      1352 x 1014
I20240610 15:53:00.853372 139728917594112 feature_extraction.cc:286]   Camera:          #2 - SIMPLE_RADIAL
I20240610 15:53:00.853376 139728917594112 feature_extraction.cc:289]   Focal Length:    1622.40px
I20240610 15:53:00.853386 139728917594112 feature_extraction.cc:302]   Features:        7820
I20240610 15:53:00.857477 139730277408768 timer.cc:91] Elapsed time: 0.055 [minutes]
/home/zomhussa/4dgs/4DGaussians/database.py:11: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
  return array.tostring()
/home/zomhussa/4dgs/4DGaussians/database.py:17: DeprecationWarning: The binary mode of fromstring is deprecated, as it behaves surprisingly on unicode inputs. Use frombuffer instead
  return np.fromstring(blob, dtype=dtype).reshape(*shape)
I20240610 15:53:01.037186 139837249699840 misc.cc:198] 
==============================================================================
Exhaustive feature matching
==============================================================================
I20240610 15:53:01.085564 139837249699840 feature_matching.cc:231] Matching block [1/1, 1/1]
I20240610 15:53:01.134742 139837249699840 feature_matching.cc:46]  in 0.049s
I20240610 15:53:01.134897 139837249699840 timer.cc:91] Elapsed time: 0.002 [minutes]
I20240610 15:53:01.226635 140239616794624 misc.cc:198] 
==============================================================================
Loading model
==============================================================================
I20240610 15:53:01.226825 140239616794624 misc.cc:198] 
==============================================================================
Loading database
==============================================================================
I20240610 15:53:01.230371 140239616794624 database_cache.cc:54] Loading cameras...
I20240610 15:53:01.230443 140239616794624 database_cache.cc:64]  2 in 0.000s
I20240610 15:53:01.230483 140239616794624 database_cache.cc:72] Loading matches...
I20240610 15:53:01.230548 140239616794624 database_cache.cc:78]  1 in 0.000s
I20240610 15:53:01.230588 140239616794624 database_cache.cc:94] Loading images...
I20240610 15:53:01.231338 140239616794624 database_cache.cc:143]  2 in 0.001s (connected 2)
I20240610 15:53:01.231460 140239616794624 database_cache.cc:154] Building correspondence graph...
I20240610 15:53:01.231700 140239616794624 database_cache.cc:190]  in 0.000s (ignored 0)
I20240610 15:53:01.231756 140239616794624 timer.cc:91] Elapsed time: 0.000 [minutes]
I20240610 15:53:01.233173 140239616794624 misc.cc:198] 
==============================================================================
Triangulating image #1 (0)
==============================================================================
I20240610 15:53:01.233289 140239616794624 sfm.cc:473] => Image sees 0 / 974 points
I20240610 15:53:01.233537 140239616794624 sfm.cc:478] => Triangulated 0 points
I20240610 15:53:01.233571 140239616794624 misc.cc:198] 
==============================================================================
Triangulating image #2 (1)
==============================================================================
I20240610 15:53:01.233663 140239616794624 sfm.cc:473] => Image sees 0 / 974 points
I20240610 15:53:01.233927 140239616794624 sfm.cc:478] => Triangulated 0 points
I20240610 15:53:01.233962 140239616794624 misc.cc:198] 
==============================================================================
Retriangulation
==============================================================================
I20240610 15:53:01.234068 140239616794624 incremental_mapper.cc:175] => Completed observations: 0
I20240610 15:53:01.234098 140239616794624 incremental_mapper.cc:178] => Merged observations: 0
I20240610 15:53:01.234150 140239616794624 misc.cc:198] 
==============================================================================
Bundle adjustment
==============================================================================
F20240610 15:53:01.234272 140239616794624 sfm.cc:514] Check failed: bundle_adjuster.Solve(reconstruction.get()) 
    @     0x7f8c1c18578a  google::LogMessage::Fail()
    @     0x7f8c1c186f45  google::LogMessageFatal::~LogMessageFatal()
    @     0x556f5e693291  colmap::RunPointTriangulatorImpl(std::shared_ptr<colmap::Reconstruction> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&,�?
    @     0x556f5e693786  colmap::RunPointTriangulator(int, char**)
    @     0x556f5e673685  main
    @     0x7f8c1a7f6083  __libc_start_main
    @     0x556f5e679fa4  (unknown)
colmap.sh: line 20: 143810 Aborted                 (core dumped) colmap point_triangulator --database_path $workdir/colmap/database.db --image_path $workdir/colmap/images --input_path $workdir/colmap/sparse_custom --output_path $workdir/colmap/sparse/0 --clear_points 1
I20240610 15:53:01.355672 140201049677824 misc.cc:198] 
==============================================================================
Reading reconstruction
==============================================================================
F20240610 15:53:01.355728 140201049677824 reconstruction.cc:683] cameras, images, points3D files do not exist at /home/zomhussa/4dgs/SpacetimeGaussians/data/ringstable/colmap/sparse/0
    @     0x7f832150d78a  google::LogMessage::Fail()
    @     0x7f832150ef45  google::LogMessageFatal::~LogMessageFatal()
    @     0x55ff55b88d7c  colmap::Reconstruction::Read(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @     0x55ff55a8462f  colmap::RunImageUndistorter(int, char**)
    @     0x55ff55a44685  main
    @     0x7f831fb7e083  __libc_start_main
    @     0x55ff55a4afa4  (unknown)
colmap.sh: line 23: 143836 Aborted                 (core dumped) colmap image_undistorter --image_path $workdir/colmap/images --input_path $workdir/colmap/sparse/0 --output_path $workdir/colmap/dense/workspace
I20240610 15:53:01.478018 140714622971904 patch_match.cc:231] Reading workspace...
F20240610 15:53:01.478185 140714622971904 reconstruction.cc:683] cameras, images, points3D files do not exist at /home/zomhussa/4dgs/SpacetimeGaussians/data/ringstable/colmap/dense/workspace/sparse
    @     0x7ffb5830378a  google::LogMessage::Fail()
    @     0x7ffb58304f45  google::LogMessageFatal::~LogMessageFatal()
    @     0x561a79090d7c  colmap::Reconstruction::Read(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @     0x561a793c1461  colmap::mvs::Model::ReadFromCOLMAP(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_tr�?
    @     0x561a793c3af3  colmap::mvs::Model::Read(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @     0x561a793c78b2  colmap::mvs::Workspace::Workspace(colmap::mvs::Workspace::Options const&)
    @     0x561a793ccaec  colmap::mvs::CachedWorkspace::CachedWorkspace(colmap::mvs::Workspace::Options const&)
    @     0x561a790d6e74  colmap::mvs::PatchMatchController::ReadWorkspace()
    @     0x561a790e11ed  colmap::mvs::PatchMatchController::Run()
    @     0x561a790cd614  colmap::Thread::RunFunc()
    @     0x7ffb56d7fe95  execute_native_thread_routine
    @     0x7ffb582a9609  start_thread
    @     0x7ffb56a6f353  clone
colmap.sh: line 24: 143861 Aborted                 (core dumped) colmap patch_match_stereo --workspace_path $workdir/colmap/dense/workspace
I20240610 15:53:01.602423 139881016057856 misc.cc:205] 
StereoFusion::Options
---------------------
I20240610 15:53:01.602461 139881016057856 fusion.cc:87] mask_path: 
I20240610 15:53:01.602471 139881016057856 fusion.cc:88] max_image_size: -1
I20240610 15:53:01.602490 139881016057856 fusion.cc:89] min_num_pixels: 5
I20240610 15:53:01.602496 139881016057856 fusion.cc:90] max_num_pixels: 10000
I20240610 15:53:01.602502 139881016057856 fusion.cc:91] max_traversal_depth: 100
I20240610 15:53:01.602508 139881016057856 fusion.cc:92] max_reproj_error: 2
I20240610 15:53:01.602520 139881016057856 fusion.cc:93] max_depth_error: 0.01
I20240610 15:53:01.602526 139881016057856 fusion.cc:94] max_normal_error: 10
I20240610 15:53:01.602533 139881016057856 fusion.cc:95] check_num_images: 50
I20240610 15:53:01.602541 139881016057856 fusion.cc:96] use_cache: 0
I20240610 15:53:01.602547 139881016057856 fusion.cc:97] cache_size: 32
I20240610 15:53:01.602553 139881016057856 fusion.cc:100] bbox_min: -3.40282e+38 -3.40282e+38 -3.40282e+38
I20240610 15:53:01.602564 139881016057856 fusion.cc:101] bbox_max: 3.40282e+38 3.40282e+38 3.40282e+38
I20240610 15:53:01.602575 139881016057856 fusion.cc:148] Reading workspace...
F20240610 15:53:01.602611 139881016057856 misc.cc:282] Check failed: file.is_open() /home/zomhussa/4dgs/SpacetimeGaussians/data/ringstable/colmap/dense/workspace/stereo/fusion.cfg
    @     0x7f395158278a  google::LogMessage::Fail()
    @     0x7f3951583f45  google::LogMessageFatal::~LogMessageFatal()
    @     0x55f748fda7ed  colmap::ReadTextFileLines(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @     0x55f74928fa28  colmap::mvs::StereoFusion::Run()
    @     0x55f748fe3614  colmap::Thread::RunFunc()
    @     0x7f394fffee95  execute_native_thread_routine
    @     0x7f3951528609  start_thread
    @     0x7f394fcee353  clone
colmap.sh: line 25: 143887 Aborted                 (core dumped) colmap stereo_fusion --workspace_path $workdir/colmap/dense/workspace --output_path $workdir/colmap/dense/workspace/fused.ply

Did someone experience this issue when trying to work with stereo data ? If someone has any idea about what might be going wrong then please help me with it. Thanks

guanjunwu commented 5 months ago

Hi, sorry for late reply. It seems like you just use 2 fixed pose camera to generate point clouds? I think colmap can work if we have more than 10 images with there's poses...

lolwarmaze commented 5 months ago

Yes, I am only using a stereo camera setup with two fixed cameras with known poses. However, since I can also obtain depth images from my camera, I am now using depth info to generate an initial point cloud because COLMAP seems to be not working with only two known fixed camera poses setup.

azzarelli commented 3 months ago

I know I'm a bit late to this, but you could always try generating it with instant splat? I hear its good with sparse image sets.