Apply the best paper approaches to our datasets

[gkiavash]

Select some of the methods you have studied and try to apply them to our datasets (use the provided open source code from the papers' repos)

• [x] run pixel perfect + colmap on linux using GPU
• [x] learn their code, pycolmap
• [x] Create a point cloud from a cornflakes box and refine it with pixel-perfect.
• [ ] Then evaluate if it is actually refining

[gkiavash]

[2022/10/27 14:04:48 pixsfm INFO] Loaded dense extractor with configuration:
{'cache_format': 'chunked',
 'device': 'auto',
 'dtype': 'half',
 'fast_image_load': False,
 'l2_normalize': True,
 'load_cache_on_init': False,
 'max_edge': 1600,
 'model': {'name': 'dsift'},
 'overwrite_cache': True,
 'patch_size': 8,
 'pyr_scales': [1.0],
 'resize': 'LANCZOS',
 'sparse': True,
 'use_cache': True}
[2022/10/27 14:04:48 pixsfm INFO] Building matching graph...
[2022/10/27 14:04:48 pixsfm INFO] Extracting dense features...
100%|██████████| 7/7 [00:05<00:00,  1.22it/s]
[2022/10/27 14:05:38 pixsfm INFO] Loading featuremaps from H5 File.
100%[████████████████████] 7/7 [00:00, 170.731it/s]
[2022/10/27 14:05:38 pixsfm INFO] Computing tracks...
[2022/10/27 14:05:38 pixsfm INFO] # graph nodes: 4481
[2022/10/27 14:05:38 pixsfm INFO] # graph edges: 3803
[2022/10/27 14:05:38 pixsfm INFO] # tracks: 1711
[2022/10/27 14:05:38 pixsfm INFO] Start feature-metric keypoint adjustment.
[2022/10/27 14:05:38 pixsfm INFO] Start topological-reference keypoint adjustment.
100%[████████████████████] 4481/4481 [00:00, 8600.76it/s]
[2022/10/27 14:05:39 pixsfm INFO] KA Time: 0.521975s, cost change: 0.0174412 --> 0.0162724

==============================================================================
Custom feature matching
==============================================================================

Matching block [1/1] in 0.000s
Elapsed time: 0.000 [minutes]

==============================================================================
Loading database
==============================================================================

Loading cameras... 1 in 0.000s
Loading matches... 21 in 0.000s
Loading images... 7 in 0.001s (connected 7)
Building correspondence graph... in 0.001s (ignored 0)

Elapsed time: 0.000 [minutes]

==============================================================================
Triangulating image #1 (0)
==============================================================================

  => Image sees 0 / 775 points
  => Triangulated 431 points

==============================================================================
Triangulating image #2 (1)
==============================================================================

  => Image sees 151 / 812 points
  => Triangulated 367 points

==============================================================================
Triangulating image #3 (2)
==============================================================================

  => Image sees 396 / 833 points
  => Triangulated 119 points

==============================================================================
Triangulating image #4 (3)
==============================================================================

  => Image sees 329 / 1087 points
  => Triangulated 130 points

==============================================================================
Triangulating image #5 (4)
==============================================================================

  => Image sees 296 / 815 points
  => Triangulated 109 points

==============================================================================
Triangulating image #6 (5)
==============================================================================

  => Image sees 315 / 589 points
  => Triangulated 67 points

==============================================================================
Triangulating image #7 (6)
==============================================================================

  => Image sees 524 / 1024 points
  => Triangulated 73 points

==============================================================================
Retriangulation
==============================================================================

  => Completed observations: 24
  => Merged observations: 116

==============================================================================
Bundle adjustment
==============================================================================

iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  2.609484e+04    0.00e+00    4.89e+03   0.00e+00   0.00e+00  1.00e+04        0    5.45e-03    1.42e-02
   1  1.594511e+04    1.01e+04    2.77e+02   1.70e+01   9.92e-01  3.00e+04        0    3.79e-03    1.81e-02
   2  1.586427e+04    8.08e+01    5.97e+00   2.14e+00   9.98e-01  9.00e+04        0    2.27e-03    2.03e-02
   3  1.586424e+04    3.06e-02    2.48e-02   1.09e-01   9.97e-01  2.70e+05        0    2.26e-03    2.26e-02

Bundle adjustment report
------------------------
    Residuals : 6700
   Parameters : 2688
   Iterations : 4
         Time : 0.022655 [s]
 Initial cost : 1.97351 [px]
   Final cost : 1.53876 [px]
  Termination : Convergence

  => Completed observations: 8
  => Merged observations: 17
  => Filtered observations: 345
  => Changed observations: 0.110448

==============================================================================
Bundle adjustment
==============================================================================

iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  2.665297e+03    0.00e+00    3.44e+02   0.00e+00   0.00e+00  1.00e+04        0    1.82e-03    2.82e-03
   1  2.321811e+03    3.43e+02    8.27e+00   4.64e+00   1.00e+00  3.00e+04        0    4.21e-03    7.03e-03
   2  2.321597e+03    2.14e-01    1.91e-01   6.91e-01   1.00e+00  9.00e+04        0    3.26e-03    1.03e-02

Bundle adjustment report
------------------------
    Residuals : 5992
   Parameters : 2454
   Iterations : 3
         Time : 0.0103528 [s]
 Initial cost : 0.66694 [px]
   Final cost : 0.622454 [px]
  Termination : Convergence

  => Completed observations: 13
  => Merged observations: 16
  => Filtered observations: 0
  => Changed observations: 0.009680

==============================================================================
Bundle adjustment
==============================================================================

iter      cost      cost_change  |gradient|   |step|    tr_ratio  tr_radius  ls_iter  iter_time  total_time
   0  2.374993e+03    0.00e+00    2.82e+02   0.00e+00   0.00e+00  1.00e+04        0    3.40e-03    4.50e-03
   1  2.355700e+03    1.93e+01    5.20e-01   5.48e-01   1.00e+00  3.00e+04        0    4.26e-03    8.77e-03

Bundle adjustment report
------------------------
    Residuals : 6018
   Parameters : 2448
   Iterations : 2
         Time : 0.00882506 [s]
 Initial cost : 0.62821 [px]
   Final cost : 0.625654 [px]
  Termination : Convergence

  => Completed observations: 0
  => Merged observations: 0
  => Filtered observations: 0
  => Changed observations: 0.000000

==============================================================================
Extracting colors
==============================================================================

[2022/10/27 14:05:40 pixsfm INFO] Extracting dense features...
100%|██████████| 7/7 [00:01<00:00,  5.07it/s]
100%[████████████████████] 7/7 [00:00, 3500.00it/s]
[2022/10/27 14:05:42 pixsfm INFO] Loading featuremaps from H5 File.
[2022/10/27 14:05:42 pixsfm INFO] Extracting references and costmaps.
100%[████████████████████] 816/816 [00:00, 3867.29it/s]
  0%[                    ]   0/101 [00:00, -nanit/s][2022/10/27 14:05:42 pixsfm INFO] Start costmap bundle adjustment.
100%[████████████████████] 101/101 [00:01, 64.4132it/s]
[2022/10/27 14:05:43 pixsfm INFO] BA Time: 1.56841s, cost change: 0.0442656 --> 0.0347051
done

[gkiavash]

The comment above is the log of applying pixelsfm on my own room dataset which contains only 7 images. My room's point cloud in the previous task: https://github.com/gkiavash/Master-Thesis-Structure-from-Motion/issues/1#issuecomment-1266985779

Left ply: refined, Right ply: initial

One of the problems is the limitation of my GPU. I had to resize the images. Also, the size of the patches used for keypoints' descriptor is less than its default

[gkiavash]

COLMAP model statistics:

initial:

refined:

[gkiavash]

Images are taken from a cereal box on a table: Right image: colmap Left image refined by pixel perfect