search

versatran01 / dsol

DSOL: Direct Sparse Odometry Lite
Apache License 2.0
372 stars 64 forks source link

DSOL running slower than expected #4

Closed shreyknsl closed 2 years ago

shreyknsl commented 2 years ago

Hi, thanks for open-sourcing the code. We were trying to implement DSOL using both, the iros22 and main branch, and it seems to be running quite slow as it is taking multiple seconds to process each image frame from the tartan_air gascola dataset. We have tried different launch file parameters such as, changing the frequency and wait time, turning off visualization and logging as well. We have even tried it on different processors (going up to i9 11th gen.), but the time taken do not change drastically. I am attaching a summary snapshot of the parameters and output as tested on MacBook Air using the Parallels virtual machine with default configuration for your reference. The times increase as we move ahead in the dataset sequence.

---------Parameters-----------

SUMMARY
========

PARAMETERS
 * /dsol_data/adjust/affine: False
 * /dsol_data/adjust/c2: 4
 * /dsol_data/adjust/dof: 4
 * /dsol_data/adjust/grad_factor: 1.0
 * /dsol_data/adjust/max_iters: 6
 * /dsol_data/adjust/max_levels: 3
 * /dsol_data/adjust/max_outliers: 0
 * /dsol_data/adjust/min_depth: 0.1
 * /dsol_data/adjust/rel_change: 0.01
 * /dsol_data/adjust/stereo: True
 * /dsol_data/align/affine: False
 * /dsol_data/align/c2: 4
 * /dsol_data/align/dof: 4
 * /dsol_data/align/grad_factor: 1.0
 * /dsol_data/align/max_iters: 6
 * /dsol_data/align/max_levels: 0
 * /dsol_data/align/max_outliers: 1
 * /dsol_data/align/min_depth: 0.1
 * /dsol_data/align/rel_change: 0.01
 * /dsol_data/align/stereo: False
 * /dsol_data/cloud_max_depth: 50.0
 * /dsol_data/data_dir: /media/psf/robo/c...
 * /dsol_data/data_max_depth: 100.0
 * /dsol_data/end: 0
 * /dsol_data/freq: 10.0
 * /dsol_data/log: 5
 * /dsol_data/motion_alpha: 0.5
 * /dsol_data/odom/init_align: True
 * /dsol_data/odom/init_depth: False
 * /dsol_data/odom/init_stereo: True
 * /dsol_data/odom/marg: False
 * /dsol_data/odom/min_track_per_kf: 10
 * /dsol_data/odom/min_track_ratio: 0.35
 * /dsol_data/odom/num_kfs: 4
 * /dsol_data/odom/num_levels: 4
 * /dsol_data/odom/reinit: True
 * /dsol_data/odom/vis_min_depth: 0.5
 * /dsol_data/reverse: False
 * /dsol_data/save: 
 * /dsol_data/select/cell_size: 16
 * /dsol_data/select/max_grad: 64
 * /dsol_data/select/max_ratio: 0.6
 * /dsol_data/select/min_grad: 4
 * /dsol_data/select/min_ratio: 0.4
 * /dsol_data/select/nms_size: 1
 * /dsol_data/select/reselect: True
 * /dsol_data/select/sel_level: 1
 * /dsol_data/start: 0
 * /dsol_data/stereo/half_cols: 3
 * /dsol_data/stereo/half_rows: 2
 * /dsol_data/stereo/match_level: 3
 * /dsol_data/stereo/min_depth: 0.5
 * /dsol_data/stereo/min_zncc: 0.8
 * /dsol_data/stereo/refine_size: 1
 * /dsol_data/tbb: 1
 * /dsol_data/vis: False
 * /dsol_data/wait_ms: 8
 * /rosdistro: noetic
 * /rosversion: 1.15.14

NODES
  /
    dsol_data (dsol/sv_dsol_node_data)

auto-starting new master
process[master]: started with pid [944472]
ROS_MASTER_URI=http://localhost:11311/

setting /run_id to 1955a36a-03d2-11ed-bda9-0b7ac2dd2c25
process[rosout-1]: started with pid [944483]
started core service [/rosout]
process[dsol_data-2]: started with pid [944490]
[ INFO] [1657843643.404443861]: tartan_air(dir=/media/psf/robo/catkin_ws/src/dsol/tartan_air/P001, size=382, dtypes=[image, depth, intrin, pose])
[ INFO] [1657843643.405418708]: Data range: [0, 382)
[ INFO] [1657843643.405431500]: Reverse: false
[ INFO] [1657843643.405734408]: Freq: 10
[ INFO] [1657843643.405745949]: Max depth: 100
[ INFO] [1657843643.416015278]: DirectOdometry(
    OdomCfg(tbb=1, log=5, vis=false, marg=false, num_kfs=4, num_levels=4, min_track_ratio=0.35, vis_min_depth=0.5, reinit=true, init_depth=false, init_stereo=true, init_align=true)
    Camera(w=0, h=0, fxycxy=[0 0 0 0], b=0, scale=1)
    PixelSelector(cfg=SelectCfg(sel_level=1, cell_size=16, min_grad=4, max_grad=64, nms_size=1, min_ratio=0.4, max_ratio=0.6, reselect=true), grid_border=1)
    StereoMatcher(cfg=StereoCfg(half_rows=2, half_cols=3, match_level=3, refine_size=1, min_depth=0.5, min_zncc=0.8))
    FrameAligner(cfg=DirectOptmCfg(init_level=0, max_iters=6, max_xs=0.1), DirectCostCfg(affine=false, stereo=false, c2=4, dof=4, max_outliers=1, grad_factor=1, min_depth=0.1))
    BundleAdjuster(cfg=DirectOptmCfg(init_level=0, max_iters=6, max_xs=0.1), DirectCostCfg(affine=false, stereo=true, c2=4, dof=4, max_outliers=0, grad_factor=1, min_depth=0.1))
    KeyframeWindow(size=0/4))
[ INFO] [1657843643.416354519]: wait_ms: 8
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20220714 17:07:23.416422 944490 ocv.cpp:122] Press 's' to step, 'r' to play, 'p' to pause, 'space' to toggle play/pause, 'esc' to quit
[ INFO] [1657843643.480953533]: motion_alpha: 0.5
[ INFO] [1657843643.481054614]: === 0 ===
[ INFO] [1657843643.503414512]: Camera(w=640, h=480, fxycxy=[320 320 320 240], b=0.25, scale=1)
[ INFO] [1657843643.524947808]: T_c0_w:
    0.984318    -0.176402 -2.77556e-17      49.6318
-2.77556e-17 -4.44089e-16            1      3.05804
   -0.176402    -0.984318 -4.44089e-16      14.8371
           0            0            0            1

---------Output-----------

I20220714 16:41:46.706459 939354 odom.cpp:554] Manager: dsol
[All_Keyframe    ] n: 2        | last: 2.117205403s   | mean: 1.0637994855s  | min: 10.393568ms    | max: 2.117205403s   | sum: 2.127598971s   |
[All_Tracking    ] n: 16       | last: 353.075553ms   | mean: 455.432054ms   | min: 175.646us      | max: 608.229543ms   | sum: 7.286912867s   |
[K0_SetOccMask   ] n: 2        | last: 66.784us       | mean: 61.5345us      | min: 56.285us       | max: 66.784us       | sum: 123.069us      |
[K1_SelectPixels ] n: 2        | last: 987.885us      | mean: 1.034462ms     | min: 987.885us      | max: 1.081039ms     | sum: 2.068924ms     |
[K2_InitPoints   ] n: 2        | last: 2.539954ms     | mean: 4.32093ms      | min: 2.539954ms     | max: 6.101906ms     | sum: 8.64186ms      |
[K3_InitDepths   ] n: 2        | last: 102.405us      | mean: 262.8235us     | min: 102.405us      | max: 423.242us      | sum: 525.647us      |
[K5_BundleAdjust ] n: 1        | last: 2.111605888s   | mean: 2.111605888s   | min: 2.111605888s   | max: 2.111605888s   | sum: 2.111605888s   |
[K5_InitAlign    ] n: 2        | last: 16.623us       | mean: 18.9565us      | min: 16.623us       | max: 21.29us        | sum: 37.913us       |
[K6_InitPatches  ] n: 2        | last: 1.885864ms     | mean: 2.297835ms     | min: 1.885864ms     | max: 2.709806ms     | sum: 4.59567ms      |
[P0_ConvertGray  ] n: 32       | last: 104.821us      | mean: 121.526us      | min: 104.404us      | max: 331.253us      | sum: 3.888838ms     |
[T0_MakePyramid  ] n: 32       | last: 137.484us      | mean: 157.364us      | min: 129.651us      | max: 285.216us      | sum: 5.035655ms     |
[T1_TrackFrame   ] n: 15       | last: 352.938069ms   | mean: 485.6488415ms  | min: 345.37753ms    | max: 608.0996ms     | sum: 7.284732625s   |
[ INFO] [1657842106.706868622]: OdomStatus(add_kf=false, remove_kf=false, total_kfs=2)
[ INFO] [1657842106.707184794]: === 16 ===
I20220714 16:41:46.745659 939354 odom.cpp:286] Track frame with window size: 2
I20220714 16:41:47.085711 939354 odom.cpp:295] DirectStatus(num_kfs=2, num_points=819, num_levels=4, num_iters=0/17, num_costs=598, cost=1.58e+05, converged=true)
I20220714 16:41:47.085772 939354 odom.cpp:298] 0: KeyframeStatus(pixels=1055, depths=1040, patches=4184 | info_bad= 15, info_uncert=660, info_ok=256, info_max=124)
I20220714 16:41:47.085788 939354 odom.cpp:298] 1: KeyframeStatus(pixels=1049, depths=1041, patches=4170 | info_bad= 10, info_uncert=600, info_ok=332, info_max=107)
I20220714 16:41:47.085803 939354 odom.cpp:358] win 2, tracked 394, depths 1040.5, ratio: 37.87%
[ INFO] [1657842107.085848365]: OdomStatus(add_kf=false, remove_kf=false, total_kfs=2)
[ INFO] [1657842107.086061798]: === 17 ===
I20220714 16:41:47.127633 939354 odom.cpp:286] Track frame with window size: 2
I20220714 16:41:47.495066 939354 odom.cpp:295] DirectStatus(num_kfs=2, num_points=819, num_levels=4, num_iters=0/19, num_costs=576, cost=1.81e+05, converged=true)
I20220714 16:41:47.495116 939354 odom.cpp:298] 0: KeyframeStatus(pixels=1055, depths=1040, patches=4184 | info_bad= 15, info_uncert=660, info_ok=256, info_max=124)
I20220714 16:41:47.495131 939354 odom.cpp:298] 1: KeyframeStatus(pixels=1049, depths=1041, patches=4170 | info_bad= 10, info_uncert=600, info_ok=332, info_max=107)
I20220714 16:41:47.495144 939354 odom.cpp:358] win 2, tracked 366, depths 1040.5, ratio: 35.18%
[ INFO] [1657842107.495186976]: OdomStatus(add_kf=false, remove_kf=false, total_kfs=2)
[ INFO] [1657842107.495372705]: === 18 ===

Are we missing any tunable configuration parameters? Any help would be appreciated.

Thank you

iandouglas96 commented 2 years ago

Are you building with Release? Try ‘catkin config —cmake-args -DCMAKE_BUILD_TYPE=Release’

On Jul 14, 2022, at 8:09 PM, shreyknsl @.***> wrote:

 Hi, thanks for open-sourcing the code. We were trying to implement DSOL using both, the iros22 and main branch, and it seems to be running quite slow as it is taking multiple seconds to process each image frame from the tartan_air gascola dataset. We have tried different launch file parameters such as, changing the frequency and wait time, turning off visualization and logging as well. We have even tried it on different processors (going up to i9 11th gen.), but the time taken do not change drastically. I am attaching a summary snapshot of the parameters and output as tested on MacBook Air using the Parallels virtual machine with default configuration for your reference. The times increase as we move ahead in the dataset sequence.

---------Parameters-----------

SUMMARY

PARAMETERS

  • /dsol_data/adjust/affine: False
  • /dsol_data/adjust/c2: 4
  • /dsol_data/adjust/dof: 4
  • /dsol_data/adjust/grad_factor: 1.0
  • /dsol_data/adjust/max_iters: 6
  • /dsol_data/adjust/max_levels: 3
  • /dsol_data/adjust/max_outliers: 0
  • /dsol_data/adjust/min_depth: 0.1
  • /dsol_data/adjust/rel_change: 0.01
  • /dsol_data/adjust/stereo: True
  • /dsol_data/align/affine: False
  • /dsol_data/align/c2: 4
  • /dsol_data/align/dof: 4
  • /dsol_data/align/grad_factor: 1.0
  • /dsol_data/align/max_iters: 6
  • /dsol_data/align/max_levels: 0
  • /dsol_data/align/max_outliers: 1
  • /dsol_data/align/min_depth: 0.1
  • /dsol_data/align/rel_change: 0.01
  • /dsol_data/align/stereo: False
  • /dsol_data/cloud_max_depth: 50.0
  • /dsol_data/data_dir: /media/psf/robo/c...
  • /dsol_data/data_max_depth: 100.0
  • /dsol_data/end: 0
  • /dsol_data/freq: 10.0
  • /dsol_data/log: 5
  • /dsol_data/motion_alpha: 0.5
  • /dsol_data/odom/init_align: True
  • /dsol_data/odom/init_depth: False
  • /dsol_data/odom/init_stereo: True
  • /dsol_data/odom/marg: False
  • /dsol_data/odom/min_track_per_kf: 10
  • /dsol_data/odom/min_track_ratio: 0.35
  • /dsol_data/odom/num_kfs: 4
  • /dsol_data/odom/num_levels: 4
  • /dsol_data/odom/reinit: True
  • /dsol_data/odom/vis_min_depth: 0.5
  • /dsol_data/reverse: False
  • /dsol_data/save:
  • /dsol_data/select/cell_size: 16
  • /dsol_data/select/max_grad: 64
  • /dsol_data/select/max_ratio: 0.6
  • /dsol_data/select/min_grad: 4
  • /dsol_data/select/min_ratio: 0.4
  • /dsol_data/select/nms_size: 1
  • /dsol_data/select/reselect: True
  • /dsol_data/select/sel_level: 1
  • /dsol_data/start: 0
  • /dsol_data/stereo/half_cols: 3
  • /dsol_data/stereo/half_rows: 2
  • /dsol_data/stereo/match_level: 3
  • /dsol_data/stereo/min_depth: 0.5
  • /dsol_data/stereo/min_zncc: 0.8
  • /dsol_data/stereo/refine_size: 1
  • /dsol_data/tbb: 1
  • /dsol_data/vis: False
  • /dsol_data/wait_ms: 8
  • /rosdistro: noetic
  • /rosversion: 1.15.14

NODES / dsol_data (dsol/sv_dsol_node_data)

auto-starting new master process[master]: started with pid [944472] ROS_MASTER_URI=http://localhost:11311/

setting /run_id to 1955a36a-03d2-11ed-bda9-0b7ac2dd2c25 process[rosout-1]: started with pid [944483] started core service [/rosout] process[dsol_data-2]: started with pid [944490] [ INFO] [1657843643.404443861]: tartan_air(dir=/media/psf/robo/catkin_ws/src/dsol/tartan_air/P001, size=382, dtypes=[image, depth, intrin, pose]) [ INFO] [1657843643.405418708]: Data range: [0, 382) [ INFO] [1657843643.405431500]: Reverse: false [ INFO] [1657843643.405734408]: Freq: 10 [ INFO] [1657843643.405745949]: Max depth: 100

OdomCfg(tbb=1, log=5, vis=false, marg=false, num_kfs=4, num_levels=4, min_track_ratio=0.35, vis_min_depth=0.5, reinit=true, init_depth=false, init_stereo=true, init_align=true) Camera(w=0, h=0, fxycxy=[0 0 0 0], b=0, scale=1) PixelSelector(cfg=SelectCfg(sel_level=1, cell_size=16, min_grad=4, max_grad=64, nms_size=1, min_ratio=0.4, max_ratio=0.6, reselect=true), grid_border=1) StereoMatcher(cfg=StereoCfg(half_rows=2, half_cols=3, match_level=3, refine_size=1, min_depth=0.5, min_zncc=0.8)) FrameAligner(cfg=DirectOptmCfg(init_level=0, max_iters=6, max_xs=0.1), DirectCostCfg(affine=false, stereo=false, c2=4, dof=4, max_outliers=1, grad_factor=1, min_depth=0.1)) BundleAdjuster(cfg=DirectOptmCfg(init_level=0, max_iters=6, max_xs=0.1), DirectCostCfg(affine=false, stereo=true, c2=4, dof=4, max_outliers=0, grad_factor=1, min_depth=0.1)) KeyframeWindow(size=0/4)) [ INFO] [1657843643.416354519]: wait_ms: 8 WARNING: Logging before InitGoogleLogging() is written to STDERR I20220714 17:07:23.416422 944490 ocv.cpp:122] Press 's' to step, 'r' to play, 'p' to pause, 'space' to toggle play/pause, 'esc' to quit [ INFO] [1657843643.480953533]: motion_alpha: 0.5 [ INFO] [1657843643.481054614]: === 0 === [ INFO] [1657843643.503414512]: Camera(w=640, h=480, fxycxy=[320 320 320 240], b=0.25, scale=1)

0.984318    -0.176402 -2.77556e-17      49.6318

-2.77556e-17 -4.44089e-16 1 3.05804 -0.176402 -0.984318 -4.44089e-16 14.8371 0 0 0 1 ---------Output-----------

I20220714 16:41:46.706459 939354 odom.cpp:554] Manager: dsol [All_Keyframe ] n: 2 | last: 2.117205403s | mean: 1.0637994855s | min: 10.393568ms | max: 2.117205403s | sum: 2.127598971s | [All_Tracking ] n: 16 | last: 353.075553ms | mean: 455.432054ms | min: 175.646us | max: 608.229543ms | sum: 7.286912867s | [K0_SetOccMask ] n: 2 | last: 66.784us | mean: 61.5345us | min: 56.285us | max: 66.784us | sum: 123.069us | [K1_SelectPixels ] n: 2 | last: 987.885us | mean: 1.034462ms | min: 987.885us | max: 1.081039ms | sum: 2.068924ms | [K2_InitPoints ] n: 2 | last: 2.539954ms | mean: 4.32093ms | min: 2.539954ms | max: 6.101906ms | sum: 8.64186ms | [K3_InitDepths ] n: 2 | last: 102.405us | mean: 262.8235us | min: 102.405us | max: 423.242us | sum: 525.647us | [K5_BundleAdjust ] n: 1 | last: 2.111605888s | mean: 2.111605888s | min: 2.111605888s | max: 2.111605888s | sum: 2.111605888s | [K5_InitAlign ] n: 2 | last: 16.623us | mean: 18.9565us | min: 16.623us | max: 21.29us | sum: 37.913us | [K6_InitPatches ] n: 2 | last: 1.885864ms | mean: 2.297835ms | min: 1.885864ms | max: 2.709806ms | sum: 4.59567ms | [P0_ConvertGray ] n: 32 | last: 104.821us | mean: 121.526us | min: 104.404us | max: 331.253us | sum: 3.888838ms | [T0_MakePyramid ] n: 32 | last: 137.484us | mean: 157.364us | min: 129.651us | max: 285.216us | sum: 5.035655ms | [T1_TrackFrame ] n: 15 | last: 352.938069ms | mean: 485.6488415ms | min: 345.37753ms | max: 608.0996ms | sum: 7.284732625s | [ INFO] [1657842106.706868622]: OdomStatus(add_kf=false, remove_kf=false, total_kfs=2) [ INFO] [1657842106.707184794]: === 16 === I20220714 16:41:46.745659 939354 odom.cpp:286] Track frame with window size: 2 I20220714 16:41:47.085711 939354 odom.cpp:295] DirectStatus(num_kfs=2, num_points=819, num_levels=4, num_iters=0/17, num_costs=598, cost=1.58e+05, converged=true) I20220714 16:41:47.085772 939354 odom.cpp:298] 0: KeyframeStatus(pixels=1055, depths=1040, patches=4184 | info_bad= 15, info_uncert=660, info_ok=256, info_max=124) I20220714 16:41:47.085788 939354 odom.cpp:298] 1: KeyframeStatus(pixels=1049, depths=1041, patches=4170 | info_bad= 10, info_uncert=600, info_ok=332, info_max=107) I20220714 16:41:47.085803 939354 odom.cpp:358] win 2, tracked 394, depths 1040.5, ratio: 37.87% [ INFO] [1657842107.085848365]: OdomStatus(add_kf=false, remove_kf=false, total_kfs=2) [ INFO] [1657842107.086061798]: === 17 === I20220714 16:41:47.127633 939354 odom.cpp:286] Track frame with window size: 2 I20220714 16:41:47.495066 939354 odom.cpp:295] DirectStatus(num_kfs=2, num_points=819, num_levels=4, num_iters=0/19, num_costs=576, cost=1.81e+05, converged=true) I20220714 16:41:47.495116 939354 odom.cpp:298] 0: KeyframeStatus(pixels=1055, depths=1040, patches=4184 | info_bad= 15, info_uncert=660, info_ok=256, info_max=124) I20220714 16:41:47.495131 939354 odom.cpp:298] 1: KeyframeStatus(pixels=1049, depths=1041, patches=4170 | info_bad= 10, info_uncert=600, info_ok=332, info_max=107) I20220714 16:41:47.495144 939354 odom.cpp:358] win 2, tracked 366, depths 1040.5, ratio: 35.18% [ INFO] [1657842107.495186976]: OdomStatus(add_kf=false, remove_kf=false, total_kfs=2) [ INFO] [1657842107.495372705]: === 18 === Are we missing any tunable configuration parameters? Any help would be appreciated.

Thank you

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

shreyknsl commented 2 years ago

Thank you, it worked!

versatran01 commented 2 years ago

I also recommend setting the OpenCV num threads to 4~8 in the node. I noticed that depending on how you build your OpenCV, it may not be using tbb for its internal multithreading. Thus, some of the OpenCV function will try to use all the threads which lead to oversubscription. I will add it to the node's main function later.