Implementing Custom Feature-Descriptor to Replace ORB-Descriptor (with KLT: false)

[f5ces27er5c767s]

Hi @goldbattle thank you so much for your quick response in my last issue. I managed to resolve that issue but have run into another one, if you ever have the time :)

I am implementing my own custom feature-descriptor to replace the ORB-descriptor framework of OpenVINS but am running into the problem that my features and their descriptors are extracted, allocated to pts0 and desc0 in ov_core/src/track/TrackDescriptor.cpp, but the framework then just stops there at frame/imafe 1 not continuing after this first frame/image. Please ignore the debugs (using these to try and locate the problem) but I get this:

~/catkin_ws$ roslaunch ov_msckf_tcb subscribe-fpsp-irs.launch config_path:=/home/user/TCBVIO/rs_custom-fpsp-irs-july2-KLT-FALSE-AND-BIT-DESCRIPTOR/estimator_config.yaml
... logging to /home/user/.ros/log/1700ae80-46dc-11ef-b7d7-0013eff20e63/roslaunch-user-Legion-Pro-5-16IRX8-16330.log
Checking log directory for disk usage. This may take a while.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://192.168.0.XXX/

SUMMARY
========

CLEAR PARAMETERS
 * /ov_msckf_tcb/

PARAMETERS
 * /ov_msckf_tcb/config_path: /home/user/...
 * /ov_msckf_tcb/max_cameras: 1
 * /ov_msckf_tcb/record_timing_filepath: /tmp/traj_timing.txt
 * /ov_msckf_tcb/record_timing_information: False
 * /ov_msckf_tcb/use_stereo: True
 * /ov_msckf_tcb/verbosity: INFO
 * /rosdistro: melodic
 * /rosversion: 1.14.13

NODES
  /
    ov_msckf_tcb (ov_msckf_tcb/run_subscribe_msckf_tcb)

ROS_MASTER_URI=http://192.168.0.115:11311

process[ov_msckf_tcb-1]: started with pid [16385]
overriding node verbosity with value from ROS!
Setting printing level to: INFO
overriding node max_cameras with value from ROS!
overriding node max_cameras with value from ROS!
overriding node use_stereo with value from ROS!
parameter T_imu_cam not found, trying T_cam_imu instead (will return T_imu_cam still)!
overriding node record_timing_information with value from ROS!
overriding node record_timing_filepath with value from ROS!
overriding node use_stereo with value from ROS!
parameter T_imu_cam not found, trying T_cam_imu instead (will return T_imu_cam still)!
subscribing to IMU: /camera/imu
subscribing to cam (mono): /super_image_modified
Starting feed_new_camera
Number of images: 1
Calling feed_monocular with msg_id 0
Starting feed_monocular with msg_id: 0
Applying HISTOGRAM equalization
Image and mask processed
First frame or lost tracking, initializing descriptors
[DEBUG] Calling perform_detection_monocular
[DEBUG] perform_detection_monocular called
[DEBUG] processed_image size: [256 x 256]
[DEBUG] processed_image type: 0

...
233 1 252 127 255 15 0 0 233 1 252 127 255 15 0 0 233 1 252 127 255 15 0 0 233 1 252 127 255 15 0 0 
14 0 192 7 124 12 0 0 14 0 192 7 124 12 0 0 14 0 192 7 124 12 0 0 14 0 192 7 124 12 0 0 
14 0 224 15 252 15 0 0 14 0 224 15 252 15 0 0 14 0 224 15 252 15 0 0 14 0 224 15 252 15 0 0 
47 0 240 62 255 15 0 0 47 0 240 62 255 15 0 0 47 0 240 62 255 15 0 0 47 0 240 62 255 15 0 0 
3 4 248 63 252 9 0 0 3 4 248 63 252 9 0 0 3 4 248 63 252 9 0 0 3 4 248 63 252 9 0 0 
243 1 252 252 255 15 0 0 243 1 252 252 255 15 0 0 243 1 252 252 255 15 0 0 243 1 252 252 255 15 0 0 
[DEBUG] perform_detection_monocular completed
[DEBUG] Number of keypoints: 80
[DEBUG] Descriptor size: [32 x 80]
[DEBUG] Number of IDs: 80
[DEBUG] Image size: [256 x 256]
[DEBUG] Mask size: [256 x 256]
[DEBUG] Keypoints size: 80
[DEBUG] IDs size: 80
[DEBUG] Descriptors size: [32 x 80]
[TIME]: 0.0067 seconds total (150.4 hz, 171988274988.03 ms behind)

I can also attach my edited TrackDescriptor.cpp code as follows if you would like to see it, Thank you kindly :)

The size 32 sequence of numbers is just a debug print statement of a specific corner keypoint's descriptor.

Definitely apologize for all the questions but @goldbattle did you have any suggestions how to get the framework working after frame 1 for frame-to-frame feature tracking? What is the next steps after modifying perform_detection_monocular? :)

My idea was:

1. edit perform_detection_monocular() making sure has same array sizes and information as original ORB-descriptor outputs
2. as my custom descriptor is same format as ORB, I left robust_match() and its sub-functions alone for matching between frames.

Kindly, what do you think? :)

[goldbattle]

I am not sure what the issue is, but you can run with gdb and debug to figure out what it is doing. Probably stepping through the code to find where it is hanging would be a good start.