fafancier
commented
4 years ago Hello,@patmarion #93
I figure out how to label my own object with Azure Kinect and output the 8 vertices of 3d bounding box with good accuracy.
1. Firstly
you need to use lcm logger record your camera's rgbd stream. you need to check the resolution of lcm-republisher. for me 1280 720 -> 640 480 and pre-calculate the intrinsics of the new image.
in lcm_republisher.cpp. crop and resize if you are not 640 * 480
else if (rgb_msg->encoding == sensor_msgs::image_encodings::BGRA8)
{
cv::cvtColor(rgb, rgb, CV_RGBA2BGR);
//center_crop here
cv::Rect rect(160, 0,960,720);
cv::Mat roi= rgb(rect);
cv::resize(roi,rgb,cv::Size(640,480));
}
else
{
ROS_ERROR("Unexpected image encoding %s in input RGB image, only bgr8 and rgb8 and rgba8 encodings are supported.", rgb_msg->encoding.c_str());
ros::shutdown();
}
//$ check depth encoding
if (depth_msg->encoding == sensor_msgs::image_encodings::TYPE_16UC1)
{
if (debug_print_statements_)
ROS_INFO("Depth image format is 16UC1");
}
else if (depth_msg->encoding == sensor_msgs::image_encodings::TYPE_32FC1)
{
if (debug_print_statements_)
ROS_INFO("Converting depth from 32FC1 to 16UC1");
//$ convert to 16UC1
depth.convertTo(depth, CV_16UC1, 1000.0);
//center_crop
cv::Rect rect(160, 0,960,720);
cv::Mat roi= depth(rect);
cv::resize(roi,depth,cv::Size(640,480));
}
else
{
ROS_ERROR("Unexpected image encoding %s in input depth image, only 16UC1 and 32FC1 encodings are supported.", depth_msg->encoding.c_str());
ros::shutdown();
}
publishLCM(timestamp, rgb, depth);
}2. Secondly
change the intrinsics in bot_frames.cfg ( I'm not sure for the necessary) like this. using the pre-calculate intrinsics
cameras {
OPENNI_FRAME_LEFT {
lcm_channel = "OPENNI_FRAME_LEFT";
coord_frame = "OPENNI_FRAME_LEFT";
intrinsic_cal {
width = 640;
height= 480;
# width = 1280;
# height= 720;
distortion_model = "plumb-bob";
distortion_k = [0,0,0];
distortion_p = [0,0];
# pinhole = [ 528.0, 528.0, 0, 320.0, 240.0 ]; # fx fy skew cx cy
pinhole = [ 406.340666, 406.388667,0,321.667,242.333 ]; # fx fy skew cx cy
# example calibrated values
# pinhole = [ 528.01442863461716, 528.01442863461716, 0, 321.06398107, 242.97676897 ];
}
}
}3.Thirdly
you need to write a camera.cfg using your new intrinsics. make sure elasticfuion run with the following param -cal ***/camera.cfg
# call ElasticFusion
cameraConfig_arg = ""
if os.path.isfile(path+"/camera.cfg"):
cameraConfig_arg += " -cal "+ path+"/camera.cfg"
print 'cameraConfig_arg', cameraConfig_arg
os.system(path_to_ElasticFusion_executable + " -l ./" + lcmlog_filename + cameraConfig_arg)4. Fourthly
just run the pipeline as the author's routine.
if you want to label the 3d bounding box like me like the following picture.
you may need to calculate the projection 2d point as the green point.
In rendertrainingimages.py implement the projection operation.
def getProjectionPixel(self,verterx3d):
vertex3dList=[]
for i in range(verterx3d.GetNumberOfPoints()):
point = verterx3d.GetPoint(i)
vertex3dList.append(point)
vertex3dArray=np.array(vertex3dList)
vertex3dArray=vertex3dArray.transpose()
projectPixels=np.matmul(self.projectMatrix,vertex3dArray)
projectPixels=projectPixels/projectPixels[-1,:]
return projectPixels[:-1,:]
def setProjectMatrix():
principalX = 321.543
principalY = 242.333
fx=406.340666
fy=406.3886
return (np.array([[fx,0,principalX],[0,fy,principalY],[0,0,1]]))
akeaveny
anthony-simetri
If yes, how so?
Thank you.