Positioning not smooth

[duda1202]

Hi,

I would like to know if you had issues for tracking when a robot is following because, at least to me, the transform keeps jumping position from close to far as if there is no filter or something to smooth the coordinates to be consistent. Is this expected?

[zhanghanduo]

Hi. You are correct. Actually the tracking method this repo is using is a native one. From the time of last update you can see it is kind of out dated. I am now working on this project and according to the licence, I cannot make the recent repo public yet. Currently we are using a more advanced tracking method, refer to the KITTI tracking benchmark. Maybe we will consider make it public when our sponsor allows. Sorry about that.

[duda1202]

Hi,

Thank you for the quick reply. Do you know when it would be possible to make it public or when the sponsor might allow you to make it public?

[zhanghanduo]

I have no idea. Actually the tracking part was also not done by me. Recently I am really busy. Maybe I can send you the paper of that tracking method some time. It is actually not hard, you can read the abstract of the link I sent you. That might inspire you to some extent. Actually it is quite an engineering technique rather than theory innovation.

[duda1202]

Hi, that would be perfect, if you do not mind sending it to me. Thanks!

[astronaut71]

Hi. Im intersting only to get the 3D Bounding Box of the detected objects. Can I use this package for those purpose? I have stereo cameras and stereo images for input. Need 3D Bounding Box of the detected objects in order to perform Collision checking/detection.. Or what should I modify in order get the 3D Bounding Box of the detected objects?

[zhanghanduo]

@astronaut71 This package is not a repo designed and maintained well in a modular way, also the so-called 3D bounding box is not in the sense of academical way -- (frustum pointnet) this is the academical way. Our generated 3d boxes are just pseudo "3D" [https://gas they don't have orientation and the only difference they are different from 2D objects is that they have the actual realworld scale (width, height and distance to camera in meters).

Actually all the modules contrib somewhat to the final "3D bounding box". Tracking makes the output smoother, ostacle detection helps collision avoidance even 2d object detector cannot detect it (2d object detector uses YOLOv3 and can only differentiate some common classes like vehicles, person, etc). So this is more like a complete project for collision check.

[astronaut71]

ok. does https://github.com/charlesq34/frustum-pointnets works with YOLOv3 or Retinanet detector? And can be that bounding box used in ROS?

[zhanghanduo]

No. It uses its own end-to-end detector, input 3D laser scan, output direct 3d bounding boxes. I was just taking that as an example. YOLOv3 or retinanet is just 2D detector, which is totally different with 3D detector.

[astronaut71]

Ok. Do you know any 3D Detectors or how I can use YOLO or Retinanet or any other Deep Learning CNN Detector to get the 3D Bounding Box of the detected objects?

[zhanghanduo]

Firstly I need to know what is the use of 3D detection. If you only care about collision detection, my repository already works as it provides the detailed 3d size and position of all obstacles. If you care about the individual orientation, moving direction and even trajectory of each object, then you have to use some 3D detectors. Secondly, if you really want that end-to-end 3D detector, you have to know that currently this technique is more in academical stage rather than industrial because it's still not very material. Most 3d detectors are based on point cloud input like this: https://github.com/AI-liu/Complex-YOLO and the frustum-ppointnets. However, https://github.com/HKUST-Aerial-Robotics/Stereo-RCNN is a good example for 3D detection using stereo camera with Pytorch, for your reference. It is built on the basis of faster rcnn but the network architecture was changed a lot. Also the speed is not so ideal for real-time use. If you want to use YOLO or retinanet for 3d bounding box, you can read some papers first, like complex-yolo, or cubeslam. I am not currently developing this kind of thing, so I am sorry I cannot provide more resources apart from this code.

[astronaut71]

Ok. Im doing a perception task for robotic tower crane system. So I have some trajectory to follow from point A to B already given from path planning guy. So i dont care much about that as its already given. So on the robot (tower crane) way from point A to B I have to detect objects or obstacles to check the collision and if any collision detected then have to tell to the path planning guy to locally planned the path. So for example objects Im detecting are workers, vehicles, blocks in the construction field. So I have to know the 3D Bounding Box of each individual detected objects. After that have to know their spacial correlation (so for example is the worker behind of front of the vehicle. Hope more clear now. Any help?

[zhanghanduo]

Then I don't suggest you to use deep learning based methods as they are essentially "classification", or at least you cannot fully rely on them. You don't care about their catagory but only the size and locations so you can "redesign" the path. For example, you have some weird obstacle within the construction area, which is not listed in the classification list, neither a man nor a vehicle, you will be in the risk of ignoring it. So you should focus on obstacle detection rather than object detection. There are some algos about it, like: https://github.com/tkwoo/StereoVisionforADAS, https://github.com/gishi523/multilayer-stixel-world.

[astronaut71]

no. I have many sensors so will not miss any objects. I do Sensor Fission so will not miss any. I have to do objects detection first to learn the scene and the environment. We are academic and read a lot of papers and research work regrading the problem and the application , so we are sure the approach is a good one. As I build the octomap of the environment and i have to know which space is oocupied and which is free. So thats why need the Deep Learning and objects detection and need the 3D Bounding Box. More clear now?

[astronaut71]

As said Im not doing path Planning and designing any path. I only care of Collision checking/detection but not collision avoidance. As input i got path and navigate trajectory from point A to B. So on that way I only care of obstacles or objects that might cause collision. Thats all

[zhanghanduo]

I don't know whether you understand what I mean. No matter avoidance or checking/detection, what I mean is that object detector cannot detect everything, only some common objects. So it has potential danger once some obstacles are not detected.

[astronaut71]

yes. I know but thats why I have LIDAR to detect other small or any other objects.

[zhanghanduo]

Since you have lidar and camera together you can directly use them. There are dozens of point cloud based 3D object detectos like I introduced. AVOID, frustum pointnet2, etc.

[astronaut71]

AVOID?

[zhanghanduo]

AVOD, sorry. https://github.com/kujason/avod

[astronaut71]

so these are end to end detectors?

[zhanghanduo]

Yes. You should search such information yourself. There are quite some I think.

[astronaut71]

I think your package is ok to try as I only have to check collision. I already have YOLO using this https://github.com/AlexeyAB/darknet. Im running on Ubunut16 and ROS kinetic. Can use your Bounding Box in ROS?

[astronaut71]

hi. Can you tell me please how to use your package with my own training data? I mean with my custom data set.

[zhanghanduo]

Sorry for the late reply. You can first read through the readme and find this package: https://github.com/zhanghanduo/obstacle_msgs. It is the ROS msg of 3d objects. The latest darknet code is not compatible with this package. Try to use git clone --recursive -b devel git@github.com:zhanghanduo/cubicle_detect.git.

[astronaut71]

I already clone and compile your package and https://github.com/zhanghanduo/obstacle_msgs. Build with catkin build -DCMAKE_BUILD_TYPE=Release . No errors only some warnings. Went through Readme but not sure about config file when Use your own detection objects. Pls can you help on that?

[zhanghanduo]

Sorry the code was not managed well and I haven't updated the code since last year. I will update the latest code tomorrow. (Today I have some work to do).

Before you see the latest code, you can first read the launch/long.launch file and edit.

1. uncomment the line of <include file="$(find undistort_images)/launch/stereo_long.launch" /> because it is for rectification. For your own stereo image, edit this 7 lines: param name="image_left_topic" value="/long/left/image_rect" param name="image_right_topic" value="/long/right/image_rect" param name="image_info_left" value="/long/left/camera_info" param name="image_info_right" value="/long/right/camera_info" param name="image_height" value="844" param name="image_width" value="1280" param name="scale" value="2" image_info is sensor_msgs/CameraInfo, scale=2 represents scale down half, scale = 1 means original image.

2. In darknet_ros/config/ros.yaml, "obstacle_boxes" means the object output topic, "object_detector" and "bounding_boxes" are not so useful, are deprecated in the latest code. "detection_image" is the 2D yolo detection result.

[astronaut71]

ok. Sure. Btw, I see you are in NTU. Im in NTU too . lol

[zhanghanduo]

ok. Sure. Btw, I see you are in NTU. Im in NTU too . lol

My email is zhanghanduo@gmail.com. We can discuss when we are not busy. :)

[astronaut71]

ok. Thanks

[duda1202]

Hey, are you updating the code with the new tracking?

[duda1202]

hey, sorry, just wanted to know if you will still upload it today.

[zhanghanduo]

Sorry for the late reply. I have updated the code into branch "devel". However, it does not contain the code of new tracking algorithm, yet it is still better than before. The reason I was lazy to update the code is that it is tightly associated with some other tasks and I don't want to rewrite the config and launch files again. So if you meet any strange things, just try to comment them and run again.

[duda1202]

Hi,

Sorry to keep bothering you. Ive been trying to use your package on my master thesis to fuse with a laser leg tracking and its due soon. I have been trying to get the tracking to work in the new version but it does not. First, which tracking are we suppose to use: tracking or the trackingF--? While the detection works, the tracking does not recognize anything in the image, even with the debug prints I can find out why. Would you know why it would not be working? Thank you.

[zhanghanduo]

Hi. I have several suggestions.

1. Firstly, you backup your current code and re-git-clone the package without recursive: git clone -b devel git@github.com:zhanghanduo/cubicle_detect.git. Then you enter sgm, mkdir build, cmake .., make to generate a .so library file for stereo matching.
2. Download yolov3-spp weights: https://pjreddie.com/media/files/yolov3-spp.weights into /darknet_ros/yolo_network_config/weights.
3. If catkin make no error, check your launch file "enable_stereo" true, enable_classification true or false up to your choice, and you have to give the correct image_left,right topic and image_info_left, right topic!
4. If still no tracking, please give me some useful outputs or your customized settings for my reference. Otherwise I cannot give any advice.

[duda1202]

I had done everything. I guess what was wrong is that I was trying to use the yolov3-tiny config with the weights instead of spp. With spp it works fine. Thanks!