/darknet_ros/detection_image only viewable if BOTH enable_opencv=true and Xserver is running

[tsender]

I am quite new to machine learning and using YOLO. I am a member on my universitiy's underwater robotics team and we are using a Nvidia Jetson TX1 and two Point Grey Blackfly cameras for object recognition with YOLO. Depending on the scenario as described below, either the /darknet_ros/detection_image is viewable (I can see the detected objects) or it is not viewable (all that is published is pure static). When looking through the code, it does not make sense as to why I am encountering this weird problem.

Our Jetson is running Ubuntu 16.04, my laptop also has 16.04, and both computers are running ROS Kinetic Kame. Here are the scenarios and the results from each. NOTE: in every scenario, both cameras are plugged into the Jetson, and both my computer and the Jetson are tethered to a router for ssh communication. And, in all scenarios, I am able to use ROS' rqt_image_view to view the raw camera output perfectly fine:

1. I login in to the Jetson user on the Jetson through its GUI and I run darknet and the cameras. The ros.yaml field "enable_opencv" is set to "true", and upon running it, the output displays "Xserver is running". This method works just fine. The OpenCV window with the detection image pops up and I CAN also view this image from my computer via rqt_image_view by telling my computer the ROS Master is the Jetson.
2. I login in to the Jetson user on the Jetson through its GUI and I run darknet and the cameras. The ros.yaml field "enable_opencv" is set to "false", and upon running it, the output displays "Xserver is running". In this scenario, darknet still runs just fine, and I can see the FPS and objects detected in the terminal screen. HOWEVER, the detection image is garbage - it is pure static if I try to view this image from my computer.
3. I ssh into the Jetson from a terminal on my computer and I run darknet and the cameras. The ros.yaml field "enable_opencv" is set to "true", and upon running it, the output displays "Xserver is not running". Same output as #2 (detection image is pure static).
4. I ssh into the Jetson from a terminal on my computer and I run darknet and the cameras. The ros.yaml field "enable_opencv" is set to "false", and upon running it, the output displays "Xserver is not running". Same output as #2 (detection image is pure static).
5. I ssh (using the argument -X) into the Jetson from a terminal on my computer and I run darknet and the cameras. The ros.yaml field "enable_opencv" is set to "true", and upon running it, the output displays "Xserver is running". Same output as #1 ("Xserver is running", detection image pops up in OpenCV window on my computer, and I CAN also view this image in rqt_image_view on my computer).
6. I ssh (using the argument -X) into the Jetson from a terminal on my computer and I run darknet and the cameras. The ros.yaml field "enable_opencv" is set to "false", and upon running it, the output displays "Xserver is running". Similar output as #2 (detection image is pure static, and I can NOT view this image in rqt_image_view on my computer).

Another member on our team also tried using his desktop to run darknet_ros and encountered the same issue when setting "enable_opencv" to "false" - the detection image was all static. It appears that the Xserver and the field "enable_opencv" have something to do with the detection image looking like a real image to us humans vs static. The weird thing is that none of the code related to the Xserver or the "enable_opencv" field have anything to do with the output of the detection image (from my understanding of the code in YoloObjectDetector.cpp)

In all scenarios when the detection image is static, these are the scenarios where we need to be able to view the detection image to properly verify what YOLO is outputting. This is because the Jetson will be launched via our main computer using a ROS launch machine tag.

Any assistance would be greatly appreciated. Thanks.

[tsender]

Never mind, I figured it out. It seems that the darknet_ros interface was configured in a weird way when it came to displaying the detected image. If darknet was set to display the detected image, it would also publish the detected image. But if darknet was NOT set to display the detected image (enable_opencv = false, or Xserver is not running), then the detected image would never be published. I don't think it should have been configured like that, since most people would want the ability to view the detected image in ROS' rqt_image_view or in image_view for debugging purposes.

So, I modified the darkness_ros interface slightly and also added a boolean field to the show_image_cv() function in darknet/src/image.c so the user can tell darknet if they want to display the detected image in an OpenCV pop-up window. It seemed rather weird that the show_image_cv() function would do all or nothing (publish and display the message, or neither). Now, regardless of there being an OpenCV window popping up, it ALWAYS publishes the detected image on ROS for the user to see.

[RhysMcK]

Hi @tsender , I am encountering this exact problem, are you able to share the specific changes you made to get this to work?

Cheers.

[Harsharma2308]

Hi, @tsender Could you explain the changes needed to be made? Thanks

[tsender]

I deeply apologize for responding so late - I have been quite busy with my school work. I am attaching my new YoloObjectDetector.cpp file (please replace the original with this one). If you search for the word "REVISED" you will find what was changed along with some comments. You will also need to replace the show_image_cv() code inside darkness_ros/darknet/src/image.c with the code in the second attached file.

NOTE: I had to save as a .txt to upload the files. You can read my second post when I closed this issue for more info.

YoloObjectDetector.txt show_image_cv.txt

[Harsharma2308]

I guess you must have had to change image.h as well at darkness_ros/darknet/src/image.h where the function show_image_cv() is originally declared. I changed inside image.h the definition void show_image_cv(image p, const char *name, IplImage *disp, bool displayDetectedImage); Also in function, void show_image in image.c, calls show_image_cv with three arguments , so I changed that to show_image_cv(copy, name, disp, 1);

After this I am getting error in YoloObjectDetector.cpp-

/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp: In member function ‘bool darknet_ros::YoloObjectDetector::readParameters()’:
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:59:58: error: ‘displayDetectedImage_’ was not declared in this scope
   nodeHandle_.param("image_view/display_detected_image", displayDetectedImage_, true);
                                                          ^~~~~~~~~~~~~~~~~~~~~
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp: In member function ‘void darknet_ros::YoloObjectDetector::init()’:
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:143:57: error: ‘cameraTopicName’ was not declared in this scope
   nodeHandle_.param("subscribers/camera_reading/topic", cameraTopicName,
                                                         ^~~~~~~~~~~~~~~
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:145:62: error: ‘cameraQueueSize’ was not declared in this scope
   nodeHandle_.param("subscribers/camera_reading/queue_size", cameraQueueSize, 1);
                                                              ^~~~~~~~~~~~~~~
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp: In member function ‘void* darknet_ros::YoloObjectDetector::fetchInThread()’:
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:412:35: error: ‘getIplImage’ was not declared in this scope
   IplImage* ROS_img = getIplImage();
                                   ^
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp: In member function ‘void* darknet_ros::YoloObjectDetector::displayInThread(void*)’:
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:429:61: error: ‘displayDetectedImage_’ was not declared in this scope
   show_image_cv(buff_[(buffIndex_ + 1)%3], "YOLO V3", ipl_, displayDetectedImage_);
                                                             ^~~~~~~~~~~~~~~~~~~~~
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp: In member function ‘void darknet_ros::YoloObjectDetector::yolo()’:
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:511:35: error: ‘getIplImage’ was not declared in this scope
   IplImage* ROS_img = getIplImage();
                                   ^
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:522:23: error: ‘displayDetectedImage_’ was not declared in this scope
   if (!demoPrefix_ && displayDetectedImage_) { // Use displayDetectedImage_
                       ^~~~~~~~~~~~~~~~~~~~~
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp: At global scope:
/home/harsh/catkin_ws/src/darknet_ros/darknet_ros/src/YoloObjectDetector.cpp:559:43: error: no ‘IplImage* darknet_ros::YoloObjectDetector::getIplImage()’ member function declared in class ‘darknet_ros::YoloObjectDetector’
 IplImage* YoloObjectDetector::getIplImage()
                                           ^
darknet_ros/darknet_ros/CMakeFiles/darknet_ros_lib.dir/build.make:1716: recipe for target 'darknet_ros/darknet_ros/CMakeFiles/darknet_ros_lib.dir/src/YoloObjectDetector.cpp.o' failed
make[2]: *** [darknet_ros/darknet_ros/CMakeFiles/darknet_ros_lib.dir/src/YoloObjectDetector.cpp.o] Error 1
CMakeFiles/Makefile2:725: recipe for target 'darknet_ros/darknet_ros/CMakeFiles/darknet_ros_lib.dir/all' failed
make[1]: *** [darknet_ros/darknet_ros/CMakeFiles/darknet_ros_lib.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2
Invoking "make -j12 -l12" failed

[Harsharma2308]

I did add bool displayDetectedImage_; in darknet_ros/darknet_ros/include/darknet_ros/YoloObjectDetector.hpp show_image_cv is called from other places as well and changed definition is causing error. Also some variables have been commented out and that seems to be causing error. Please check the file. Thanks though

[tsender]

Yes, I did not mention revising the header files (I should have, but I just assumed you'd be able to figure that one out). Keep in mind I used a different version of the code when I did this over the summer. It seems there have been code changes since then that I was not aware of. Regarding the cameraTopicName and cameraQueueSize errors you got, I actually placed those variables inside the .hpp file to make them class variables. As a note, I revised this code quite a bit to meet the needs for my system and did not remember everything that I changed. I did my best to restore it back to its original state, but seem to have missed a few things and apologize for any confusion.

I already mentioned what needs to be changed in my previous posts. You have all the information needed to figure out how to make your system compile. Because we are working on different versions of the darknet_ros code, you may just have to spend a bit more time to figure out how to make it work.

[marcelinomalmeidan]

I just had the same issue with a TX2. I guess it would happen in any embedded system if you ran through SSH. However, I do not recommend using the solution suggested by @tsender, since it changes the code within darknet as well (one would not be able to update darknet, or would have to fix image.c and image.h every time darknet is updated). Alternatively, I copied the code from within show_image_cv into YoloObjectDetector.cpp, avoiding the issues I mentioned above. If anyone is interested in the same fix, just download and compile my branch. Thank you very much @tsender for finding the issue, you made it easy to fix it on my side as well!

[zerosnsa]

我只是在 TX2 上遇到了同样的问题。我想如果你通过SSH运行，它会发生在任何嵌入式系统中。但是，我不建议使用 ，因为它也会更改其中的代码（无法更新暗网，或者必须修复并且每次都更新）。或者，我从内部将代码复制到 ，避免了我上面提到的问题。如果有人对相同的修复程序感兴趣，只需下载并编译我的分支即可。非常感谢您发现问题，您也让我轻松解决了这个问题！darknet``image.c``image.h``darknet``show_image_cv``YoloObjectDetector.cpp

If it is a ROS Noetic system, what other changes need to be made