Need better documentation for running on physical turtlebot

[nburek]

Users that are trying to deploy and run the sample applications on physical turtlebots are having difficulty getting them actually running. The PersonDetection app seems to be one of the worst. I have included the troubleshooting steps provided to us by one user who managed to get the person detection application running on a turtlebot. We should use this input to develop new documentation on how to get this running.

Troubleshooting RoboMaker sample app: Navigation and Person Recognition

Environment

• Turtlebot 3 (Waffle Pi) with Raspberry Pi 3 B+, Raspbian OS, and ROS Kinetic full install.
• Following the instructions from https://docs.aws.amazon.com/robomaker/latest/dg/gs-deploy.html to build and bundle the “Navigation and Person Recognition” sample app in Cloud9. Note: I created a new Development Environment in Cloud9 today, and tried to re-downloaded, build, and bundle the sample app from scratch. Unfortunately, the build colcon build --build-base armhf_build --install-base armhf_install failed with the following messages:

[  8%] Performing build step for 'KVS_SDK_IMPORT'
cmake version 3.5.1 CMake suite maintained and supported by Kitware (kitware.com/cmake).
Checking log4cplus at /ws/armhf_build/kinesis_manager/external/kinesis-video-native-build/downloads/local/lib/liblog4cplus.dylib/.so
log4cplus lib not found. Installing
/usr/bin/xz: (stdin): File format not recognized
/bin/tar: Child returned status 1
/bin/tar: Error is not recoverable: exiting now
kvssdk/CMakeFiles/KVS_SDK_IMPORT.dir/build.make:113: recipe for target 'kvssdk/KVS_SDK_IMPORT-prefix/src/KVS_SDK_IMPORT-stamp/KVS_SDK_IMPORT-build' failed
make[2]: *** [kvssdk/KVS_SDK_IMPORT-prefix/src/KVS_SDK_IMPORT-stamp/KVS_SDK_IMPORT-build] Error 2
CMakeFiles/Makefile2:122: recipe for target 'kvssdk/CMakeFiles/KVS_SDK_IMPORT.dir/all' failed
make[1]: *** [kvssdk/CMakeFiles/KVS_SDK_IMPORT.dir/all] Error 2
Makefile:138: recipe for target 'all' failed
make: *** [all] Error 2
Failed   <<< kinesis_manager    [ Exited with code 2 ]

Therefore, I had to give up and use the bundle file created two weeks ago.

• Download the bundle to local PC, and copy it to Turtlebot.
• Untar the bundle file to /home/pi/person-detection/.

Attempt 1: Failed

According to the documentation at https://github.com/aws-robotics/aws-robomaker-sample-application-persondetection, run the following commands:

export BUNDLE_CURRENT_PREFIX=/home/pi/person-detection
source $BUNDLE_CURRENT_PREFIX/setup.sh
roslaunch person_detection_robot deploy_person_detection.launch

This consistently fails to bring up the raspicam_node, and displays the following error messages:

[raspicam_node-5] process has died [pid 24637, exit code -11, cmd /opt/ros/kinetic/lib/raspicam_node/raspicam_node __name:=raspicam_node __log:=/home/pi/.ros/log/31a66b7a-4576-11e9-9605-b827eb6ec255/raspicam_node-5.log].
log file: /home/pi/.ros/log/31a66b7a-4576-11e9-9605-b827eb6ec255/raspicam_node-5*.log

Attempt 2: Failed

1. Open a new Terminal.
2. Go to the person_detection_robot package folder:

   cd /home/pi/person-detection/opt/install/person_detection_robot/share/person_detection_robot/

   a. Open file ./config/h264_encoder_config.yaml and update line 6 to subscription_topic: "/raspicam_node/image". b. Open file ./launch/deploy_person_detection.launch and comment out the entire node raspicam_node.

3. Go to the raspicam_node package’s launch folder:

   cd /home/pi/catkin_ws/src/raspicam_node/launch

   a. Create a new file camerav3.launch, with the following content:

   <launch>
   <arg name="enable_raw" default="true"/>
   <arg name="enable_imv" default="false"/>
   <arg name="camera_id" default="0"/>
   <arg name="camera_frame_id" default="raspicam"/>
   <arg name="camera_name" default="camerav2_1280x960"/>
   
   <node type="raspicam_node" pkg="raspicam_node" name="raspicam_node" output="screen">
   <param name="private_topics" value="true"/>
   
   <param name="camera_frame_id" value="$(arg camera_frame_id)"/>
   <param name="enable_raw" value="$(arg enable_raw)"/>
   <param name="enable_imv" value="$(arg enable_imv)"/>
   <param name="camera_id" value="$(arg camera_id)"/>
   
   <param name="camera_info_url" value="package://raspicam_node/camera_info/camerav2_1280x960.yaml"/>
   <param name="camera_name" value="$(arg camera_name)"/>
   <param name="width" value="640"/>
   <param name="height" value="480"/>
   
   <param name="framerate" value="30"/>
   </node>
   </launch>

4. Run roslaunch raspicam_node camerav3.launch. This should bring up the camera correctly. To confirm, try rqt or rostopic echo /raspicam_node/image.
5. Open a new Terminal.
6. Run the following commands:

   export ROS_PACKAGE_PATH=/home/pi/catkin_ws/src:/opt/ros/kinetic/share:/home/pi/person-detection/opt
   export LAUNCH_ID=1554865002672
   export BUNDLE_CURRENT_PREFIX=/home/pi/person-detection
   source $BUNDLE_CURRENT_PREFIX/setup.sh

7. Run roslaunch person_detection_robot deploy_person_detection.launch. Most of the nodes should start correctly (except soundplay_node).
8. However, NO video streaming happening in Kinesis Video Stream. To confirm, run rostopic echo /video/encoded and you will see an error message saying that kinesis_video_msgs isn't available or built properly.
9. Ctrl+C to terminate the process.

Attempt 3: Failed

1. Continue from the previous attempt, and stay in the current Terminal window.
2. Run sudo apt-get install ros-kinetic-kinesis-video-msgs to install the missing library.
3. Re-run roslaunch person_detection_robot deploy_person_detection.launch, followed by rostopic echo /video/encoded to check video encoding. This time you will see messages being posted to this topic.
4. However, still NO video streaming happening in Kinesis Video Stream.
5. Go back to the terminal window and examine the console output, you will see an error on Curl returned error code 28, followed by streaming errors every few seconds:

   [ERROR] [1554947202.196626556]: [CurlHttpClient] Curl returned error code 28
   [ERROR] [1554947202.196984158]: [AWSClient] HTTP response code: 1995602712
   Exception name: 
   Error message: Unable to connect to endpoint
   0 response headers:
   ...
   [ERROR] [1554947211.574767159]: [streamErrorReportHandler] Reporting stream error. Errored timecode: 15549472006740000 Status: 1375731806
   [ERROR] [1554947216.594701267]: [streamErrorReportHandler] Reporting stream error. Errored timecode: 15549472006740000 Status: 1375731806
   [ERROR] [1554947221.610790484]: [streamErrorReportHandler] Reporting stream error. Errored timecode: 15549472006740000 Status: 1375731806
   ...

Attempt 4: Succeeded (sort of)

1. Continue from the previous attempt, and stay in the current Terminal window.
2. Since the application is interacting with two different streams: 1/ Kinesis Video Stream, and 2/ Kinesis Rekognition Data Stream, I need to know which one was causing the error.
3. Go to the person_detection_robot package’s config folder:

   cd /home/pi/person-detection/opt/install/person_detection_robot/share/person_detection_robot/config

   a. Open file ./kvs_config.yaml and update line 11 to topic_type: 1.

4. Re-run roslaunch person_detection_robot deploy_person_detection.launch.
5. Kinesis video streaming worked correctly!

This means that the Kinesis Rekognition data stream is the one having trouble, but that’s the problem for another day!

[MacInnis]

As shown above, there are lots of configuration changes that need to be made to get the app closer to working. However, there is one fundamental problem that is preventing the application from launching. The main issue appears to be a versioning conflict between what the raspicam_node (that is already installed on the TutleBot) expects, and what is bundled with the application by colcon. I've bundled the application and deployed using RoboMaker to a device. To isolate the issue, my launch file contains only the raspicam_node, and nothing else. Here's the error reported during the launch:

process[rosout-1]: started with pid [3144]
started core service [/rosout]
process[raspicam_node-2]: started with pid [3157]
/opt/ros/kinetic/lib/raspicam_node/raspicam_node: /tmp/roboMakerDeploymentPackage/61e1603d5e07125bcb2d4edd50d70cbab0ab5daad21f8082524e976b1cfc88ed//usr/lib/arm-linux-gnueabihf/libQt5Test.so.5: version `Qt_5' not found (required by /opt/ros/kinetic/lib/libopencv_highgui3.so.3.3)
/opt/ros/kinetic/lib/raspicam_node/raspicam_node: /tmp/roboMakerDeploymentPackage/61e1603d5e07125bcb2d4edd50d70cbab0ab5daad21f8082524e976b1cfc88ed//usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5: version `Qt_5' not found (required by /opt/ros/kinetic/lib/libopencv_highgui3.so.3.3)
/opt/ros/kinetic/lib/raspicam_node/raspicam_node: /tmp/roboMakerDeploymentPackage/61e1603d5e07125bcb2d4edd50d70cbab0ab5daad21f8082524e976b1cfc88ed//usr/lib/arm-linux-gnueabihf/libQt5Widgets.so.5: version `Qt_5' not found (required by /opt/ros/kinetic/lib/libopencv_highgui3.so.3.3)
/opt/ros/kinetic/lib/raspicam_node/raspicam_node: /tmp/roboMakerDeploymentPackage/61e1603d5e07125bcb2d4edd50d70cbab0ab5daad21f8082524e976b1cfc88ed//usr/lib/arm-linux-gnueabihf/libQt5Core.so.5: version `Qt_5' not found (required by /opt/ros/kinetic/lib/libopencv_highgui3.so.3.3)
/opt/ros/kinetic/lib/raspicam_node/raspicam_node: /tmp/roboMakerDeploymentPackage/61e1603d5e07125bcb2d4edd50d70cbab0ab5daad21f8082524e976b1cfc88ed//usr/lib/arm-linux-gnueabihf/libQt5Gui.so.5: version `Qt_5' not found (required by /opt/ros/kinetic/lib/libopencv_cvv3.so.3.3)
/opt/ros/kinetic/lib/raspicam_node/raspicam_node: /tmp/roboMakerDeploymentPackage/61e1603d5e07125bcb2d4edd50d70cbab0ab5daad21f8082524e976b1cfc88ed//usr/lib/arm-linux-gnueabihf/libQt5Core.so.5: version `Qt_5' not found (required by /opt/ros/kinetic/lib/libopencv_cvv3.so.3.3)
/opt/ros/kinetic/lib/raspicam_node/raspicam_node: /tmp/roboMakerDeploymentPackage/61e1603d5e07125bcb2d4edd50d70cbab0ab5daad21f8082524e976b1cfc88ed//usr/lib/arm-linux-gnueabihf/libQt5Widgets.so.5: version `Qt_5' not found (required by /opt/ros/kinetic/lib/libopencv_cvv3.so.3.3)
[raspicam_node-2] process has died [pid 3157, exit code 1, cmd /opt/ros/kinetic/lib/raspicam_node/raspicam_node __name:=raspicam_node __log:=/home/pi/.ros/log/d9f1fc52-6076-11e9-a9b5-b827eb041467/raspicam_node-2.log].
log file: /home/pi/.ros/log/d9f1fc52-6076-11e9-a9b5-b827eb041467/raspicam_node-2*.log
^C[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete

This only happens after I source the bundled application into the environment. If I do not source it, and launch the same file, the raspicam_node starts successfully.

[juanrh]

Hi @MacInnis, thanks for your patience. I've been able to have this sample application working on a TurtleBot 3 Burger running Ubuntu Xenial and ROS Kinetic. Thanks for bringing to our attention those gaps in our the documentation, I'll send a PR to update the README, and also to correct some issues in deploy_person_detection.launch (more on this below).

In order to make it work, first I have used Ubuntu Xenial instead of Raspbian. Note Ubuntu 16.04 is listed as a prerequisite in the RoboMaker documentation, and has "required support" in ROS Kinetic. On the other hand Raspbian doesn't appear as supported there, and Debian Jessie appears as "recommended support", which implies some of our package dependencies might not have been tested for Debian Jessie. Also, note that cross compilation in the RoboMaker IDE uses Ubuntu Xenial in the Docker container.

Admin:~/environment/PersonDetection/robot_ws $ head  /opt/robomaker/cross-compilation-dockerfile/Dockerfile                                                    
FROM arm32v7/ros:kinetic-ros-base-xenial

Therefore starting from Ubuntu Xenial usually makes things easier. However, we should improve our documentation about using the bundles directly. Even though there are usage instructions in the colcon bundle documentation, we could add explicit instructions along these lines:

tar xvf robot_ws_armhf_bundle.tar
tar xvzf metadata.tar.gz
# in overlays.json we see dependencies.tar.gz and workspace.tar.gz, in that order
cat overlays.json
mkdir dependencies
tar xvzf dependencies.tar.gz -C dependencies
mkdir workspace
tar xvzf workspace.tar.gz -C workspace

source /opt/ros/kinetic/setup.bash
export LAUNCH_ID=YOUR_LAUNCH_ID
# source workspaces in the same order as in overlays.json
BUNDLE_CURRENT_PREFIX=$(pwd)/dependencies source $(pwd)/dependencies/setup.sh
BUNDLE_CURRENT_PREFIX=$(pwd)/workspace source $(pwd)/workspace/setup.sh

In some of the instructions above I see some packages had to be installed, e.g. sudo apt-get install ros-kinetic-kinesis-video-msgs, which suggest an issue configuring the workspace overlay, as workspace/opt/built_workspace/kinesis_video_msgs/ is available in the workspace. With that I was able to run roslaunch person_detection_robot deploy_person_detection.launch successfully, but I wasn't able to get image in Kinesis Video Streams. I detected some misconfiguration in that launch file, that is fixed making the following changes:

<node type="raspicam_node" pkg="raspicam_node" name="raspicam_node" output="screen">
    <param name="width" value="410"/>
    <param name="height" value="308"/>
...

  <rosparam param="/h264_video_encoder/subscription_topic" subst_value="true">/raspicam_node/image</rosparam>
  <rosparam param="/h264_video_encoder/image_transport" subst_value="true">compressed</rosparam>
</launch>

I'll send a PR with this fix, as well as changes to the README with the commands to activate the overlay workspace for the bundle. It would be nice if you could try these instructions on a turtlebot running Ubuntu Xenial, to confirm the fix.

Also, can you please indicate which Raspbian version are you using? I'll try to setup the sample application on using Raspbian Jessie, but I'd like to confirm you are also using that version. Is there anything preventing you to use Ubuntu Xenial, for example the Ubuntu Mate variant, instead of Raspbian?

Finally, I was able to build the sample app in RoboMaker Cloud 9 without problems, using a fresh development environment, so I wasn't able to reproduce that issue. But please feel free to let us know in this thread if you still have issues building, it would help us if you include the specific build commands usef for that failed build

Thanks again for your patience, and for trying our sample application

[timrobotson]

I tested the latest version of Person Detection with the following setup:

• Waffle Pi Robot
• Raspberry Pi B
• Raspicam
• Ubuntu 16.04
• ROS Kinetic

Using master of all dependencies instead of the v1.0.0 tag (in https://github.com/aws-robotics/aws-robomaker-sample-application-persondetection/blob/master/robot_ws/.rosinstall)

and it worked correctly. The video streamed to Kinesis and the robot reported 'I see tim' in the logs.

I'm going to create a new PR to update the .rosinstall file to use the master branch of all dependencies now. However the instructions are correct and working after the updates in #25.