osrf-migration
commented
7 years ago Closed osrf-migration closed 7 years ago
osrf-migration
commented
7 years ago osrf-migration
commented
7 years ago osrf-migration
commented
7 years ago Original comment by Nate Koenig (Bitbucket: Nathan Koenig).
Can you make sure that you followed every step of the installation tutorial?
Also, have you ever installed Gazebo and/or ROS in the past?
osrf-migration
commented
7 years ago Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
osrf-migration
commented
7 years ago Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
The crux of the error seems to be here:
process[controller_manager_x1_2740_5196629567473667321-1]: started with pid [2765]
exception subscribing to JointAPS_Angle_Rad : /pelvis/waist/JointAPS_Angle_Rad has not been registered.
exception subscribing to JointAPS_Vel_Radps : /pelvis/waist/JointAPS_Vel_Radps has not been registered.
exception subscribing to JointTorque_Des_Nm : /pelvis/waist/JointTorque_Des_Nm has not been registered.
exception subscribing to JointTorque_Meas_Nm : /pelvis/waist/JointTorque_Meas_Nm has not been registered.
exception subscribing to Position_Des_Rad : /pelvis/waist/Position_Des_Rad has not been registered.
exception subscribing to Velocity_Des_Radps : /pelvis/waist/Velocity_Des_Radps has not been registered.
2016-11-06 20:38:26 localhost ncl_cpp: [ERROR ] [gov.nasa.RobotInterface.addJointsAndActuators] Exception encountered while attempting to add an actuator! Check your URDF for correct XML formatting!
[ERROR] [1478482706.638611814, 3.792000000]: Exception encountered while attempting to add an actuator! Check your URDF for correct XML formatting!
2016-11-06 20:38:26 localhost ncl_cpp: [ERROR ] [gov.nasa.HardwareInterface.makeHandle] Robot Hardware has failed Initialized properly.
[ERROR] [1478482706.638921238, 3.792000000]: Fatal exception error encountered initializing RobotHardwareInterface, exiting
2016-11-06 20:38:26 localhost ncl_cpp: [FATAL ] [gov.nasa.ControllerExec.ORO_main] Error encountered during configure hook for controller_manager component!cc @jordanlack
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
Is this happening every time for you? It's likely a race condition as @scpeters mentioned. We have 2 nodes that are launched and the 2nd one needs the first one to be alive when it launches. We currently hack in waits to try to make sure the first node has time to get started. I see this sometimes, but it never happens for me when things go smoothly. I see it sometimes when gazebo segfaults or something.
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
I did follow all the steps (and by the way, suggested it may be a race condition). I have used ROS for a long time and am familiar with setups.
This is on a new Ubuntu/Inidgo installation (because I had 16.04 and kinetic running, so I did a clean install). I did not install Gazebo. It also happens on a second machine that already had 14.04/Indigo installed. That machine also did not have Gazebo installed.
It crashes every time on both machines for one reason or another, either the seg fault or the controller issue. If it gets all the way through to showing the world with the robot, then the robot collapses after about 30 seconds. None of the input messages have any effect.
If I exclude the following part of the launch file, then it loads most of the time and the robot remains in the harness. This works for both qual1.world and qual2.world:
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
Could you try roslaunch val_gazebo val_sim_gazebo_sync.launch or roslaunch val_gazebo val_sim.launch? If that fails with the same exception then it's definitely our(NASA's) fault. Basically i'm not sure how commenting out those lines you have there are helping at all. That makes me think something else is bringing down one of our nodes and the controller_manager is just where the first exception comes in. Not really sure how that could happen though...Let me know if your able to reliably launch those two launch files.
osrf-migration
commented
7 years ago Original comment by Erica Tiberia (Bitbucket: T_AL).
I also have the same setup as Dan. I tried the first simulation, Val launched in the simulation but I received the following errors.
ROS_MASTER_URI=http://localhost:11311
core service [/rosout] found
process[urdf_spawner-1]: started with pid [5656]
spawn_model script started
[INFO] [WallTime: 1478546649.517734] [0.000000] Loading model xml from ros parameter
[INFO] [WallTime: 1478546649.536395] [0.000000] Waiting for service /gazebo/spawn_urdf_model
[INFO] [WallTime: 1478546649.539783] [0.000000] Calling service /gazebo/spawn_urdf_model
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[sensor_number], child of element[sensor] not defined in SDF. Ignoring[sensor_number]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[sensor_number], child of element[sensor] not defined in SDF. Ignoring[sensor_number]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
[ INFO] [1478546650.475775468]: Camera Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478546650.475916877]: Camera Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478546650.488222610]: Camera Plugin (ns = /)
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
It's a bit difficult to parse through that. Could you format that please next time? It looks to me though like you don't have any critical problems there though. A lot of those are typical things that I don't think would cause the sim to not work. I see an [ERROR] msg in there that we should change to [WARN].
A lot of those SDF warnings are because we put a bunch of tags in our URDF that our controller manager parses out, so those aren't anything to be concerned about.
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
Jordan, the launch files you suggested both load the model OK. There are a bunch of warnings, but gazebo comes up very quickly and the robot model loads fine. See my additional comments for the behavior when I break up the launch file into two launch files, where you can see the first part loads OK (leaving the robot in the harness) and then the second part generates errors.
Just curious, you have gotten the tutorial to work with a completely new install of Ubuntu 14.04 and ROS Indigo, right? Just want to be sure there is not some environment variable or something like that I'm missing.
Here is the log from the launch file you asked me to run:
#!python
starting: roslaunch val_gazebo spawn_urdf_model_from_robot_description.launch modelName:=valkyrie zOffset:=1.25 __name:=urdf_spawner_x1_16607_3462358118607665818 __log:=/home/dbarry/.ros/log/b3087a50-a532-11e6-bd66-e4a47193b50c/urdf_spawner_x1_16607_3462358118607665818-7.log
... logging to /home/dbarry/.ros/log/b3087a50-a532-11e6-bd66-e4a47193b50c/roslaunch-x1-16862.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://127.0.0.1:33335/
SUMMARY
========
PARAMETERS
* /rosdistro: indigo
* /rosversion: 1.11.20
NODES
/
urdf_spawner (gazebo_ros/spawn_model)
ROS_MASTER_URI=http://localhost:11311
core service [/rosout] found
process[urdf_spawner-1]: started with pid [16880]
spawn_model script started
[INFO] [WallTime: 1478554792.532189] [0.000000] Loading model xml from ros parameter
[INFO] [WallTime: 1478554792.550197] [0.000000] Waiting for service /gazebo/spawn_urdf_model
[INFO] [WallTime: 1478554792.553837] [0.000000] Calling service /gazebo/spawn_urdf_model
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[sensor_number], child of element[sensor] not defined in SDF. Ignoring[sensor_number]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[sensor_number], child of element[sensor] not defined in SDF. Ignoring[sensor_number]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
[ INFO] [1478554793.515908277]: Camera Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478554793.516240213]: Camera Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478554793.530558016]: Camera Plugin (ns = /) <tf_prefix_>, set to ""
[ INFO] [1478554793.535582269]: Laser Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478554793.535735219]: Starting GazeboRosLaser Plugin (ns = /)!
[ INFO] [1478554793.546875073]: Camera Plugin (ns = /) <tf_prefix_>, set to ""
[ INFO] [1478554793.546998273]: GPU Laser Plugin (ns = /) <tf_prefix_>, set to ""
[ INFO] [1478554793.562322079]: LoadThread function completed
[INFO] [WallTime: 1478554793.637370] [0.000000] Spawn status: SpawnModel: Successfully spawned model
[urdf_spawner-1] process has finished cleanly
log file: /home/dbarry/.ros/log/b3087a50-a532-11e6-bd66-e4a47193b50c/urdf_spawner-1*.log
all processes on machine have died, roslaunch will exit
[ INFO] [1478554794.022449946]: SharedMemoryInterfacePlugin: Loading
shutting down processing monitor...
... shutting down processing monitor complete
done
2016-11-07 16:39:54 localhost ncl_cpp: [ERROR ] [gov.nasa.SharedMemorySimInterfacePlugin.findUrdfSensors] Possibly missing an api or node tag from one or more imu sensors
2016-11-07 16:39:54 localhost ncl_cpp: [WARN ] [gov.nasa.SharedMemorySimInterfacePlugin.findUrdfSensors] Imu sensor head_imu_sensor doesn't have <imuTransform> tag. It is likely that the data you receive from this sensor will be incorrect.
[ INFO] [1478554794.082361202]: SharedMemorySimInterfacePlugin: Loading complete!
[urdf_spawner_x1_16607_3462358118607665818-7] process has finished cleanly
log file: /home/dbarry/.ros/log/b3087a50-a532-11e6-bd66-e4a47193b50c/urdf_spawner_x1_16607_3462358118607665818-7*.log
[ INFO] [1478554794.387790691]: imu plugin missing <frameName>, defaults to <bodyName>
[ INFO] [1478554794.388730500]: imu plugin missing <frameName>, defaults to <bodyName>
[ INFO] [1478554794.389804788]: imu plugin missing <frameName>, defaults to <bodyName>Original comment by dan (Bitbucket: dan77062).
Also, here is the modified launch file, along with the log. In this case, the world and the robot load OK and the robot just stays in the harness.
#!python
<launch>
<!-- start with robot harnessed by default. -->
<!-- set:
roslaunch srcsim qual1.launch init:=true
to automatically lower robot harnessed, start high level controller and detach harness. -->
<!-- set:
roslaunch srcsim qual1.launch init:=true walk_test:=walk
to automatically walk forward after harness detach. -->
<arg name="debug" default="false" />
<arg name="extra_gazebo_args" default=""/>
<arg name="init" default="false" />
<arg name="paused" default="false" />
<arg name="use_local_build" default="false" />
<arg name="verbose" default="true" />
<arg name="walk_test" default="false"/>
<arg name="world_name" default="$(find srcsim)/worlds/qual2.world" />
<include file="$(find val_gazebo)/launch/val_sim_gazebo_sync.launch" pass_all_args="false">
<arg name="debug" value="$(arg debug)" />
<arg name="extra_gazebo_args" default="$(arg extra_gazebo_args)"/>
<arg name="paused" value="$(arg paused)" />
<arg name="verbose" value="$(arg verbose)" />
<arg name="world_name" value="$(arg world_name)" />
</include>
<include file="$(find ihmc_valkyrie_ros)/launch/valkyrie_ros_api.launch" pass_all_args="false">
<arg name="use_local_build" value="$(arg use_local_build)" />
</include>
</launch>log output:
#!python
dbarry@x1:/opt/ros/indigo/share/srcsim/launch$ gedit myQ2A.launch
dbarry@x1:/opt/ros/indigo/share/srcsim/launch$ roslaunch srcsim myQ2A.launch
... logging to /home/dbarry/.ros/log/457c946a-a534-11e6-b4e2-e4a47193b50c/roslaunch-x1-18436.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://127.0.0.1:33092/
SUMMARY
========
PARAMETERS
* /ihmc_ros/robot_name: valkyrie
* /ihmc_ros/valkyrie/left_arm_joint_names: ['leftShoulderPit...
* /ihmc_ros/valkyrie/left_foot_frame_name: leftFoot
* /ihmc_ros/valkyrie/right_arm_joint_names: ['rightShoulderPi...
* /ihmc_ros/valkyrie/right_foot_frame_name: rightFoot
* /robot_description: <?xml version="1....
* /rosdistro: indigo
* /rosversion: 1.11.20
* /use_sim_time: True
* /valkyrie/robot_description: <?xml version="1....
NODES
/
IHMCValkyrieROSAPI (ihmc_ros_java_adapter/gradlew)
control_py_x1_18436_74504223220918311 (robonet_tools/control)
controller_manager_x1_18436_4664789296967194160 (val_deploy/delayed_roslaunch.sh)
gazebo (gazebo_ros/gzserver)
gazebo_gui (gazebo_ros/gzclient)
robot_state_publisher (robot_state_publisher/state_publisher)
smtcore_x1_18436_8394651460433446721 (shared_memory_transport/smtcore)
urdf_spawner_x1_18436_3884517551069303702 (val_deploy/delayed_roslaunch.sh)
auto-starting new master
process[master]: started with pid [18448]
ROS_MASTER_URI=http://localhost:11311
setting /run_id to 457c946a-a534-11e6-b4e2-e4a47193b50c
process[rosout-1]: started with pid [18461]
started core service [/rosout]
process[smtcore_x1_18436_8394651460433446721-2]: started with pid [18470]
process[control_py_x1_18436_74504223220918311-3]: started with pid [18480]
process[controller_manager_x1_18436_4664789296967194160-4]: started with pid [18487]
delaying: 4 seconds
process[gazebo-5]: started with pid [18489]
process[gazebo_gui-6]: started with pid [18493]
delaying: 6 seconds
process[urdf_spawner_x1_18436_3884517551069303702-7]: started with pid [18497]
process[robot_state_publisher-8]: started with pid [18499]
process[IHMCValkyrieROSAPI-9]: started with pid [18500]
Log file /home/dbarry/.log already exists, proceeding...
Log file /home/dbarry/.log already exists, proceeding...
Gazebo multi-robot simulator, version 7.4.0
Copyright (C) 2012-2016 Open Source Robotics Foundation.
Released under the Apache 2 License.
http://gazebosim.org
[ INFO] [1478555461.212353948]: Finished loading Gazebo ROS API Plugin.
[Msg] Waiting for master.
[ INFO] [1478555461.213066339]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...
[Msg] Connected to gazebo master @ http://127.0.0.1:11345
[Msg] Publicized address: 192.168.1.249
> Loading > buildSrc[ INFO] [1478555462.383495051, 0.021000000]: waitForService: Service [/gazebo/set_physics_properties] is now available.
> settings[ INFO] [1478555462.453519153, 0.090000000]: Physics dynamic reconfigure ready.
starting: roslaunch val_deploy val_control_sim.launch controller_manager_looprate:=500 __name:=controller_manager_x1_18436_4664789296967194160 __log:=/home/dbarry/.ros/log/457c946a-a534-11e6-b4e2-e4a47193b50c/controller_manager_x1_18436_4664789296967194160-4.log
... logging to /home/dbarry/.ros/log/457c946a-a534-11e6-b4e2-e4a47193b50c/roslaunch-x1-18762.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://127.0.0.1:40084/
SUMMARY
========
PARAMETERS
* /rosdistro: indigo
* /rosversion: 1.11.20
NODES
/
controller_manager_x1_18762_4363294844662033416 (val_controller_manager_rtt/controller_exec)
ROS_MASTER_URI=http://localhost:11311
core service [/rosout] found
process[controller_manager_x1_18762_4363294844662033416-1]: started with pid [18793]
starting: roslaunch val_gazebo spawn_urdf_model_from_robot_description.launch modelName:=valkyrie zOffset:=1.25 __name:=urdf_spawner_x1_18436_3884517551069303702 __log:=/home/dbarry/.ros/log/457c946a-a534-11e6-b4e2-e4a47193b50c/urdf_spawner_x1_18436_3884517551069303702-7.log
2016-11-07 16:51:07 localhost ncl_cpp: [NOTICE ] [gov.nasa.HardwareInterface.makeHandle] Robot Hardware has Initialized. Ready to start loading Controllers.
... logging to /home/dbarry/.ros/log/457c946a-a534-11e6-b4e2-e4a47193b50c/roslaunch-x1-18861.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
> Resolving dependencies ':runtime'started roslaunch server http://127.0.0.1:36218/
SUMMARY
========
PARAMETERS
* /rosdistro: indigo
* /rosversion: 1.11.20
NODES
/
urdf_spawner (gazebo_ros/spawn_model)
ROS_MASTER_URI=http://localhost:11311
core service [/rosout] found
process[urdf_spawner-1]: started with pid [18914]
spawn_model script started
[INFO] [WallTime: 1478555468.462656] [0.000000] Loading model xml from ros parameter
[INFO] [WallTime: 1478555468.508224] [5.860000] Waiting for service /gazebo/spawn_urdf_model
[INFO] [WallTime: 1478555468.517156] [5.867000] Calling service /gazebo/spawn_urdf_model
:buildSrc:compileJava UP-TO-DATE
> Loading > buildSrc > :buildSrc:compileGroovyWarning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[sensor_number], child of element[sensor] not defined in SDF. Ignoring[sensor_number]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[sensor_number], child of element[sensor] not defined in SDF. Ignoring[sensor_number]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[imuTransform], child of element[sensor] not defined in SDF. Ignoring[imuTransform]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[node], child of element[sensor] not defined in SDF. Ignoring[node]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[api], child of element[sensor] not defined in SDF. Ignoring[api]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
Warning [parser.cc:778] XML Element[port], child of element[sensor] not defined in SDF. Ignoring[port]. You may have an incorrect SDF file, or an sdformat version that doesn't support this element.
:buildSrc:compileGroovy UP-TO-DATE
:buildSrc:processResources UP-TO-DATE
:buildSrc:classes UP-TO-DATE
:buildSrc:jar UP-TO-DATE
:buildSrc:assemble UP-TO-DATE
:buildSrc:compileTestJava UP-TO-DATE
:buildSrc:compileTestGroovy UP-TO-DATE
:buildSrc:processTestResources UP-TO-DATE
:buildSrc:testClasses UP-TO-DATE
:buildSrc:test UP-TO-DATE
:buildSrc:check UP-TO-DATE
:buildSrc:build UP-TO-DATE
> Configuring > 0/1 projects > root project[ INFO] [1478555470.036675136, 6.116000000]: Camera Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478555470.037261219, 6.116000000]: Camera Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478555470.045953231, 6.116000000]: Camera Plugin (ns = /) <tf_prefix_>, set to ""
[ INFO] [1478555470.046415560, 6.116000000]: Camera Plugin (ns = /) <tf_prefix_>, set to ""
[ INFO] [1478555470.134867150, 6.116000000]: Laser Plugin (robotNamespace = /), Info: Using the 'robotNamespace' param: '/'
[ INFO] [1478555470.137985311, 6.116000000]: Starting GazeboRosLaser Plugin (ns = /)!
[ INFO] [1478555470.199057684, 6.116000000]: GPU Laser Plugin (ns = /) <tf_prefix_>, set to ""
[ INFO] [1478555470.202693639, 6.116000000]: LoadThread function completed
> Resolving dependencies ':ros'[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[multicamera] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[gpu_ray] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[force_torque] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[force_torque] not suppported.
[INFO] [WallTime: 1478555470.515247] [6.116000] Spawn status: SpawnModel: Successfully spawned model
[urdf_spawner-1] process has finished cleanly
log file: /home/dbarry/.ros/log/457c946a-a534-11e6-b4e2-e4a47193b50c/urdf_spawner-1*.log
all processes on machine have died, roslaunch will exit
shutting down processing monitor...
... shutting down processing monitor complete
done
[Wrn] [ColladaLoader.cc:1461] Polylist input semantic: 'COLOR' is currently not supported
[urdf_spawner_x1_18436_3884517551069303702-7] process has finished cleanly
log file: /home/dbarry/.ros/log/457c946a-a534-11e6-b4e2-e4a47193b50c/urdf_spawner_x1_18436_3884517551069303702-7*.log
[Wrn] [msgs.cc:1793] Conversion of sensor type[force_torque] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[force_torque] not suppported.
[ INFO] [1478555471.324676345, 6.116000000]: SharedMemoryInterfacePlugin: Loading
2016-11-07 16:51:11 localhost ncl_cpp: [ERROR ] [gov.nasa.SharedMemorySimInterfacePlugin.findUrdfSensors] Possibly missing an api or node tag from one or more imu sensors
2016-11-07 16:51:11 localhost ncl_cpp: [WARN ] [gov.nasa.SharedMemorySimInterfacePlugin.findUrdfSensors] Imu sensor head_imu_sensor doesn't have <imuTransform> tag. It is likely that the data you receive from this sensor will be incorrect.
[ INFO] [1478555471.427918735, 6.116000000]: SharedMemorySimInterfacePlugin: Loading complete!
[ INFO] [1478555471.928210210, 6.116000000]: imu plugin missing <frameName>, defaults to <bodyName>
[ INFO] [1478555471.946729753, 6.116000000]: imu plugin missing <frameName>, defaults to <bodyName>
[ INFO] [1478555471.947519661, 6.116000000]: imu plugin missing <frameName>, defaults to <bodyName>
[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[imu] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[multicamera] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[gpu_ray] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[force_torque] not suppported.
[Wrn] [msgs.cc:1793] Conversion of sensor type[force_torque] not suppported.
:parseYaml UP-TO-DATE
:runJavaDelegate
Loading robot model from: 'models/val_description/sdf/valkyrie_sim.sdf'
[INFO] (NetworkParameters.java:38): Looking for network parameters in network parameters file at /opt/ros/indigo/share/ihmc_valkyrie_ros/configurations/IHMCNetworkParametersSim.ini
[INFO] (NetworkParameters.java:42): Found Network parameters file at /opt/ros/indigo/share/ihmc_valkyrie_ros/configurations/IHMCNetworkParametersSim.ini
[INFO] (NetworkParameters.java:69): Looking for network parameters in environment variables
[INFO] (NetworkParameters.java:70): Environment variables will override entries in the network parameters file.
Connecting to controller using TCP on localhost
[WARN] YoVariable: desiredMomentumRateLinearX is getting created with a null registry
[WARN] YoVariable: desiredMomentumRateLinearY is getting created with a null registry
[WARN] YoVariable: desiredMomentumRateLinearZ is getting created with a null registry
[WARN] YoVariable: achievedMomentumRateLinearX is getting created with a null registry
[WARN] YoVariable: achievedMomentumRateLinearY is getting created with a null registry
[WARN] YoVariable: achievedMomentumRateLinearZ is getting created with a null registry
[WARN] YoVariable: residualRootJointForceX is getting created with a null registry
[WARN] YoVariable: residualRootJointForceY is getting created with a null registry
[WARN] YoVariable: residualRootJointForceZ is getting created with a null registry
[WARN] YoVariable: residualRootJointTorqueX is getting created with a null registry
[WARN] YoVariable: residualRootJointTorqueY is getting created with a null registry
[WARN] YoVariable: residualRootJointTorqueZ is getting created with a null registry
log4j:WARN No appenders could be found for logger (org.ros.internal.node.client.Registrar).
log4j:WARN Please initialize the log4j system properly.
IHMC ROS API node successfully started.
> Building 50% > :runJavaDelegate
Original comment by dan (Bitbucket: dan77062).
Now, if I load the modified launch file as in the comment above, then run the rest of the original launch file as a separate launch file, errors show up and the robot eventually collapses.
Here is the separate launch file, followed by the log from its terminal window, followed by the log from the terminal window where the first launch file (see comment above) is running.
#!python
<launch>
<arg name="debug" default="false" />
<arg name="extra_gazebo_args" default=""/>
<arg name="init" default="false" />
<arg name="paused" default="false" />
<arg name="use_local_build" default="false" />
<arg name="verbose" default="true" />
<arg name="walk_test" default="false"/>
<arg name="world_name" default="$(find srcsim)/worlds/qual2.world" />
<include file="$(find ihmc_valkyrie_ros)/launch/val_wholebody_control_gazebo.launch"/>
<group if="$(arg init)">
<node name="startup_robot" pkg="srcsim" type="init_robot.sh" args="$(arg walk_test)" output="screen"/>
</group>
</launch>log from this terminal window:
#!python
dbarry@x1:/opt/ros/indigo/share/srcsim/launch$ roslaunch srcsim myQshort.launch extra_gazebo_args:="-r" init:="true"
... logging to /home/dbarry/.ros/log/3c127a36-a537-11e6-9ddf-e4a47193b50c/roslaunch-x1-22095.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://127.0.0.1:34585/
SUMMARY
========
PARAMETERS
* /ihmc_valkyrie_control_java_bridge/jvm_args: -Djava.class.path...
* /ihmc_valkyrie_control_java_bridge/main_class: us.ihmc.valkyrieR...
* /ihmc_valkyrie_control_java_bridge/type: ihmc_ros_control/...
* /ihmc_valkyrie_control_java_bridge/working_dir: /home/dbarry/valk...
* /joint_state_controller/publish_rate: 50
* /joint_state_controller/type: joint_state_contr...
* /rosdistro: indigo
* /rosversion: 1.11.20
NODES
/
controller_spawner_x1_22095_6947010783665914578 (controller_manager/spawner)
startup_robot (srcsim/init_robot.sh)
ROS_MASTER_URI=http://localhost:11311
core service [/rosout] found
process[controller_spawner_x1_22095_6947010783665914578-1]: started with pid [22113]
wait 60s for system to load
process[startup_robot-2]: started with pid [22114]
[INFO] [WallTime: 1478556824.954298] [0.000000] Controller Spawner: Waiting for service controller_manager/load_controller
[INFO] [WallTime: 1478556824.957467] [0.000000] Controller Spawner: Waiting for service controller_manager/switch_controller
[INFO] [WallTime: 1478556824.959027] [0.000000] Controller Spawner: Waiting for service controller_manager/unload_controller
[INFO] [WallTime: 1478556824.960807] [0.000000] Loading controller: ihmc_valkyrie_control_java_bridge
[ERROR] [WallTime: 1478556826.601918] [21.537000] Failed to load ihmc_valkyrie_control_java_bridge
[INFO] [WallTime: 1478556826.602379] [21.537000] Controller Spawner: Loaded controllers:
[INFO] [WallTime: 1478556826.823066] [21.587000] Started controllers:
lower harness
publishing and latching message for 3.0 seconds
switch to high level control after 20 seconds
publishing and latching message for 3.0 seconds
detach in 5 seconds
done
[startup_robot-2] process has finished cleanly
log file: /home/dbarry/.ros/log/3c127a36-a537-11e6-9ddf-e4a47193b50c/startup_robot-2*.log
publishing and latching message for 3.0 secondsand the continuation of the log from the terminal window where the shortened launch file from the comment above is running:
#!python
> Building 50% > :runJavaDelegate[ INFO] [1478556825.024713139, 21.250000000]: Starting JVM with arguments: -Djava.class.path=ValkyrieController.jar -XX:+UseSerialGC -Xmx4g -Xms4g -XX:NewSize=3g -XX:MaxNewSize=3g -XX:CompileThreshold=1000 -verbosegc -Djava.library.path=lib/
Starting Java VM from path /home/dbarry/valkyrie
Started Java VM: success
[ERROR] [1478556825.586647120, 21.327000000]: Nov 07, 2016 5:13:45 PM us.ihmc.valkyrieRosControl.ValkyrieAffinity <init>
[ERROR] [1478556825.586701008, 21.327000000]: SEVERE: WARNING: Hyper-Threading is enabled. Expect higher amounts of jitter
[ERROR] [1478556825.590066749, 21.328000000]: Cannot create controller object
[ERROR] [1478556825.590127477, 21.328000000]: Initializing controller 'ihmc_valkyrie_control_java_bridge' failed
[ INFO] [1478556825.590388801, 21.328000000]: Stopping VM
[ERROR] [1478556825.591343194, 21.328000000]: Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
[ERROR] [1478556825.591761442, 21.328000000]: at java.util.ArrayList.rangeCheck(ArrayList.java:635)
[ERROR] [1478556825.592029007, 21.328000000]: at java.util.ArrayList.get(ArrayList.java:411)
[ERROR] [1478556825.592177665, 21.328000000]: at us.ihmc.affinity.Package.getCore(Package.java:45)
[ERROR] [1478556825.599444161, 21.328000000]: at us.ihmc.valkyrieRosControl.ValkyrieAffinity.<init>(ValkyrieAffinity.java:31)
[ERROR] [1478556825.600246263, 21.328000000]: at us.ihmc.valkyrieRosControl.ValkyrieRosControlController.<init>(ValkyrieRosControlController.java:107)Original comment by Jordan Lack (Bitbucket: jordanlack).
@scpeters has this launch configuration worked reliably for you all?
@dan77062 try launching the full thing again and in another terminal run tail -n 50 -f /var/log/nasa_common_logging.log and keep an eye on the log messages as it launches to see if there are some log messages that are important but aren't getting to your console maybe.
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
Jordan, there is no log file by that name in /var/log
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
Ah ok, this is helpful! Disregard my previous request. How many cores are on your processor? IHMC's controller does some fancy core isolation stuff, so if you don't have enough cores then they can't pin their processes to cores that don't exist.
@scpeters it should be clearly stated in the documentation that they must 4 cores to run this stuff. Since IHMC's controller/estimator steal 2 cores and those are hardcoded as cores(zero indexed) 1 & 2 I think, so if they don't have those cores then none of this will work.
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
4 cores. I am using a Lenovo X1 carbon with an Intel Core i7-6600U CPU, 2.60 GHz x 4, with 16 GB of RAM, and 64 bit Ubuntu 14.04
osrf-migration
commented
7 years ago osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
no not a VM. Also, it is a full up, brand new install of 14.04, with desktop Indigo
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
Hmm, ok well then i'm a bit stumped as to why this is happening then. What's happening is, in IHMC's java code, the following line controlThreadProcessor = socket.getCore(2).getDefaultProcessor(); is failing because it thinks you only have 2 cores, so with zero indexing core 2 doesn't exist.
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
interesting. Top shows all 4 cores are available and running. See below. It is a dual boot machine with Windows 10, but I don't see how that could matter.
#!python
dbarry@x1:/var/log$ cat /proc/cpuinfo | grep processor | wc -l
4#!python
dbarry@x1:/var/log$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 2
Core(s) per socket: 2
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 78
Stepping: 3
CPU MHz: 1331.093
BogoMIPS: 5615.85
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 4096K
NUMA node0 CPU(s): 0-3
dbarry@x1:/var/log$ top
top - 18:21:47 up 21:44, 4 users, load average: 0.17, 0.17, 0.45
Tasks: 270 total, 1 running, 268 sleeping, 0 stopped, 1 zombie
%Cpu0 : 3.4 us, 0.3 sy, 0.0 ni, 95.3 id, 1.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 2.7 us, 0.7 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 3.0 us, 1.0 sy, 0.0 ni, 95.4 id, 0.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 2.3 us, 1.0 sy, 0.0 ni, 96.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 16296776 total, 10715200 used, 5581576 free, 416932 buffers
KiB Swap: 16641020 total, 0 used, 16641020 free. 4269696 cached MemOriginal comment by sringer99 (Bitbucket: sringer99).
Do not know if this will help, but I am having the same issues. controller_manager crashes. zero successes.
Based on this thread and a little poking around I doubled the sleep time in /opt/nasa/indigo/share/val_deploy/scripts/delayed_roslaunch.sh
echo "delaying: $1 seconds"
sleep $1
sleep $1 # Added this extra line
shift
echo "starting: roslaunch $@"
roslaunch $@
Still get lots of error messages but the walk test now works about 90% of the time.
I have a new AMD FX 6300 with 6 cores, 16GB memory, 256GB SSD drive, running on a ASUS M5A97 motherboard.
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
@sringer99 that probably means we need to extend some of our sleeps to give some nodes more time to spin up.
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
I wrote a short java script to check on the processors available to Java. It prints out 4.
#!python
class checkProcessors {
public static void main(String[] args) {
int processors = Runtime.getRuntime().availableProcessors();
System.out.println(processors);
}
}Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
Here's some extra debugging steps:
cat /sys/devices/system/cpu/onlineand
cat /sys/devices/system/cpu/cpu0/topology/physical_package_id
cat /sys/devices/system/cpu/cpu1/topology/physical_package_id
cat /sys/devices/system/cpu/cpu2/topology/physical_package_id
cat /sys/devices/system/cpu/cpu3/topology/physical_package_idcc @dljsjr
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
The first returns 0-3. However all the others return 0. What's up with that? The core_id results are 0, 1, 0, 1 for cores 0, 1, 2, 3 respectively.
#!python
cat /sys/devices/system/cpu/online
0-3
cat /sys/devices/system/cpu/cpu0/topology/physical_package_id
0
cat /sys/devices/system/cpu/cpu1/topology/physical_package_id
0
cat /sys/devices/system/cpu/cpu2/topology/physical_package_id
0
cat /sys/devices/system/cpu/cpu3/topology/physical_package_id
0Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
The physical_package_id should be 0, as it means the socket that the CPU's are plugged into. My core_id's are: 0 1 2 3 0 1 2 3 for 0-7, so that looks fine.
osrf-migration
commented
7 years ago Original comment by Jedediyah Williams (Bitbucket: Jedediyah).
I'll chime in with @dan77062 @sringer99 and @T_AL, as I'm running into what looks like the identical problem getting the controller to run. I'm also on a fresh install of Ubuntu 14.04 / Indigo, 4 cores and 32 GB RAM, and used the updated install instructions.
@sringer99, I'm going to try out adding that extra sleep time.
osrf-migration
commented
7 years ago Original comment by Steve Peters (Bitbucket: Steven Peters, GitHub: scpeters).
Are you all using laptops?
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
yes, lenovo x1 carbon laptop and it turns out the Intel Core i7-6600U CPU does indeed have only 2 cores, 4 threads. Unfortunately, I am traveling, away from home, until after Thanksgiving and planned to work on this project during a bunch of long airplane flights. :/ Will it really not work with this machine??
osrf-migration
commented
7 years ago Original comment by Jedediyah Williams (Bitbucket: Jedediyah).
I'm on a desktop.
AMD A10-6800 quad-core, MSI GeForce GTX 950, 32 GB RAM
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
I found the code location and it seems this only comes up twice and only one of those looks applicable. Furthermore, pinning to specific cores happened at a change on Aug 12, 2015. Before that the code had an if....else that covered for fewer cores. Can you change the values to be 0 and 1 instead of 1 and 2, then make it available? I'd be happy to test it. Here is the link to the code at the ihmc github site
osrf-migration
commented
7 years ago Original comment by Jeremy White (Bitbucket: knitfoo).
Another me too report. I'm a bit odd, in that I had a working configuration. It seemed like I had sporadic crashes earlier, but I had a good 2-3 hour session yesterday where everything worked. Today, I went to run, having only done a suspend/resume cycle on the laptop (Lenovo, 16G, i7), and now I get constant failures.
Top of the log from running roslaunch -v val_gazebo val_sim_gazebo_sync.launch:
#!Java
delaying: 4 seconds
delaying: 6 seconds
Log file /home/jwhite/.log already exists, proceeding...
Log file /home/jwhite/.log already exists, proceeding...
starting: roslaunch val_deploy val_control_sim.launch controller_manager_looprate:=500 __name:=controller_manager_jertop_13066_6395671467336390700 __log:=/home/jwhite/.ros/log/b76bd212-a55b-11e6-ac34-28d244520677/controller_manager_jertop_13066_6395671467336390700-4.log
exception subscribing to JointAPS_Angle_Rad : /pelvis/waist/JointAPS_Angle_Rad has not been registered.
exception subscribing to JointAPS_Vel_Radps : /pelvis/waist/JointAPS_Vel_Radps has not been registered.
exception subscribing to JointTorque_Des_Nm : /pelvis/waist/JointTorque_Des_Nm has not been registered.
exception subscribing to JointTorque_Meas_Nm : /pelvis/waist/JointTorque_Meas_Nm has not been registered.
exception subscribing to Position_Des_Rad : /pelvis/waist/Position_Des_Rad has not been registered.
exception subscribing to Velocity_Des_Radps : /pelvis/waist/Velocity_Des_Radps has not been registered.
[31m[ERROR] [1478572407.598196595]: Exception encountered while attempting to add an actuator! Check your URDF for correct XML formatting![0m
[31m[ERROR] [1478572407.598589478]: Fatal exception error encountered initializing RobotHardwareInterface, exiting[0m
2016-11-07 20:33:27 localhost ncl_cpp: [ERROR ] [gov.nasa.RobotInterface.addJointsAndActuators] Exception encountered while attempting to add an actuator! Check your URDF for correct XML formatting!
2016-11-07 20:33:27 localhost ncl_cpp: [ERROR ] [gov.nasa.HardwareInterface.makeHandle] Robot Hardware has failed Initialized properly.
2016-11-07 20:33:27 localhost ncl_cpp: [FATAL ] [gov.nasa.ControllerExec.ORO_main] Error encountered during configure hook for controller_manager component!
[31m================================================================================REQUIRED process [controller_manager_jertop_13276_698330915684010381-1] has died!I get those failures with either qual script and with either the sync or val_sim.launch.
osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
Tried to solve by lying to linux about how many cores I have. This does make it list 0-7 but does not fix the problem-- same errors.
#!python
echo '0-7' > /tmp/online
sudo mount --bind /tmp/online /sys/devices/system/cpu/onlineOriginal comment by Jeremy White (Bitbucket: knitfoo).
One more note - I rebooted, and now I'm back to a working state.
osrf-migration
commented
7 years ago Original comment by sringer99 (Bitbucket: sringer99).
Could this be a video card issue? I am using a Radeon HD 5450 which does not match the recommended card. Got a new GTX 750 ti on the way.
What video card/system is everyone using? Maybe we can find a common thread to this problem.
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
For those having the problem where it launches successfully sometimes and sometimes you get the following exception,
#!bash
exception subscribing to JointAPS_Angle_Rad : /pelvis/waist/JointAPS_Angle_Rad has not been registered.
exception subscribing to JointAPS_Vel_Radps : /pelvis/waist/JointAPS_Vel_Radps has not been registered.
exception subscribing to JointTorque_Des_Nm : /pelvis/waist/JointTorque_Des_Nm has not been registered.
exception subscribing to JointTorque_Meas_Nm : /pelvis/waist/JointTorque_Meas_Nm has not been registered.
exception subscribing to Position_Des_Rad : /pelvis/waist/Position_Des_Rad has not been registered.
exception subscribing to Velocity_Des_Radps : /pelvis/waist/Velocity_Des_Radps has not been registered.
[31m[ERROR] [1478572407.598196595]: Exception encountered while attempting to add an actuator! Check your URDF for correct XML formatting![0mI have some debug steps that'll help me be certain what the problem is. For the debug, do roslaunch val_gazebo val_sim.launch, but first comment out the following lines in val_sim.launch,
#!xml
<include file="$(find val_gazebo)/launch/val_empty_world.launch">
<arg name="paused" value="$(arg paused)"/>
<arg name="use_sim_time" value="$(arg use_sim_time)"/>
<arg name="extra_gazebo_args" default="$(arg extra_gazebo_args)"/>
<arg name="gui" value="$(arg gui)" />
<arg name="headless" value="$(arg headless)"/>
<arg name="debug" value="$(arg debug)" />
<arg name="physics" default="$(arg physics)"/>
<arg name="verbose" value="$(arg verbose)"/>
<arg name="world_name" value="$(arg world_name)"/>
</include>Then launch it multiple times and see if you get that exception still or if it launches cleanly 100% of the time. If it is still failing occasionally, in the following line change the 4 to larger values and see if that fixes it,
#!xml
<node name="$(anon controller_manager)" pkg="val_deploy" type="delayed_roslaunch.sh" args="4 val_deploy val_control_sim.launch controller_manager_looprate:=$(arg controller_manager_looprate)" output="screen"/>Let me know how it goes!
osrf-migration
commented
7 years ago Original comment by Erica K (Bitbucket: Erica K).
Jordan, I've had 100% failure up to now, but I pass your debug test (even without the larger values).
osrf-migration
commented
7 years ago Original comment by DouglasS (Bitbucket: DouglasS).
Hey folks, just wanted to chime in on what's happening from the IHMC side of things. At the very least, this definitely relates to the problem that @dan77062 is having.
The control algorithm for the robot is designed to run in an environment with a real-time kernel patch and core isolation. On the physical Val hardware this requires a quad core processor with hyper threading and turboboost disabled. We reserve Core 0 for IRQ/interrupt handling, cores 1 and 2 for our state estimation and control algorithm threads respectively, and core 4 for all other processes that have to run. Note that these are physical cores, not virtual cores, because hyper threading is problematic in a realtime environment which is why we disable it on the real robot. This is why we inspect the sysfs for the computer topology instead of relying on proc or top or the method that Dan used in his short Java program.
Obviously all of that isn't really needed in simulation because we run in lock-step with the physics. We're now tracking this issue as well: https://github.com/ihmcrobotics/ihmc-open-robotics-software/issues/99
We should have a fix for it and a release in the next few days.
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
It is likely that your running into the same problem that @dan77062 is, which is due to you not having enough physical cores on your processor. A way to verify that is to do what @dan77062 did above and launch the IHMC controller stuff separately and you should get this error,
#!bash
[ERROR] [1478556825.591343194, 21.328000000]: Exception in thread "main" java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
[ERROR] [1478556825.591761442, 21.328000000]: at java.util.ArrayList.rangeCheck(ArrayList.java:635)
[ERROR] [1478556825.592029007, 21.328000000]: at java.util.ArrayList.get(ArrayList.java:411)
[ERROR] [1478556825.592177665, 21.328000000]: at us.ihmc.affinity.Package.getCore(Package.java:45)
[ERROR] [1478556825.599444161, 21.328000000]: at us.ihmc.valkyrieRosControl.ValkyrieAffinity.<init>(ValkyrieAffinity.java:31)
[ERROR] [1478556825.600246263, 21.328000000]: at us.ihmc.valkyrieRosControl.ValkyrieRosControlController.<init>(ValkyrieRosControlController.java:107)Original comment by sringer99 (Bitbucket: sringer99).
I can get it to run 100% of the time at 8 and 4 seconds. Fails every time at 2 seconds with the same error as we have been getting. Note: I have the doubler on the delay script.
roslaunch val_gazebo val_sim.launch
... logging to /home/sringer/.ros/log/5a68a7cc-a5d1-11e6-bc2b-f832e4bba978/roslaunch-Mingo-Mtn-Robotics-24044.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://Mingo-Mtn-Robotics:43310/
SUMMARY
========
PARAMETERS
/robot_description: <?xml version="1....
/rosdistro: indigo
/rosversion: 1.11.20
NODES
/
control_py_Mingo_Mtn_Robotics_24044_828331878780894790 (robonet_tools/control)
controller_manager_Mingo_Mtn_Robotics_24044_2827938922632370286 (val_deploy/delayed_roslaunch.sh)
smtcore_Mingo_Mtn_Robotics_24044_2209162283755310182 (shared_memory_transport/smtcore)
auto-starting new master
process[master]: started with pid [24056]
ROS_MASTER_URI=http://localhost:11311
setting /run_id to 5a68a7cc-a5d1-11e6-bc2b-f832e4bba978
process[rosout-1]: started with pid [24069]
started core service [/rosout]
process[smtcore_Mingo_Mtn_Robotics_24044_2209162283755310182-2]: started with pid [24084]
process[control_py_Mingo_Mtn_Robotics_24044_828331878780894790-3]: started with pid [24087]
process[controller_manager_Mingo_Mtn_Robotics_24044_2827938922632370286-4]: started with pid [24088]
delaying: 1 seconds
Log file /home/sringer/.log already exists, proceeding...
Log file /home/sringer/.log already exists, proceeding...
starting: roslaunch val_deploy val_control_sim.launch controller_manager_looprate:=500 __name:=controller_manager_Mingo_Mtn_Robotics_24044_2827938922632370286
__log:=/home/sringer/.ros/log/5a68a7cc-a5d1-11e6-bc2b-f832e4bba978/controller_manager_Mingo_Mtn_Robotics_24044_2827938922632370286-4.log
... logging to /home/sringer/.ros/log/5a68a7cc-a5d1-11e6-bc2b-f832e4bba978/roslaunch-Mingo-Mtn-Robotics-24096.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.
started roslaunch server http://Mingo-Mtn-Robotics:40870/
SUMMARY
========
PARAMETERS
/rosdistro: indigo
/rosversion: 1.11.20
NODES
/
controller_manager_Mingo_Mtn_Robotics_24096_6578599215757810822 (val_controller_manager_rtt/controller_exec)
ROS_MASTER_URI=http://localhost:11311
core service [/rosout] found
process[controller_manager_Mingo_Mtn_Robotics_24096_6578599215757810822-1]: started with pid [24114]
exception subscribing to JointAPS_Angle_Rad : /pelvis/waist/JointAPS_Angle_Rad has not been registered.
exception subscribing to JointAPS_Vel_Radps : /pelvis/waist/JointAPS_Vel_Radps has not been registered.
exception subscribing to JointTorque_Des_Nm : /pelvis/waist/JointTorque_Des_Nm has not been registered.
exception subscribing to JointTorque_Meas_Nm : /pelvis/waist/JointTorque_Meas_Nm has not been registered.
exception subscribing to Position_Des_Rad : /pelvis/waist/Position_Des_Rad has not been registered.
exception subscribing to Velocity_Des_Radps : /pelvis/waist/Velocity_Des_Radps has not been registered.
2016-11-08 08:35:29 localhost ncl_cpp: [ERROR ] [gov.nasa.RobotInterface.addJointsAndActuators] Exception encountered while attempting to add an actuator! Check your URDF for correct XML formatting!
[ERROR] [1478622929.865724729]: Exception encountered while attempting to add an actuator! Check your URDF for correct XML formatting!
2016-11-08 08:35:29 localhost ncl_cpp: [ERROR ] [gov.nasa.HardwareInterface.makeHandle] Robot Hardware has failed Initialized properly.
[ERROR] [1478622929.866061262]: Fatal exception error encountered initializing RobotHardwareInterface, exiting
2016-11-08 08:35:29 localhost ncl_cpp: [FATAL ] [gov.nasa.ControllerExec.ORO_main] Error encountered during configure hook for controller_manager component!
================================================================================REQUIRED process
[controller_manager_Mingo_Mtn_Robotics_24096_6578599215757810822-1] has died!
process has died [pid 24114, exit code 1, cmd /opt/nasa/indigo/lib/val_controller_manager_rtt/controller_exec --rate 500 -s
name:=controller_manager_Mingo_Mtn_Robotics_24096_6578599215757810822 log:=/home/sringer/.ros/log/5a68a7cc-a5d1-11e6- bc2bf832e4bba978/controller_manager_Mingo_Mtn_Robotics_24096_6578599215757810822-1.log].
log file: /home/sringer/.ros/log/5a68a7cc-a5d1-11e6-bc2b-f832e4bba978/controller_manager_Mingo_Mtn_Robotics_24096_6578599215757810822-1*.log
Initiating shutdown!
================================================================================ [controller_manager_Mingo_Mtn_Robotics_24096_6578599215757810822-1] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
[controller_manager_Mingo_Mtn_Robotics_24044_2827938922632370286-4] process has finished cleanly
log file: /home/sringer/.ros/log/5a68a7cc-a5d1-11e6-bc2b-f832e4bba978/controller_manager_Mingo_Mtn_Robotics_24044_2827938922632370286-4*.log
^C[control_py_Mingo_Mtn_Robotics_24044_828331878780894790-3] killing on exit
[smtcore_Mingo_Mtn_Robotics_24044_2209162283755310182-2] killing on exit
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
Ok thanks @sringer99. I'm a bit confused as to why those delays need to be so long, because the processes that are being waited for are pretty quick to spin up. Only guess I have is it's something to do with roslaunch or maybe your computer in general? Anywho, i'll extend those wait times and work them into our next release. Thanks for taking the time to debug a bit!
osrf-migration
commented
7 years ago Original comment by sringer99 (Bitbucket: sringer99).
Here are the log files for my last fail
osrf-migration
commented
7 years ago Original comment by Rud Merriam (Bitbucket: rmerriam).
Jordan, could you make the delays arguments for the startup so those who don't have the problem can start the system more quickly?
osrf-migration
commented
7 years ago osrf-migration
commented
7 years ago Original comment by dan (Bitbucket: dan77062).
Douglas Stephen, Appreciate the help on this one and look forward to trying out the fixed version.
osrf-migration
commented
7 years ago Original comment by Erica Tiberia (Bitbucket: T_AL).
Memory: 11.6 GIB Processor Intel Core i5-6200U CPU@ 2.30GHz x4 OS-type 64-bit
I have increased the delay in (delayed_roslaunch.sh) file to 8 seconds and I commented out the lines to lower the harness and detach Val in the (init_robot.sh) file.
Everything loads properly and I can send commands to lower the harness and detach Val.
I can get the sensor data, camera etc. All the topics are there as you can see below
but any message that I publish to the ihmc_ros/valkyrie/control has no effect.
I have attached the image of rqt_graph showing all the nodes, it includes the published action for Val to move his right arm.
osrf-migration
commented
7 years ago Original comment by Jordan Lack (Bitbucket: jordanlack).
Erica could you post that as its own issue? @nmertins should probably see that as well
osrf-migration
commented
7 years ago Original comment by Nate Koenig (Bitbucket: Nathan Koenig).
osrf-migration
commented
7 years ago Original comment by Nate Koenig (Bitbucket: Nathan Koenig).
Issue #16 was marked as a duplicate of this issue.
Original report (archived issue) by dan (Bitbucket: dan77062).
The original report had attachments: master.log, roslaunch-Mingo-Mtn-Robotics-24044.log, control_py_Mingo_Mtn_Robotics_24044_828331878780894790-3-stdout.log, roslaunch-Mingo-Mtn-Robotics-24096.log, rosout.log, rosout-1-stdout.log
The controller manager crashes as it tries to initialize the hardware interface. This continues after updating with the fix for the $HOME/valkyrie directory missing issue (issue #4). See the log below.
I am wondering if it is a race condition. It will load when gazebo_gui seg faults (that happens about 20% of the time). Log of that follows the controller manager crash log.
Here is the log of the controller manager crashing:
Here the controller manager loads after gazebo_gui seg faults: