search

ros-industrial / motoman

ROS-Industrial Motoman support (http://wiki.ros.org/motoman)
146 stars 195 forks source link

Cannot move GP4 + YRC1000micro with ROS using `rostopic pub` or MoveIt + RViz configuration #472

Closed adamlouis closed 2 years ago

adamlouis commented 2 years ago

Hi! I am trying to get a new GP4 robot working with ROS + MoveIt using a YRC1000micro controller.

I can't get to the robot to move ... but I seem close. I would appreciate any help or debugging steps to get our robot moving.

We purchased our robot with the ROS package installed.

summary

Why can't I move the robot?

What simple test can I run to verify configuration of remote control of the GP4 over ROS with the YRC1000?

details

1. set up motoman driver

First, I followed the instructions here: https://wiki.ros.org/motoman_driver/Tutorials/Usage

I start roscore in terminal 1

terminal 1

roscore

then, I load the gp4 urdf & joint names

command terminal

roslaunch motoman_gp4_support load_gp4.launch
rosparam set controller_joint_names "[joint_1_s,joint_2_l,joint_3_u,joint_4_r,joint_5_b,joint_6_t]"

then, I run the driver for our controller w/ the robot's pre-set IP

terminal 2

roslaunch motoman_driver robot_interface_streaming_yrc1000.launch robot_ip:=192.168.1.31

2. run a moveit config

next, I run my move it configuration.

I adapted my configuration from this one in motoman_experimental: https://github.com/ros-industrial/motoman_experimental/tree/kinetic-devel/motoman_mpl80_moveit_config

My modifications were:

I forked the motoman_experimental repo and created a branch here with my motoman_gp4_moveit_config configuration: https://github.com/adamlouis/motoman_experimental/tree/motoman_gp4_moveit_config/motoman_gp4_moveit_config

I would be happy to clean that up & find a way to include it in motoman_experimental when I figure out what's wrong.

then, I run it with

terminal 3

roslaunch motoman_gp4_moveit_config demo.launch

at this point, if I move the robot using our teach pendant, the movement is reflected in Rviz.

this is good & tells me that I am able to read state from the yrc1000micro.

3. prepare for execution

now, I am ready to move the robot

command terminal

rosservice call /robot_enable "{}"

I hear relays click and see the servo lights turn on

output in command terminal 

success: True
message: "Motoman robot is now enabled and will accept motion commands."

output in terminal 2

Motoman robot is now enabled and will accept motion commands.

This is good - in theory, I can move the robot.

4. Try to move with RViz + MoveIt

I try to move the robot 2 ways & get the same behavior.

First, in Rviz + MoveIt, I set a goal position. I choose a goal position only small distance from the current position. I click "Plan" and "Execute".

On "Execute", I get the following output:

output in terminal 2 

Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.

output in terminal 3 

Controller is taking too long to execute trajectory (the expected upper bound for the trajectory execution was 2.526993 seconds). Stopping trajectory.

If I run the command to enable the service

command terminal

rosservice call /robot_enable "{}"

& repeat the above steps, I get the same error.

5. Try to move with rostopic

It's possible I wrote a bad moveit config, so I try to run some trivial command to move the robot with ROS.

The easiest thing seemed like: rostopic pub /joint_trajectory_action/goal control_msgs/FollowJointTrajectoryActionGoal. Is there some simpler command to test "write" commands to the yrc1000 over ROS?

This command fails with the same errors as above:

output in terminal 2 

Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.

output in terminal 3 

Controller is taking too long to execute trajectory (the expected upper bound for the trajectory execution was 2.526993 seconds). Stopping trajectory.

Here's my full command. The positions I provide are the robot's current positions.

rostopic pub /joint_trajectory_action/goal control_msgs/FollowJointTrajectoryActionGoal "header:
  seq: 0
  stamp:
    secs: 0
    nsecs: 0
  frame_id: ''
goal_id:
  stamp:
    secs: 0
    nsecs: 0
  id: ''
goal:
  trajectory:
    header:
      seq: 0
      stamp:
        secs: 0
        nsecs: 0
      frame_id: ''
    joint_names:
    - joint_1_s
    - joint_2_l
    - joint_3_u
    - joint_4_r
    - joint_5_b
    - joint_6_t
    points:
    - positions: [0.10212502628564835, 0.10033370554447174, 0.05954036861658096, 0.007940606214106083, -1.3190429210662842, 4.0864481925964355]
      velocities: [0,0,0,0,0,0]
      accelerations: [0,0,0,0,0,0,0]
      effort: [0,0,0,0,0,0]
      time_from_start: {secs: 0, nsecs: 0}
  path_tolerance:
  - {name: '', position: 0.0, velocity: 0.0, acceleration: 0.0}
  goal_tolerance:
  - {name: '', position: 0.0, velocity: 0.0, acceleration: 0.0}
  goal_time_tolerance: {secs: 0, nsecs: 0}"

Some notes:

[ERROR] [1647042622.182945880]: Failed to find topic_list parameter
[ WARN] [1647042622.190244041]: Failed to get '~port' parameter: using default (50242)
[ERROR] [1647042622.194533748]: Failed to find topic_list parameter
[ WARN] [1647042622.210121014]: Tried to connect when socket already in connected state

Thanks for reading & thanks in advance for any help! Would appreciate any pointers to help get us up an running.

marip8 commented 2 years ago

Thanks for the detailed issue explanation; I don't see any obvious issues with your approach or your MoveIt config.

MoveIt has some dynamic reconfigure parameters relating to trajectory execution that might be causing your issue. You might try turning off execution_duration_monitoring or increasing the allowed_execution_duration_scaling. Your planned trajectory should have time stamps associated with each waypoint (from the time parameterization stage of planning) and MoveIt does some monitoring to make sure that the trajectory execution on the controller actually completes within the amount of time the planned trajectory should have taken (with a scale factor applied to it). Disabling the execution_duration_monitoring circumvents this check, and increasing the allowed_execution_duration_scaling increases the scale factor for the trajectory time limit. Making these changes has resolved a similar issue for me recently, but I haven't delved into why this problem exists in the first place since my trajectory seemed to be time parameterized properly

image

gavanderhoorn commented 2 years ago

rostopic pub-ing to an action goal topic is not a good idea. Not just with motoman_driver, but in general.

I'd suggest using code similar to what is linked in #229.

I'm slightly confused as well: AFAIK and IIRC, goals with a JointTrajectory with a single point in them should not be accepted. motoman_driver absolutely needs two points: current state and destination point. The rostopic pub command line shown only includes a destination point.

Additionally, I'd suggest to use motoman_gp4_support which provides convenience .launch files which take care of everything wrt configuration of the driver for this particular robot variant.

Starting the driver then only needs:

roslaunch motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31
 controller:=yrc1000

nothing more is needed.

@adamlouis: please try again with just that robot_interface_streaming_gp4.launch file, nothing else.

And please start it using roslaunch --screen, and copy-paste the full output from the terminal into a comment here, after you've invoked robot_enable again and tried to execute a trajectory again (preferably using MoveIt or the action client script linked in #229, not rostopic).

I'd also suggest updating your ROSCONSOLE_FORMAT env var to '[${severity}] [${time}] [${node}] [${logger}]: ${message}', so it's clear which node printed which line.

Edit: and don't use demo.launch to control a real robot.

The MoveIt configuration you've shared includes a moveit_planning_execution.launch. That would be the proper entrypoint normally. I'm not sure how you've changed it, so it could be it's not in working order

gavanderhoorn commented 2 years ago

@marip8 wrote:

MoveIt has some dynamic reconfigure parameters relating to trajectory execution that might be causing your issue. [...] Making these changes has resolved a similar issue for me recently, but I haven't delved into why this problem exists in the first place since my trajectory seemed to be time parameterized properly

If that was with motoman_driver, something is wrong, as that should not be necessary.

Unless perhaps the joint limits for the specific robot were not configured correctly and whatever parameterised the trajectory is expecting the robot to be able to move faster than it really can.

marip8 commented 2 years ago

If that was with motoman_driver, something is wrong, as that should not be necessary.

Agreed; I'm going to work with the robot more this week, so hopefully I can figure out what is going on. The fix I mentioned might at least get the robot moving in case there are other parameterization/joint velocity limit issues that are MoveIt/URDF related.

adamlouis commented 2 years ago

Thank you @gavanderhoorn & @marip8 for you quick replies and suggestions!

Here's what I've done & observed:

1. MoveIt duration params

You might try turning off execution_duration_monitoring or increasing the allowed_execution_duration_scaling

I did this by adding these 2 lines to my MoveIt launch file:

<param name="move_group/trajectory_execution/allowed_execution_duration_scaling" value="4.0" />
<param name="move_group/trajectory_execution/execution_duration_monitoring" value="false" />

When I do this, I do not get a timeout error, but the robot does not move.

2. use motoman_gp4_support launch files

Additionally, I'd suggest to use motoman_gp4_support which provides convenience .launch files which take care of everything wrt configuration of the driver for this particular robot variant.

Starting the driver then only needs:

roslaunch motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31 controller:=yrc1000

nothing more is needed.

done

  1. update console format

I'd also suggest updating your ROSCONSOLE_FORMAT env var to '[${severity}] [${time}] [${node}] [${logger}]: ${message}', so it's clear which node printed which line.

done

4. MoveIt config changes

Edit: and don't use demo.launch to control a real robot. The MoveIt configuration you've shared includes a moveit_planning_execution.launch. That would be the proper entrypoint normally. I'm not sure how you've changed it, so it could be it's not in working order

Fair point - I will rename this to something appropriate moving forward.

Likewise, because my MoveIt configuration might be bad, I will use your simple_trajectory_action_client.py script for further tests.

5. Run with motoman_gp4_support launch files + your script

And please start it using roslaunch --screen, and copy-paste the full output from the terminal into a comment here....

Below are the outputs of the 2 terminals - the ROS driver terminal and the simple_trajectory_action_client.py terminal.

I updated the joint names in simple_trajectory_action_client.py to match the joint names of my robot.

Likewise, I ran the script several times with small modifications (e.g. move a different joint a different amount) but none worked.

Strangely, the Python script exits with success, but with the ROS log shows the same error line,

[ERROR] [1647298434.963551694] [/motion_streaming_interface] [ros.motoman_driver]: Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry.

At the bottom, it shows the line:

[ INFO] [1647298439.967259077] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action.controllerStateCB]: Inside goal constraints - stopped moving- return success for action

Which is strange, due to 1) the prior error and 2) the fact that the robot did not move

terminal 1: roslaunch --screen motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31 controller:=yrc1000

Click to expand ``` $ roslaunch --screen motoman_gp4_support robot_interface_streaming_gp4.launch robot_ip:=192.168.1.31 controller:=yrc1000 ... logging to /home/adam/.ros/log/6b59d292-a3e9-11ec-af51-5d9e0201ff92/roslaunch-adam-ubuntu-53282.log Checking log directory for disk usage. This may take a while. Press Ctrl-C to interrupt Done checking log file disk usage. Usage is <1GB. started roslaunch server http://adam-ubuntu:43285/ SUMMARY ======== PARAMETERS * /controller_joint_names: ['joint_1_s', 'jo... * /robot_ip_address: 192.168.1.31 * /rosdistro: noetic * /rosversion: 1.15.14 NODES / io_relay (motoman_driver/io_relay) joint_state (motoman_driver/robot_state) joint_trajectory_action (industrial_robot_client/joint_trajectory_action) motion_streaming_interface (motoman_driver/motion_streaming_interface) auto-starting new master process[master]: started with pid [53321] ROS_MASTER_URI=http://localhost:11311 setting /run_id to 6b59d292-a3e9-11ec-af51-5d9e0201ff92 process[rosout-1]: started with pid [53361] started core service [/rosout] process[joint_state-2]: started with pid [53364] process[motion_streaming_interface-3]: started with pid [53368] process[io_relay-4]: started with pid [53370] process[joint_trajectory_action-5]: started with pid [53371] [ INFO] [1647298347.184634433] [/joint_state] [ros.simple_message]: Added message handler for message type: 0 [ INFO] [1647298347.187798055] [/joint_state] [ros.motoman_driver]: Robot state connecting to IP address: '192.168.1.31:50241' [ERROR] [1647298347.188439160] [/joint_state] [ros.motoman_driver]: Failed to find topic_list parameter [ INFO] [1647298347.190032841] [/joint_state] [ros.industrial_utils]: Adding joint_1_s to list parameter [ INFO] [1647298347.190067773] [/joint_state] [ros.industrial_utils]: Adding joint_2_l to list parameter [ INFO] [1647298347.190083084] [/joint_state] [ros.industrial_utils]: Adding joint_3_u to list parameter [ INFO] [1647298347.190094583] [/joint_state] [ros.industrial_utils]: Adding joint_4_r to list parameter [ INFO] [1647298347.190105306] [/joint_state] [ros.industrial_utils]: Adding joint_5_b to list parameter [ INFO] [1647298347.190116246] [/joint_state] [ros.industrial_utils]: Adding joint_6_t to list parameter [ INFO] [1647298347.190140721] [/joint_state] [ros.industrial_utils]: Found user-specified joint names in 'controller_joint_names': [joint_1_s, joint_2_l, joint_3_u, joint_4_r, joint_5_b, joint_6_t] [ INFO] [1647298347.190607239] [/joint_state] [ros.simple_message]: Connected to server [ INFO] [1647298347.190642770] [/joint_state] [ros.simple_message]: Initializing message manager with default comms fault handler [ INFO] [1647298347.190663735] [/joint_state] [ros.simple_message]: Default communications fault handler successfully initialized [ INFO] [1647298347.190680806] [/joint_state] [ros.simple_message]: Initializing message manager [ INFO] [1647298347.190701048] [/joint_state] [ros.simple_message]: Added message handler for message type: 1 [ INFO] [1647298347.193239179] [/joint_state] [ros.simple_message]: Added message handler for message type: 10 [ INFO] [1647298347.193701622] [/joint_state] [ros.simple_message]: Added message handler for message type: 15 [ INFO] [1647298347.193936324] [/motion_streaming_interface] [ros.motoman_driver]: Joint Trajectory Interface connecting to IP address: '192.168.1.31:50240' [ WARN] [1647298347.194079574] [/io_relay] [ros.motoman_driver.io.init]: Failed to get '~port' parameter: using default (50242) [ INFO] [1647298347.194338138] [/joint_state] [ros.simple_message]: Entering message manager spin loop [ERROR] [1647298347.195537643] [/motion_streaming_interface] [ros.motoman_driver]: Failed to find topic_list parameter [ INFO] [1647298347.196025652] [/io_relay] [ros.simple_message]: Connected to server [ INFO] [1647298347.196308786] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_1_s to list parameter [ INFO] [1647298347.196335098] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_2_l to list parameter [ INFO] [1647298347.196351681] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_3_u to list parameter [ INFO] [1647298347.196366141] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_4_r to list parameter [ INFO] [1647298347.196383065] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_5_b to list parameter [ INFO] [1647298347.196397435] [/motion_streaming_interface] [ros.industrial_utils]: Adding joint_6_t to list parameter [ INFO] [1647298347.196418679] [/motion_streaming_interface] [ros.industrial_utils]: Found user-specified joint names in 'controller_joint_names': [joint_1_s, joint_2_l, joint_3_u, joint_4_r, joint_5_b, joint_6_t] [ INFO] [1647298347.196436590] [/motion_streaming_interface] [ros.motoman_driver]: MotomanJointTrajectoryStreamer: init [ INFO] [1647298347.196451851] [/motion_streaming_interface] [ros.motoman_driver]: JointTrajectoryStreamer: init [ INFO] [1647298347.196746734] [/motion_streaming_interface] [ros.simple_message]: Connected to server [ WARN] [1647298347.197159913] [/motion_streaming_interface] [ros.motoman_driver]: Unable to read velocity limits from 'robot_description' param. Velocity validation disabled. [ INFO] [1647298347.199751812] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_1_s to list parameter [ INFO] [1647298347.200548324] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_2_l to list parameter [ INFO] [1647298347.200565047] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_3_u to list parameter [ INFO] [1647298347.200576109] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_4_r to list parameter [ INFO] [1647298347.200586007] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_5_b to list parameter [ INFO] [1647298347.200595394] [/joint_trajectory_action] [ros.industrial_utils]: Adding joint_6_t to list parameter [ INFO] [1647298347.200612311] [/joint_trajectory_action] [ros.industrial_utils]: Found user-specified joint names in 'controller_joint_names': [joint_1_s, joint_2_l, joint_3_u, joint_4_r, joint_5_b, joint_6_t] [ INFO] [1647298347.200633131] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Filtered joint names to 6 joints [ INFO] [1647298347.202854303] [/motion_streaming_interface] [ros.motoman_driver]: Unlocking mutex [ INFO] [1647298347.202914524] [/motion_streaming_interface] [ros.motoman_driver]: Starting Motoman joint trajectory streamer thread [ INFO] [1647298347.208008060] [/motion_streaming_interface] [ros.motoman_driver]: Connecting to robot motion server [ WARN] [1647298347.208033019] [/motion_streaming_interface] [ros.simple_message]: Tried to connect when socket already in connected state [ WARN] [1647298388.531971119] [/motion_streaming_interface] [ros.motoman_driver]: Motoman robot is now enabled and will accept motion commands. [ INFO] [1647298434.959602612] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Received new goal [ INFO] [1647298434.959656093] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Publishing trajectory [ INFO] [1647298434.959967726] [/motion_streaming_interface] [ros.motoman_driver]: Receiving joint trajectory message [ERROR] [1647298434.963551694] [/motion_streaming_interface] [ros.motoman_driver]: Failed to initialize MotoRos motion, trajectory execution ABORTED. If safe, call the 'robot_enable' service to (re-)enable Motoplus motion and retry. [ INFO] [1647298434.975282255] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.001286794] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.028803686] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.055601711] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.082140510] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.108650533] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.135041384] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.162159451] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.188905900] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.216550190] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.242780508] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.268879837] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.295539047] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.322248321] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.349102237] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.376684058] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.403537282] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.430806033] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.456939815] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.484267806] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.511258691] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.537291667] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.564528661] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.591329756] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.617849671] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.644477661] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.671872648] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.698306085] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.725477130] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.752301939] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.779548415] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.805874489] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.833276650] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.860251607] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.887337118] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.913151404] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.939842989] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.967217206] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298435.994118431] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.021437349] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.048077848] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.074274566] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.101025607] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.128105722] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.154253643] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.180386932] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.206255243] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.232766980] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.259924718] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.286555589] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.313560552] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.340344600] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.367745929] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.393934041] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.421619893] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.448299819] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.475355254] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.501351084] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.528121664] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.554262449] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.580262202] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.606431521] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.632564639] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.658484650] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.685035169] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.712117584] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.738450885] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.764850187] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.791476241] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.817691037] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.844245325] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.870302000] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.897187885] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.923694338] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.950168063] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298436.976419138] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.003536242] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.031253847] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.057672779] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.084606732] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.109900948] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.137142298] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.163803615] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.190628417] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.217124848] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.244268124] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.271487790] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.297850455] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.323929271] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.350128199] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.376166205] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.402366173] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.429401302] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.455745401] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.482037620] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.509243217] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.535677997] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.561533255] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.587994363] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.614257397] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.640767418] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.667777953] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.694034761] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.720785926] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.748370154] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.775601206] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.802566869] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.828742146] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.855879166] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.882397609] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.908750815] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.935197625] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.960852864] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298437.989396021] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.015669117] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.042167842] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.069136018] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.096235973] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.123296425] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.149574600] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.176174498] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.202080652] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.228797080] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.255543647] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.281334397] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.308175983] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.334267086] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.360369185] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.387260913] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.413880862] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.440918704] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.468138096] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.495743115] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.521706496] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.549170295] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.575519758] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.601535793] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.628312172] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.655258645] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.681601756] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.708356643] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.735632712] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.761657324] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.788209057] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.814598284] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.841213200] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.868219782] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.894256828] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.920613671] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.946138273] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298438.973239237] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.000421539] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.030329479] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.056596942] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.084242021] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.111413581] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.138029332] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.165591482] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.192382409] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.219608162] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.246233770] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.273145049] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.300270665] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.326470285] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.353417242] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.379720990] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.407385266] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.433316636] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.460897226] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.487847045] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.515507703] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.541349512] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.567955305] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.594453186] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.620919609] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.647749614] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.675298149] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.701256165] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.727695760] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.754164699] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.781093216] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.807805481] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.834129718] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.860679536] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.887384559] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.914229354] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.939980983] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action]: Waiting to check for goal completion until halfway through trajectory [ INFO] [1647298439.967259077] [/joint_trajectory_action] [ros.industrial_robot_client.joint_trajectory_action.controllerStateCB]: Inside goal constraints - stopped moving- return success for action ```

terminal 2: python3 simple_trajectory_action_client.py

[INFO] [1647298434.646652] [/simple_trajectory_action_client] [rosout]: Waiting for driver's action server to become available ..
[INFO] [1647298434.928927] [/simple_trajectory_action_client] [rosout]: Connected to trajectory action server
[INFO] [1647298434.958179] [/simple_trajectory_action_client] [rosout]: Submitting goal ..
[INFO] [1647298434.959258] [/simple_trajectory_action_client] [rosout]: Waiting for completion ..
[INFO] [1647298439.968406] [/simple_trajectory_action_client] [rosout]: Done.

Thanks again for your help!

gavanderhoorn commented 2 years ago

I'm starting to suspect your YRC1000 is somehow not configured correctly.

Could you do one more test with the action client script, and make a wireshark trace of the network traffic between the YRC1000 and your ROS PC at the same time? Then please attach it here (make sure to compress it).

Additionally:

  1. could you tell us which INFORM job you've installed from here (there are a couple, and it matters)
  2. could you make a CMOS backup, zip it and attach it here?
ted-miller commented 2 years ago

Please also send that CMOS to the Partner Support team in the separate other email chain. They can check to ensure there isn't anything configured that would prevent robot motion.

gavanderhoorn commented 2 years ago

There is a parallel support effort, next to this thread?

@adamlouis: could you please let us know when that has reached a conclusion?

I'll close this one for now, as I don't like potentially duplicating effort.

ted-miller commented 2 years ago

@gavanderhoorn: I asked him to open this issue because I didn't see anything wrong with the description of his procedure. It appears that the robot-side is working ok. I figured there was something wrong in the support package that I'm not aware of.

I just wanted to have our support group check to make sure there isn't some safety-circuit configured improperly or something.

adamlouis commented 2 years ago

Thanks all! I will work on this later today.

I'm starting to suspect your YRC1000 is somehow not configured correctly.

Could you do one more test with the action client script, and make a wireshark trace of the network traffic between the YRC1000 and your ROS PC at the same time? Then please attach it here (make sure to compress it).

Additionally:

could you tell us which INFORM job you've installed from here (there are a couple, and it matters) could you make a CMOS backup, zip it and attach it here?

I will do this after I send the zip to Yaskawa/Motoman partner support & hear back

Please also send that CMOS to the Partner Support team in the separate other email chain. They can check to ensure there isn't anything configured that would prevent robot motion.

Will do later today

There is a parallel support effort, next to this thread?

@adamlouis: could you please let us know when that has reached a conclusion?

I'll close this one for now, as I don't like potentially duplicating effort.

As Ted mentioned - I emailed our contacts at Yaskawa. Over that email, I was asked to open the GitHub issue here. Later, I was asked for the same .zip you requested. I will send my zip to partner support & come back here if needed (or to report a resolution).

FYI - I have been able to get our robot up & running by programming against the YRC1000 ethernet interface. I suspect I will be able to do our initial task with just the ethernet interface. In any case, I would like to figure the ROS configuration out to support more sophisticated use cases as they emerge.

Thanks again!

gavanderhoorn commented 2 years ago

@ted-miller wrote:

@gavanderhoorn: I asked him to open this issue because I didn't see anything wrong with the description of his procedure. It appears that the robot-side is working ok. I figured there was something wrong in the support package that I'm not aware of.

I just wanted to have our support group check to make sure there isn't some safety-circuit configured improperly or something.

This all makes complete sense, but please ask people to mention that in the initial post.

Especially when things like the CMOS settings haven't been checked yet, as that is almost impossible to diagnose from "the ROS side". Knowing that's being looked at would change the approach to diagnose what's going on.

@adamlouis wrote:

FYI - I have been able to get our robot up & running by programming against the YRC1000 ethernet interface. I suspect I will be able to do our initial task with just the ethernet interface. In any case, I would like to figure the ROS configuration out to support more sophisticated use cases as they emerge.

I would suggest to still create the wireshark trace.

It should not take very long, and there is a good chance we'll be able to diagnose what's going on from that -- provided the CMOS backup shows no problems with the configuration of the YRC1000u itself.

gavanderhoorn commented 2 years ago

@adamlouis: let us know whether you'll still create the Wireshark trace and we'll re-open.

ted-miller commented 2 years ago

The support guys found that the Inform job was incorrect. (Sorry, I should have caught that.)

image

For anyone experiencing similar issue in the future, please use the standard INIT_ROS jobs posted to GitHub. The motion API command will only work if the control-group is being blocked by a WAIT command. This is a safety mechanism to prevent conflicting motion from multiple points of control.

gavanderhoorn commented 2 years ago

How did that happen?

Was the job not the one we host here?

adamlouis commented 2 years ago

There were 2 issues. The first is resolved. For the second, I have a work around (or perhaps this is how it must be done).

1. error in INIT_ROS code

We purchased our GP4 / YRC1000 / SMart Pendant with ROS pre-configured. There was already a INIT_ROS job present when we received it.

When I sent the back up .zip file of our YRC1000 to the Motoman partner support team, they noticed the error Ted mentioned above.

I edited the job to match what they provided in the email, which matches the code here.

I (or anyone here) did not deliberately edit the INIT_ROS job after we received our shipment. Though, of course it is possible I did so in error / negligence.

2. INIT_ROS does not begin execution and block on a WAIT command except when run from "PLAY" mode

After editing the job to match the code referenced above, calling rosservice call /robot_enable "{}" when in REMOTE mode on the Smart Pendant did not advance the Job to the wait instruction.

Rather, I see "INIT_ROS" as the "Current Job" on the smart pendant home screen, but when I view the "Current Job" page, the job is at the "Start Job" instruction.

If I turn the pendant key to PLAY mode and run the INIT_ROS job manually from the pendant touch screen, the job executes up until the wait instruction.

Then, if I switch the pedant to REMOTE remote mode, run rosserivce call /robot_enable "{}", and move the robot in RViz / MoveIt with the config referenced in my initial post, the robot moves as expected.

Thank you all for you support & quick replies while figuring this out!

adamlouis commented 2 years ago

I realized point 2 in my post above is true when starting other custom pendant jobs with the ethernet interface, not just when starting INIT_ROS with ROS. I had not observed this behavior when starting custom jobs over ethernet earlier.

It seems I must first go to PLAY mode before starting jobs in REMOTE mode.

In either case, I think I am unblocked & can figure this out.

Thanks all again for your help & for your work on this repo!

gavanderhoorn commented 2 years ago

You should be able to start everything in REMOTE mode.

Something does not seem to be right still.

ted-miller commented 2 years ago

We purchased our GP4 / YRC1000 / SMart Pendant with ROS pre-configured. There was already a INIT_ROS job present when we received it.

This sounds like the error came from our manufacturing area. I'll look into it.

It seems I must first go to PLAY mode before starting jobs in REMOTE mode.

This is not normal. Could you please use Wireshark to record the network traffic when you attempt robot_enable while in remote mode?

adamlouis commented 2 years ago

Thanks for the feedback!

I will try to post the wireshark output by EOD tomorrow.

Likewise, I will let you know if we figure out a more minimal set of steps reproduce the behavior.

gavanderhoorn commented 2 years ago

Friendly ping @adamlouis

adamlouis commented 2 years ago

@gavanderhoorn & @ted-miller - I've attached a zip of my wireshark traffic here. There was a fair amount of data constantly passing between my compute & the yrc1000, but the capture represents a 5-10s window during which I ran rosservice call /robot_enable "{}". The YRC is 192.168.1.31 and my computer is 192.168.1.32. Let me know if this is what you were looking for.

I can confirm that my ros terminal printed [ros.motoman_driver]: Motoman robot is now enabled and will accept motion commands. when I issued this command and the robot would not move via rviz/moveit after I did this.

I have consistently reproduced the behavior I described before & have a few more details. My flow is:

The behavior above is true of other custom jobs I wrote and not just INIT_ROS. If I use the START command on the ASCII ethernet protocol of the yrc1000 to start my own pendant jobs, I observe the same behavior that those jobs do not start until I toggle the smart pendant to PLAY mode. Again, I don't need to do anything in PLAY mode - simply toggling there & back is enough to make remote invocation of jobs work as expected.

This seems to indicate the issue is with something on the YRC / pendant side vs. on the ROS driver side given that starting jobs in REMOTE mode fails similarly even when not using any of the code in this repo.

Thanks again! Let me know if you have any other experiments / evidence you'd like me to capture.

rosservice-enable.zip

gavanderhoorn commented 2 years ago

Seems like MotoROS considers starting INIT_ROS a success:

ROS-Industrial SimpleMessage, Motoman Motion Reply (0x7d2), 76 bytes, little-endian
    Prefix
        Packet Length: 72
    Header
        Message Type: Motoman Motion Reply (0x000007d2)
        Communications Type: Service Reply (3)
        Reply Code: Success (1)
    Body
        Robot ID: -1
        Sequence Number: -1
        Command: Motion_Ctrl (2001)
        Result: Success/True (0)
        Subcode: Unknown (0)
        Data (reserved)
            J0:    0,000000000
            J1:    0,000000000
            J2:    0,000000000
            J3:    0,000000000
            J4:    0,000000000
            J5:    0,000000000
            J6:    0,000000000
            J7:    0,000000000
            J8:    0,000000000
            J9:    0,000000000

does imply there is something not entirely correct on the YRC1000 side.

the capture represents a 5-10s window during which I ran rosservice call /robot_enable "{}".

Could have been informative if you'd included the traffic when you're attempting to execute a trajectory, but I'm sort-of expecting to just see acceptance of traj pts until the buffer is full and then nothing.

EricMarcil commented 2 years ago

@adamlouis Can you check that this installation step was done properly:

Controller Configuration

Verify that the controller is enabled to receive remote commands in Remote mode:

Using standard pendant:

Rotate the pendant key-switch (upper left of pendant) fully counter-clockwise into TEACH mode.

Upgrade your Security Level to MANAGEMENT

Touching [System Info]→[Security].

Default password is all 9's. Using the pendant, select [In/OUT]→[PSEUDO INPUT SIG]

For DX100, DX200, and FS100, set input #82015 (CMD REMOTE SEL) to ON. The signal number is #87015 on YRC.

Move the cursor to the circle beside the specified input Press [INTERLOCK]+[SELECT] to turn on the input

Rotate the pendant key-switch (upper left of pendant) fully clockwise into REMOTE mode.

EricMarcil commented 2 years ago

@adamlouis Also check the Operate Enable Settings:

Upgrade your Security Level to MANAGEMENT (as above) Touching [SETUP]→[OPERATE ENABLE]. Make sure that all the operations are set to PERMIT

ted-miller commented 2 years ago

@adamlouis The wireshark indicates that the robot does briefly become "ready", but then becomes "not ready" 124 ms after the robot_enable service is called. I think there is still something wrong with the robot job.

Could you please save out a fresh copy of the following files and attach them here. Be sure to zip them; they compress extremely well.

INIT_ROS.JBI PANELBOX.LOG ALL.PRM CMOS.BIN

adamlouis commented 2 years ago

Could you please save out a fresh copy of the following files...

done!

Using the pendant, select [In/OUT]→[PSEUDO INPUT SIG] For DX100, DX200, and FS100, set input #82015 (CMD REMOTE SEL) to ON. The signal number is #87015 on YRC. Move the cursor to the circle beside the specified input Press [INTERLOCK]+[SELECT] to turn on the input

Touching [SETUP]→[OPERATE ENABLE]. Make sure that all the operations are set to PERMIT

@EricMarcil -- I referenced this in my initial post & the private email thread, but I am using a smart pendant and not a standard pendant. I have not found PSUEDO INPUT SIG, CMD REMOTE SEL, INTERLOCK, or SETUP on the smart pendant and have not seen a value of 87015 (or values w/ this order of magnitude) on any IO screen.

Do you have documentation to accomplish this with the smart pendant rather than the standard pendant?

Are you able to read the necessary data from the system backup rather than the pendant UI?

For my own reference -- are the smart pendant & standard pendant two clients of the same controller core, or do the two pendants require different controller configuration?

Thank you!

YC1000-0322-GitHub.zip

gavanderhoorn commented 2 years ago

It's (almost certainly) not the cause of the problem, but I notice you're running MotoROS v1.9.0.

That's rather old, as the current version would be v1.9.10.

You might want to upgrade. The binaries can be found here for the YRC1000u.

ted-miller commented 2 years ago

I have not found PSUEDO INPUT SIG, CMD REMOTE SEL, INTERLOCK, or SETUP on the smart pendant and have not seen a value of 87015 (or values w/ this order of magnitude) on any IO screen. Do you have documentation to accomplish this with the smart pendant rather than the standard pendant?

This step is not required for the Smart Pendant. The signal must already be enabled for the SP to work. I have updated the installation instructions to hopefully make this clearer.

For my own reference -- are the smart pendant & standard pendant two clients of the same controller core, or do the two pendants require different controller configuration?

They are two clients of the same controller core.

Are you able to read the necessary data from the system backup rather than the pendant UI?

Yes. I have found the problem. The job is set to "step" mode where it only executes one step at a time. (I didn't know this was a thing, nor can I explain why it behaves differently in PLAY mode.)

Smart Pendant instructions: (The installation wiki has been updated with instructions for both pendant types.)

  1. Rotate the pendant key-switch (top of pendant) to the middle position for PLAY mode.

  2. Touch [MENU]→[Job List]

  3. Select the INIT_ROS job from the list and touch the [RUN] button in the lower left.

  4. In the ADVANCED RUN pane, change the Mode to One Cycle.

  5. Rotate the pendant key-switch (top of pendant) fully clockwise into REMOTE mode.

ted-miller commented 2 years ago

@adamlouis: I just learned some new information about this issue. The instructions above are not sufficient long-term. It will automatically switch back to STEP mode when you toggle the key switch.

Touch MENU > System Settings > Classic Interface. Start the Classic Interface and press the Connect button. image

From the main menu, touch Setup > Operate Cond. Cursor down to the Cycle Switch In xxxx Mode fields. Change all of these to CYCLE. image

The touch the power-symbol button in the top right to return back to the Smart Pendant interface.

gavanderhoorn commented 2 years ago

@adamlouis: have you had a chance to figure this out? The last comment(s) by @ted-miller should help with the STEP mode problem.

gavanderhoorn commented 2 years ago

Assuming this has been fixed, I'm going to close the issue.

Please let us know your current status @adamlouis.