Getting 'Connection Refused' when sending joint positions through MoveIt

[hugols16]

I configured the new FANUC LR Mate 200iD with the required Karel and TP programs, and am able to run the program and connect to it using the experimental files from the ROS-Industrial fauna repository. When running roslaunch fanuc_lrmate200id_moveit_config moveit_planning_execution.launch sim:=false robot_ip:=IP_OF_ROBOT Rviz opens with the proper configuration and I can echo the joint positions using rostopic. However, when I plan and execute a path through MoveIt, I get

[ERROR] [1494438486.514967599]: Failed to connect to server, rc: -1. Error: 'Connection refused' (errno: 111)
[ERROR] [1494438486.772917317]: Failed to connect to server, rc: -1. Error: 'Connection refused' (errno: 111)
[ERROR] [1494438487.045197334]: Failed to connect to server, rc: -1. Error: 'Connection refused' (errno: 111)
[ERROR] [1494438487.303179468]: Failed to connect to server, rc: -1. Error: 'Connection refused' (errno: 111)
[ERROR] [1494438487.553366648]: Timeout connecting to robot controller.  Send new motion command to retry.
[ERROR] [1494438490.036180596]: Controller is taking too long to execute trajectory (the expected upper bound for the trajectory execution was 3.830259 seconds). Stopping trajectory.

Not sure what I'm doing wrong and if anything is missing in the setup. Thanks in advance for the help!

[gavanderhoorn]

Please tell me exactly which program(s) you have started on the robot (using the teach pendant).

[hugols16]

Hi,

The Connection refused error seems to be happening at random times. I am running the ROS program from the pendant with it being in the OFF position and the controller being turned to AUTO. When it actually does connect, I still get an error saying

[ERROR] [1494445041.600937790]: Controller is taking too long to execute trajectory (the expected upper bound for the trajectory execution was 31.209054 seconds). Stopping trajectory.

I followed the recommendations in the Troubleshooting section saying to disable the execution_duration_monitoring parameter and increase the allowed_execution_duration_scaling to no luck

[gavanderhoorn]

Is this a real system or a Roboguide instance?

re: trajectory execution monitor: if you disabled it, it should be impossible for it to print those messages. Can you check with dynamic_reconfigure (use rqt_reconfigure fi) and make sure it's really disabled?

[gavanderhoorn]

And if it's 'random' then this is going to be hard to diagnose for me from a distance.

If you are trying to run this in AUTO: make sure to reselect the ROS program every time you want to (re)start it.

If you don't, the controller might actually not be starting all programs involved. I've seen that happen before, leading to fi only ROS_STATE having been started. That would explain what you are seeing.

You can check on the web page of your controller under "program states" whether the ROS, ROS_STATE and ROS_RELAY programs are all running.

[hugols16]

That's a good point, the programs may not have all been started. I'll try to notice when it's happening and see if the other programs are also running. As for the Trajectory timeout, the new option was entered wrong, which allowed for the timeout to still happen. However, if the timeout is disabled, the rosnode is stuck in a state where it's trying to send the joint motion commands to the arm and the arm isn't responding... No timeout is issued because of the above mention, but this remedy is definitely not solving the problem

[gavanderhoorn]

However, if the timeout is disabled, the rosnode is stuck in a state where it's trying to send the joint motion commands to the arm and the arm isn't responding...

well, yes, that is the consequence of disabling the timeout.

No timeout is issued because of the above mention, but this remedy is definitely not solving the problem

no, I never said it would.

I'm pretty sure you'll find that ROS_RELAY is just not running when you get the "connection refused" errors.

[hugols16]

So assuming that you followed the tutorial steps, assigned the proper values to the proper variables, ran the robot_state_visualize node and could job the robot and see the updated configuration in RViz, and still hit the Controller is taking too long to execute trajectory error, what is there to do? Most of the work so far has been abstracted which explains why it's so hard to debug and work through it. I'm unsure if the problem is on the arm's side, the controller side or my launching files...

[gavanderhoorn]

what is there to do

I'm not sure what you're asking: the driver is not able to make the robot move in the way MoveIt wants it to (this is a Fanuc limitation, not really something we can do too much about), so you'll always run into the trajectory execution monitor trying to stop your trajectory execution. The only 'remedy' is to either disable it or change the scaling such that it is still enabled but just allows for a lot more time.

Changing the limit and / or disabling it should really be a matter of updating the trajectory_execution.launch.xml file. If that didn't work for you then that is something to try and figure out.

If you're referring to the ROS_RELAY program not running, I'd suggest first confirming that that is actually the case next time you run into the "connection refused" problem. Then we can start diagnosing that again.

[hugols16]

The Connection Refused appears when the ROS program goes to the PAUSE state. ROSSTATE and ROS_RELAY are still in the RUNNING state.

[gavanderhoorn]

Ok. That seems strange, but I've seen stranger things.

I'll see if I can reproduce this here locally.

Whenever it happens, can you try the following:

nc IP_OF_ROBOT 11000

and see what the output is?

[gavanderhoorn]

Btw: ROS going into PAUSEd seems to indicate that something triggered an event that caused that.

How are you starting the programs? Using the cycle start button or just using the TP?

[hugols16]

I'm not sure I follow your direction regarding nc IP_OF_ROBOT 11000. Once again I've previously played with the ur_driver, and build my own custom one, this is the very first time using any FANUC product, let alone this driver.

Also, the procedure for starting the program is to set the pendant to OFF, set the controller to AUTO and then press the start button (green button) on the controller.

[hugols16]

As for the causes of it happening, I noticed it happening when I try sending a joint command and get an error (related to Trajectory splicing not yet implemented, stopping current motion. although I did some research and I'm not sending the same position as the current one, so I'm unsure why it would happen, could be the topic of another issue). After that, if I cancel the ros nodes, and then try to reconnect, I will get the Connection Refused error and the ROS program on the arm itself will be paused.

[gavanderhoorn]

@hugols16 wrote:

Also, the procedure for starting the program is to set the pendant to OFF, set the controller to AUTO and then press the start button (green button) on the controller.

then I assume that you SELECT the ROS program before turning of the TP, correct?

Btw: running things in AUTO without any prior experience with the program or the robot is not something I'd recommend. Personally I always make sure to at least get a couple of rounds of program execution to fully understand what is going on.

As for the causes of it happening, I noticed it happening when I try sending a joint command and get an error (related to Trajectory splicing not yet implemented, stopping current motion. although I did some research and I'm not sending the same position as the current one

"trajectory splicing" is terminology for trying to send a new trajectory without waiting for the previous one to have completed with the intend of making them smoothly blend into each other. That is not supported by the industrial_robot_client nodes. Are you sure you're not sending multiple trajectories to the driver?

After that, if I cancel the ros nodes, and then try to reconnect, I will get the Connection Refused error and the ROS program on the arm itself will be paused.

with "cancel the ROS nodes", I assume you mean on the ROS (ie: your PC) side?

At this point I can only guess what is going on, as I don't have enough information.

Could you please give me an exact description (ie: step-by-step) of what you are doing? Both on the ROS side as well as on the controller/TP side?

this is the very first time using any FANUC product

can I very respectfully suggest that you try and get some training? From someone with some experience with Fanuc robots? It's not all terribly difficult, but these are dangerous machines and treating all my stuff as black boxes while copying magic incantations into your terminal is not a good way to approach this.

[gavanderhoorn]

[..] ROS program on the arm itself will be paused.

Slightly pedantic perhaps, but 'the arm' does not run any programs. Only the controller does that. The manipulator is basically a bunch of steel, wires, gears and motors that is completely controlled by the controller.

[hugols16]

Hi,

I was running the program in AUTO simply because of the suggestions from our FANUC contacts. I do understand the consequences of running in AUTO and have taken the necessary steps to ensure that the whole environment, including myself, is safe from any danger. I can also run it in the other modes, shouldn't be a problem.

As for the general terminology, mentioning the "arm's side" is simply my way of excluding the PC. I do understand that the program runs on the controller, not the arm.

As for training, we have and are still seeking training regarding the specifics of FANUC. This is not our first time with robotics arms or ROS-Industrial for that matter, and certainly won't be our last. However, there are features about this driver and the way FANUC runs ROS on the controller that we've never experienced before, hence our lack of knowledge.

"Cancelling the ros nodes" simply means terminating them on the PC side.

As for sending the joint commands, I was only sending one joint command (through a mock up ros node I made) that used the position from an earlier arm configuration. After reading the configuration, I moved the arm slightly so that the new current position wouldn't be the same one I just wrote down, and I would send the written down one, essentially making the arm come back to the original position.

[hugols16]

Here are the step-by-step instructions:

1. Start the ROS program on the controller
2. Start each of the launch files under fanuc_driver
3. Make sure that the controller is broadcasting the arm joint states to the driver under /joint_states
4. Run the custom ros node to send a single joint command, which is also sent to a made-up topic (for simple verification that the message was sent)
5. With the timeout, the driver would wait until the timeout is hit, and then return the error message mentioned above
6. If the timeout is disabled, the driver is kept in a state where it's waiting for something

[gavanderhoorn]

2 . Start each of the launch files under fanuc_driver

I'm not sure if with "each of the launch files" you mean all of them? If so, then please don't use those directly. Use the robot_interface_streaming_lrmate200id.launch that is supplied with fanuc_lrmate200id_support. That will take care of starting the required nodes properly.

If you haven't seen it, could you take a look at Working with ROS-Industrial Robot Support Packages? That explains how the robot support pkgs in ROS-Industrial are layed out and set up.

[gavanderhoorn]

Please check whether things work better if you use the launch file from the support package. If they don't, let's assume something is wrong with fanuc_driver itself.

Could I ask you to open a new issue over at ros-industrial/fanuc in that case?

[gavanderhoorn]

I've tried to reproduce this today and while I can cause a lot of unexpected warnings/errors (which I'll address later, thanks for getting me to do this) I can't seem to run into what you described.

Could you please tell me what it is exactly that you are starting?

[gavanderhoorn]

I'm going to close this as it's unclear whether this is a problem with fanuc_driver and / or any of the other fanuc_* packages, or whether this is a problem with the controller and / or use of the packages.

Without more information I'm afraid I can't help you.