Fatal error when unpacking RobotState packet

[akgoins]

When launching ur5_bringup.launch, I keep getting the error "Exception: Fatal error when unpacking RobotState packet" which prevents me from receiving any information on the "/io_states" topic. I am getting data on the /joint_states and /wrench topics, and I can set IO using the /set_IO service, but I am not receiving any feedback. Is this an issue with newer controllers? I have interface and controller version 3.1.16916. Here is the full debug output:

[WARN] [WallTime: 1425584960.972983] No calibration offset for joint "shoulder_pan_joint"
[WARN] [WallTime: 1425584960.973292] No calibration offset for joint "shoulder_lift_joint"
[WARN] [WallTime: 1425584960.973510] No calibration offset for joint "elbow_joint"
[WARN] [WallTime: 1425584960.973790] No calibration offset for joint "wrist_1_joint"
[WARN] [WallTime: 1425584960.974083] No calibration offset for joint "wrist_2_joint"
[WARN] [WallTime: 1425584960.974338] No calibration offset for joint "wrist_3_joint"
[INFO] [WallTime: 1425584960.974532] No calibration offsets loaded from urdf
[INFO] [WallTime: 1425584960.975878] Max velocity accepted by ur_driver: 10.0 [rad/s]
[INFO] [WallTime: 1425584960.979008] Bounds for Payload: [0.0, 10.0]
Disconnected.  Reconnecting
[INFO] [WallTime: 1425584961.013669] Programming the robot
[INFO] [WallTime: 1425584961.013935] Programming the robot at 192.168.1.50
Likely a syntax error:
4���������  URControl
BFeb 06 2015, 15:17:07
Exception in thread URConnection:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/swri/catkin_ws/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 279, in __run
    self.__on_packet(packet)
  File "/home/swri/catkin_ws/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 191, in __on_packet
    state = RobotState.unpack(buf)
  File "/home/swri/catkin_ws/src/universal_robot/ur_driver/src/ur_driver/deserialize.py", line 326, in unpack
    raise Exception("Fatal error when unpacking RobotState packet")
Exception: Fatal error when unpacking RobotState packet

Handling a request
[INFO] [WallTime: 1425584961.830251] Robot connected
[2015-03-05 13:49:21.837790] Out: hello
The action server for this driver has been started

[gavanderhoorn]

Could you tell us which version of the driver you are running? Source build, or from debs?

[INFO] [WallTime: 1425584961.013935] Programming the robot at 192.168.1.50 Likely a syntax error:

seems suspicious.

[akgoins]

I was using the debs, but I recently tried using source in case there was an update which may have fixed it. Same error both ways.

[gavanderhoorn]

Ok, well this could be a change in the binary protocol that was introduced by a recent update. v3.1.16916 seems like it is the newest version.

It would help (me, in any case) if you could make a wireshark / tcpdump capture while trying to start the driver. That would show the network traffic at the time of the failure.

[gavanderhoorn]

Ok. If I'm interpreting the UR wiki correctly, the newest version sends a VesionMessage [sic] (in a ROBOT_MESSAGE = 20 pkt) on every new connection first, before sending ROBOT_STATE = 16 pkts (see here). That is new, afaik, so we'll have to update the deserialiser classes to cope with that.

A wireshark trace would still help with this.

[gavanderhoorn]

@akgoins: the wireshark captures you sent me show that a ROBOT_MESSAGE is sent on initial connection to the secondary interface, as explained on the wiki. This appears to be a change in the protocol introduced by v3.1.16916.

deserialise.py (and possibly deserialiseRT.py) as well as driver.py will need to be updated to handle these new pkts. Up until this version, UR always stated that no other pkts than ROBOT_STATE (16) would be send out over the secondary interface. Apparently that has changed.

[gavanderhoorn]

@akgoins: a short-term workaround would be to downgrade to 3.1.16828, as that is still compatible. I'm not sure if/when I'll be able to work on this further.

@ipa-fxm, could you take a look at this?

[fmessmer]

I guess it's just an unhandled exception! "Fatal Error" is part of the error message here @akgoins could you please test #187

[fmessmer]

Hmm, I wonder why we don't see "Likely a syntax error:" from here in your log above, as this should be printed in case a mtype==20 is received and ROBOT_MESSAGE is defined as 20 (as can be seen in the UR wiki) Are you getting a mtype==25 which would be the PROGRAM_STATE_MESSAGE?

[gavanderhoorn]

@ipa-fxm: the wireshark traces @akgoins sent me show the first packet received is a ROBOT_MESSAGE (20). That is also consistent with what is described on the UR wiki.

The change proposed in PR #187 works around this by essentially ignoring that packet type. Would we perhaps want to deserialise it and use the info in those packets to our advantage?

Hmm, I wonder why we don't see "Likely a syntax error:"

The console output copy/pasted does show that line? See:

[INFO] [WallTime: 1425584961.013935] Programming the robot at 192.168.1.50 Likely a syntax error:

[fmessmer]

Arg...it's been too late...I'll deal with it another time...

[fmessmer]

I guess it would help to have a look at those wireshark captures to understand what's going on on the socket...

[gavanderhoorn]

Sure, I'll send them to you.

We might want to store these in a more accessible place. Perhaps we can add them to the ur_driver package. That would also make it possible to create some unit tests for the deserialisers, to make sure we don't introduce regressions when fixing incompatibilities with newer protocol versions.

[fmessmer]

It might be a different problem, but it's related to deserialize/deserializeRT... When fixing this issue also have a look at this thread

[fmessmer]

The new protocol revision is a b****... There are a lot of new things that break current assumptions in the ur_driver:

1. New ROBOT_MESSAGE packet

   First packet received on 30002 is a new ROBOT_MESSAGE message. Beside this ROBOT_MESSAGE, there is no ROBOT_STATE message in this first packet, which is why the ROBOT_STATE does not get instantiated and a call to members (e.g. robot_mode_data) fail.

   Actions taken: Deserialize tries to unpack both ROBOT_STATE and new ROBOT_MESSAGE here, with proper handling here

2. New ROBOT_MODE defines

   This is even nastier! While until revision 11, ROBOT_MODE RUNNING had the enum value 2, since revision 12, RUNNING has the enum value 7 (7 has been NO_POWER in the old defines!!!)

   Actions taken: Added new ROBOT_MODE defines here with proper interpretation here

3. Unable to getConnectedRobot with v3.1

   While it works for v3.0, somehow, I'm not able to get a handle to the robot for v3.1 robots (e.g. URSim_3.1.17141).

I started a new branch where I tried to fix the recently reported issues with v3 robots (including those on the ROS-I-MailingList). You can have a look at my WIP on this branch I think, I solved 1. and 2., but I still am not able to use the ur_driver with URSim_3.1.17141. I don't know whether this is because of URSim or because of 3.1.17141!

Anyone able to test this branch with a real robot v3.1 or later? Any suggestions why for a v3.1, getConnectedRobot() returns None, when called from here for v3.1?

[gavanderhoorn]

Yes, I empathise. The new protocol revision does change a lot of things which seem to break a lot of software 'out there'. Especially the RobotMode enum value changes seem strange, but there is probably a good reason for it ..

In any case, at least the new ROBOT_MESSAGE contains some version info, which finally makes version detection possible without relying on on pkt sizes being sufficiently different to implement it.

May I suggest to rework deseralize(RT).py to introduce a proper OO hierarchy of pkts and associated deserialisation functionality? Abstraction should allow us to shield higher layers from the the differences between the protocol versions.

[shaun-edwards]

This seems frustrating on all sides. Perhaps there is an easier (non-software) solution. Have we considered engaging Universal so that at the very least, we have a heads up when the software is going to break. Another approach may be to standardize on specific versions and not support all versions.

[gavanderhoorn]

Well it would help if UR didn't introduce breaking changes in what would normally be expected to be patch releases (ie: 3.1.x -> 3.1.y). But that is entirely their prerogative, of course.

(and having a heads up would be nice, but it wouldn't remove the need to extend the deserialisers to support these versions. All changes to the protocols have been cumulative so far, so all newer versions are expected to include these changes).

[ashfaqfarooqui]

@ipa-fxm I could test your WIP branch on a real robot running v3.1.16828. But it did not seem very stable, there were a couple of times the robot disconnected after reaching robot state 2, because it could not find r. Also, after pressing the emergency stop, ROS had to be restarted inorder to make any operations.

[fmessmer]

Thanks @ashfaqfarooqui for testing and reporting here!

Yes it's still WIP and thus not yet stable. This is mainly because I don't have a real hardware v3.1 around to properly test it. I only tested it with URSim 3.1. And there, I always got that error you pointed to. Something seems to be wront with the select.select part!

Unfortunately, I don't have the time to further investigate this within the next weeks. @ashfaqfarooqui would you be willing to further investigate the problem in order to be able to support your v3.1 hardware...and then share it via a PR? That would highly be appreciated!

[ashfaqfarooqui]

Yes, I continue to investigate this, it would be helpful if there was some documentation available? I have just started tweaking around with UR and learning it the hard way. Currently I am changing UR script functions in prog to use the non-depricated functions for IO interfacing. you can look at my WIP Branch.

[ashfaqfarooqui]

I have observed that port 30002 seems to be in CLOSE_WAIT mode everytime this error occurs. I think it might be because prog has opened connection and is still running on the robot. On testing the driver by removing prog I havn't been able to reproduce this error. This is still running on URsim, I will test this out on the robot next week.

Does anyone know why this port does not close?

[ashfaqfarooqui]

Update: I got a reply from UR regarding this issue,

We do know that the controller could hangs when using port 30001/30002 when using newest software. Our R&D are working on solving this issue.

[gavanderhoorn]

Hm, interesting @ashfaqfarooqui. So somehow, the connection to 30002 is lost / closed, the control box has 30002 in CLOSE_WAIT and then everything hangs?

[ashfaqfarooqui]

The problem rarely occurs on the first time you connect with the UR. On subsequent connect we are unable to connect to the UR. Yes, what I can notice is port 30002 is in CLOSE_WAIT state and the RECV-Q has some data which it has not released. Using sock.shutdown() put it into FIN_WAIT state but did not solve the issue.

The robot does not seem to hang,as we are able to control it using using the pendent. But, it is not possible to connect using a socket. PS: I think my previous question is invalid. I can see prog is shutting down correctly.

[shaun-edwards]

Does this mean the fix for the socket issue is to wait for Universal to provide a fix? By the way, is there an issue for this specific problem, since this issue is really about the Robot State packet?

[de-vri-es]

If the socket issue is what I think it is, you can avoid it in software.

I experienced an issue with the connection to the controller starting with version 3.1 where the controller in some situations stopped sending any data to new connections. This seemed to happen when either multiple connection attempts are made in a very short time period (when I triggered it I opened two connections simultaneously) or when sending a lot of data to the controller (for example, when re-uploading a script in a fast loop). I reported the issue a few weeks ago to UR, so it could be the issue they are referring to.

addemdum: I only tested this with a connection on port 30001 (the primary interface), so I don't know if data sent to the script over TCP can also trigger the issue.

[ashfaqfarooqui]

@shaun-edwards : The socket issue has come to light while fixing Robot state packet error as mentioned in this comment. Point 1 and 2 were fixed, and it seems like the root cause of point 3 is an issue in UR itself.

@de-vri-es : While that is one way to reproduce this issue, I was able to reproduce this error when the robot tries to reconnect after the emergency stop is activated once.

On running it with URsim It threw this exception :

[fmessmer]

Thanks @ashfaqfarooqui for investigating this further...keep us posted in case you hear about a possible fix from Universal Robot side...

@shaun-edwards, I created a pointing issue documenting the issue that newly evolved during the discussion here, but I'd prefer to keep the discussion here rather than migrating to the new issue...

[jmfurrer]

The problem rarely occurs on the first time you connect with the UR. On subsequent connect we are unable to connect to the UR. Yes, what I can notice is port 30002 is in CLOSE_WAIT state and the RECV-Q has some data which it has not released. Using sock.shutdown() put it into FIN_WAIT state but did not solve the issue.

@ashfaqfarooqui, I also reported this to UR some weeks ago, they are still investigating the issue. By sending a TCP RST I can get rid of the zombie connections, but it seems like URControl's TCP server for the secondary interface "turns bad" upon establishing a connection to 30002, as no robot messages are sent/received before the reset script is first sent and this also happened just after restarting the URControl process. Trying different modifications, I couldn't determine if anything particular about how driver.py makes the connection could make this problem surface.

[shaun-edwards]

I recently spoke to UR about downgrading systems, since previous versions of the UR software don't have this bug. They said it can be done the same way as a software upgrade. I would recommend this to all who are having this problem. Previous versions of the UR software will work better with ROS-Industrial.

[de-vri-es]

You also have to "upgrade" the joint firmware manually when downgrading. The procedure is documented on the UR support wiki (somewhere).

[davetcoleman]

I have run into this same problems described here, using firmware 3.1.17336 (Mar 26 2015), ROS Indigo, and the indigo-devel branch compiled from source. Any update on the new API since last month? I am happy to help test on hardware. Thanks for investigating this!

[shaun-edwards]

I have an email from 5/11 stating the bug has been patched and will be released into a software upgrade. I recommend asking UR for the upgrade.

[fmessmer]

@davetcoleman There hasn't been much progress wrt the protocol compatibility SW1.8 (and earlier) vs. SW3.0 (and later) as it was not really possible to test due the issue in #199. Also see my comment there!

I guess we can only get back to testing/fixing the deserialize.py and deserializeRT.py and the ur_driver once the TCP_NODELAY problem allows us to connect to an actual SW3.0 (and later) robot... Note that I've started something that (seemed) to work with a SW3.0.16285 here...but it needs more testing!

Furthermore, I was asking UniveralRobot what there official recommendation to customers is wrt to backwards compatibility: And they clearly state that there is a API break between SW1.8 (prior to 3.0) and SW3.0 (and newer)...thus I'm really starting to think about that it is not possible to support all software version with the same ur_driver.py! Maybe we also should provide a driver version that can be used with SW1.8 Prior to 3.0) and - separated from that - another driver that works with SW3.0 (and newer)....be it separate ur_driver_new.py or whatsoever... Another option would be to rethink the prog-based communication that has been before we made use of the RT-communication on port 30003 which would allow us to again use a protocol that is not affected by UniversalRobot software updates....but there we need to make sure we are not running into the same limitations that made us go for the deserializeRT approach...

Maybe we could achieve this, by rolling back the ur_driver python driver to a version where it was not yet messed up with SW3.0 compatibility hacks and use a ros-control-enabled c++ driver supporting all SW3.0 (and newer)...then we also would have a similarly big conceptual change as UR had :wink:

Thoughts?

[davetcoleman]

@ipa-fmw thanks for the quick response!

I am still working on getting access to the UR wiki and downloads page (I have emailed), so am unable to upgrade/downgrade firmware yet.

It does seem that the UR driver might need to be branched at this point to make compatibility easier and to prevent users of the old firmware from experiencing a break in compatibility.

I of course like the idea of a ros_control driver, but I'm afraid I do not have the cycles to work on that.

I will post my results from firmware changes after UR responds.

[gavanderhoorn]

@ipa-fxm: as CB2 controllers will not get upgrades anymore, I think we can consider it a static platform now. It should not be too hard to fix any remaining issues on our side and then freeze development for that platform. Reverting back to the old prog based interface does not make much sense to me: the current version works (relatively) well and as no changes are expected (at least on the controller side), v1.8.x controllers shouldn't be such a moving target to track for us/you as CB3/v3.x is.

Now for CB3 (and up), things are different: we can expect development to continue and breakages to occur at any time. The new ROBOT_MESSAGE does should make it easier to implement feature detection, so we should be able to handle those breakages better. Whether that is in a ros_control based driver or in the current one doesn't really matter I think. They'll both need fixing.

Depending on how much work you/we/the community is/are prepared to put into this, splitting up the current driver into pre-CB3 and CB3-and-up versions makes sense. Perhaps even having two different pkgs would be a good thing: it should be immediately clear to users which to use, and it removes the need to keep the driver bw-compatible with CB2. As the public interface (ROS API) of the pkg does not need to change, migration of users is not forced (we could just say: "CB3 compatibility is much better with ur_driver_cb3+ (or whatever), but you're free to keep using the old version") and modularity of the repository is relatively high (no huge, one-click launch files that start everything which would need to be updated), it should not pose too many problems.

An alternative to splitting up the pkg is to introduce (even more) launch file arguments and internal branches in the execution path of driver.py, but I doubt that'll be pretty.

PS: all imho, of course.

[davetcoleman]

I downgraded to last August's firmware version (SW3.0.15547) and it appears to be running normal. Seems like this issue should be revisited when UR releases the next version.

[ehsan-asadi]

Hi all, I have the same issue with ur10 and I am wondering if the problem has been solved in the new firmware or not? I keep getting the error "Exception: Fatal error when unpacking RobotState packet" which prevents me from receiving any information on the "/joint_states" topic Currently, the version of our Robot Controller is 3.2.18654. I am not sure if I have to upgrade the firmware or downgrade it to solve the problem. I would appreciate your suggestions.

[ashfaqfarooqui]

@ehsan-asadi did you try this branch ?

[ehsan-asadi]

Thanks for your suggestion, but still I do not receive any information on the "/joint_states" topic. With this branch I get this error;

Exception in thread URConnectionRT and Exception in thread URConnection:

DebugRobotState 0
Connect
DebugRobotState 1
DebugRobotState 2
Disconnected.  Reconnecting
[INFO] [WallTime: 1448090135.413308] Programming the robot
[INFO] [WallTime: 1448090135.413723] RobotState 2
Ready to program
Send_Programm
RobotState: 2
[INFO] [WallTime: 1448090135.413978] Programming the robot at 192.168.0.12
DebugRobotState 3
Sent Program
RobotMessage received
3
DebugRobotState 1
RobotVersion 3
RobotState is None
RobotStateRT has wrong length: 1060
Exception in thread URConnectionRT:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/bob/catkin_ws/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 419, in __run
    self.__on_packet(packet)
  File "/home/bob/catkin_ws/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 385, in __on_packet
    for i, q in enumerate(stateRT.q_actual):
AttributeError: 'RobotStateRT' object has no attribute 'q_actual'

UNKNOWN_MESSAGE
12
RobotMessage received
Exception in thread URConnection:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/bob/catkin_ws/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 319, in __run
    self.__on_packet(packet)
  File "/home/bob/catkin_ws/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 208, in __on_packet
    print scp.robot_message.version_message.majorVersion
AttributeError: version_message

Handling a request
[INFO] [WallTime: 1448090136.030285] Robot connected
[2015-11-21 15:15:36.034699] Out: hello

[ashfaqfarooqui]

Seems like UR has updated the realtime interface again in version 3.2. Protocol details can be fount here.

I suggest you either downgrade and use version 3.0 or 3.1 if you need it urgently.

[toliver]

@ehsan-asadi we have seen the same errors with this branch on a UR5 with version 3.2.

We have solved it by adding to it the 2 new fields necessary for v3.2 (see doc liked by @ashfaqfarooqui ). This branch has been tested and the driver works fine. There is one error reported about an unknown frame type but the robot state is received correctly and the trajectories work.

We have also noticed that the UR robots with the newest serial numbers can't be downgraded to 3.0, only to 3.1.18213 (see here).

Fortunately the socket issue seems to have been fixed in v3.2 (at least we didn't experience the same problems we were having with v3.1).

[ashfaqfarooqui]

I guess the current code is no longer valid for 3.1 and 3.2. I would actually suggest we create a PR from the branch. This will clear out a number of issues relating to connection.

[nick-pestell]

I believe I'm experiencing the same issue. I'm using UR5 with V3.2.18744. Initially I was using the debs version of UR packages and experienced similar console output to @akgoins;

... logging to /home/tactip/.ros/log/5a7576a2-0160-11e6-a222-0800275c9265/roslaunch-tactip-VirtualBox-2370.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://tactip-VirtualBox:48518/

SUMMARY
========

PARAMETERS
 * /robot_description: <?xml version="1....
 * /robot_ip_address: 164.11.73.84
 * /robot_reverse_port: 50001
 * /rosdistro: indigo
 * /rosversion: 1.11.16
 * /tf2_buffer_server/buffer_size: 120.0
 * /ur_driver/max_payload: 10.0
 * /ur_driver/max_velocity: 10.0
 * /ur_driver/min_payload: 0.0

NODES
  /
    robot_state_publisher (robot_state_publisher/state_publisher)
    tf2_buffer_server (tf2_ros/buffer_server)
    ur_driver (ur_driver/driver.py)

auto-starting new master
process[master]: started with pid [2385]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 5a7576a2-0160-11e6-a222-0800275c9265
process[rosout-1]: started with pid [2398]
started core service [/rosout]
process[robot_state_publisher-2]: started with pid [2401]
process[ur_driver-3]: started with pid [2402]
process[tf2_buffer_server-4]: started with pid [2403]
Setting prefix to 
[WARN] [WallTime: 1460542404.636722] No calibration offset for joint "shoulder_pan_joint"
[WARN] [WallTime: 1460542404.636956] No calibration offset for joint "shoulder_lift_joint"
[WARN] [WallTime: 1460542404.637113] No calibration offset for joint "elbow_joint"
[WARN] [WallTime: 1460542404.637273] No calibration offset for joint "wrist_1_joint"
[WARN] [WallTime: 1460542404.637414] No calibration offset for joint "wrist_2_joint"
[WARN] [WallTime: 1460542404.637554] No calibration offset for joint "wrist_3_joint"
[INFO] [WallTime: 1460542404.637709] No calibration offsets loaded from urdf
[INFO] [WallTime: 1460542404.638761] Max velocity accepted by ur_driver: 10.0 [rad/s]
[INFO] [WallTime: 1460542404.640150] Bounds for Payload: [0.0, 10.0]
Disconnected.  Reconnecting
[INFO] [WallTime: 1460542404.683816] Programming the robot
RobotStateRT has wrong length: 1060
[INFO] [WallTime: 1460542404.689068] Programming the robot at 164.11.73.84
Exception in thread URConnectionRT:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/ros/indigo/lib/ur_driver/driver.py", line 374, in __run
    self.__on_packet(packet)
  File "/opt/ros/indigo/lib/ur_driver/driver.py", line 340, in __on_packet
    for i, q in enumerate(stateRT.q_actual):
AttributeError: 'RobotStateRT' object has no attribute 'q_actual'

Likely a syntax error:
4?????????    URControlI8Oct 07 2015, 11:12:33
Exception in thread URConnection:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/ros/indigo/lib/ur_driver/driver.py", line 276, in __run
    self.__on_packet(packet)
  File "/opt/ros/indigo/lib/ur_driver/driver.py", line 188, in __on_packet
    state = RobotState.unpack(buf)
  File "/opt/ros/indigo/lib/python2.7/dist-packages/ur_driver/deserialize.py", line 324, in unpack
    raise Exception("Fatal error when unpacking RobotState packet")
Exception: Fatal error when unpacking RobotState packet

Handling a request
[2016-04-13 11:13:25.620330] Out: hello
[INFO] [WallTime: 1460542405.660702] Robot connected

The main difference I've noticed is that I'm not getting "action server for this driver started".

I took a look at what topics there are and I'm getting:

/io_states
/joint_states
/rosout
/rosout_agg
/tf
/tf2_buffer_server/cancel
/tf2_buffer_server/feedback
/tf2_buffer_server/goal
/tf2_buffer_server/result
/tf2_buffer_server/status
/tf_static
/wrench

Nothing is being published on /io_states or /joint_states.

I have also now tried a source version from the shadow-robot branch as suggested by @toliver.

With this version I'm still getting the "Fatal error when unpacking RobotState packet" however I'm now getting "action server for this driver has been started":

... logging to /home/tactip/.ros/log/3ca79a14-0161-11e6-a51e-0800275c9265/roslaunch-tactip-VirtualBox-2307.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
Done checking log file disk usage. Usage is <1GB.

started roslaunch server http://tactip-VirtualBox:54545/

SUMMARY
========

PARAMETERS
 * /robot_description: <?xml version="1....
 * /robot_ip_address: 164.11.73.84
 * /robot_reverse_port: 50001
 * /rosdistro: indigo
 * /rosversion: 1.11.16
 * /tf2_buffer_server/buffer_size: 120.0

NODES
  /
    robot_state_publisher (robot_state_publisher/state_publisher)
    tf2_buffer_server (tf2_ros/buffer_server)
    ur_driver (ur_driver/driver.py)

auto-starting new master
process[master]: started with pid [2322]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to 3ca79a14-0161-11e6-a51e-0800275c9265
process[rosout-1]: started with pid [2335]
started core service [/rosout]
process[robot_state_publisher-2]: started with pid [2338]
process[ur_driver-3]: started with pid [2339]
process[tf2_buffer_server-4]: started with pid [2340]
/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/driver.py:78: SyntaxWarning: The publisher should be created with an explicit keyword argument 'queue_size'. Please see http://wiki.ros.org/rospy/Overview/Publishers%20and%20Subscribers for more information.
  pub_joint_states = rospy.Publisher('joint_states', JointState)
/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/driver.py:79: SyntaxWarning: The publisher should be created with an explicit keyword argument 'queue_size'. Please see http://wiki.ros.org/rospy/Overview/Publishers%20and%20Subscribers for more information.
  pub_wrench = rospy.Publisher('wrench', WrenchStamped)
/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/driver.py:80: SyntaxWarning: The publisher should be created with an explicit keyword argument 'queue_size'. Please see http://wiki.ros.org/rospy/Overview/Publishers%20and%20Subscribers for more information.
  pub_io_states = rospy.Publisher('io_states', IOStates)
Setting prefix to 
[WARN] [WallTime: 1460542786.265385] No calibration offset for joint "shoulder_pan_joint"
[WARN] [WallTime: 1460542786.266249] No calibration offset for joint "shoulder_lift_joint"
[WARN] [WallTime: 1460542786.267097] No calibration offset for joint "elbow_joint"
[WARN] [WallTime: 1460542786.268043] No calibration offset for joint "wrist_1_joint"
[WARN] [WallTime: 1460542786.269358] No calibration offset for joint "wrist_2_joint"
[WARN] [WallTime: 1460542786.273710] No calibration offset for joint "wrist_3_joint"
[ERROR] [WallTime: 1460542786.273850] Loaded calibration offsets: {}
Disconnected.  Reconnecting
[INFO] [WallTime: 1460542786.326565] Programming the robot
[INFO] [WallTime: 1460542786.327436] Programming the robot at 164.11.73.84
Likely a syntax error:
4���������  URControlI8Oct 07 2015, 11:12:33
Exception in thread URConnection:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 238, in __run
    self.__on_packet(packet)
  File "/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 171, in __on_packet
    state = RobotState.unpack(buf)
  File "/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/deserialize.py", line 236, in unpack
    raise Exception("Fatal error when unpacking RobotState packet")
Exception: Fatal error when unpacking RobotState packet

[INFO] [WallTime: 1460542787.334902] Programming the robot at 164.11.73.84
Handling a request
[INFO] [WallTime: 1460542787.502321] Robot connected
[2016-04-13 11:19:47.507109] Out: hello
The action server for this driver has been started
Connection closed (command): EOF on recv
Handling a request
[2016-04-13 11:19:48.274226] Out: hello

Additionally in the same window after a few seconds I'm getting the following output:

----------------------------------------
Exception happened during processing of request from ('164.11.73.84', 42535)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 424, in handle
    raise Exception("Unknown message type: %i" % mtype)
Exception: Unknown message type: -16370
----------------------------------------

With this source version I'm getting additional "follow_joint_trajectory" topics:

/follow_joint_trajectory/cancel
/follow_joint_trajectory/feedback
/follow_joint_trajectory/goal
/follow_joint_trajectory/result
/follow_joint_trajectory/status
/io_states
/joint_states
/rosout
/rosout_agg
/tf
/tf2_buffer_server/cancel
/tf2_buffer_server/feedback
/tf2_buffer_server/goal
/tf2_buffer_server/result
/tf2_buffer_server/status
/tf_static
/wrench

But still nothing on /io states or /joint_states.

Additionally, I think linked to the presence of the action server, when running "test_move.py" the console output is as follows with the source version:

Waiting for server...
Connected to server

whereas with debs the server doesn't connect.

[nick-pestell]

An update - I have had some success with the shadow-robot branch. Information is being published to /joint_states initially until the exception happened from the SocketServer.py file:

----------------------------------------
Exception happened during processing of request from ('164.11.73.84', 42535)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/home/tactip/URsource/src/universal_robot/ur_driver/src/ur_driver/driver.py", line 424, in handle
    raise Exception("Unknown message type: %i" % mtype)
Exception: Unknown message type: -16370
----------------------------------------

[gavanderhoorn]

Unless you have a specific reason to prefer ur_driver, I'd recommend you take a look at ur_modern_driver, which already includes fixes for deserialisation of the updated protocols.

The only exception is if you're running Polyscope v3.2.x, because then you'll run into ThomasTimm/ur_modern_driver#17. Updating the deserialisers shouldn't be too difficult, and may make a nice PR for @ThomasTimm to merge.

[nick-pestell]

Brilliant! thank you. So far so good with the shadow-robot branch. Will check with debs.

[VahidAminZ]

@toliver I Shadow-robot branch worked with UR3 (had to merge indigo) and UR 3.3.1.

[jkwang1992]

@homesick-nick I am using the UR3 with shadow-robot and has the same problem as mentioned above. Could you tell me how to fix the problem specifically ? Such as "No calibration offset for joint "shoulder_pan_joint" " and " No calibration offsets loaded from urdf". Thank you very much !

[gavanderhoorn]

@WangJIankun1992: the warnings you include in your comment ("No calibration offset ..") are benign, and should not influence the normal functioning of the driver.

And again, to reiterate: our recommendation would be to use ur_modern_driver with CB3 controllers, not ur_driver. You're of course free to use the shadow robot fork of ur_driver, but ur_modern_driver should be a drop-in replacement in most cases for ur_driver, but with extended compatibility.