search

robotics-in-concert / rocon_multimaster

Key components for ros multimaster systems
11 stars 19 forks source link

[Gateway] insecure string pickle #280

Open jihoonl opened 10 years ago

jihoonl commented 10 years ago 
[INFO] [WallTime: 1402386399.358722] Gateway : disengaged connection with the hub [Cafe Concert][192.168.10.151:6380]
Traceback (most recent call last):
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 78, in spin
    self._gateway.spin()
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 184, in update_flipped_interface
    status = hub.get_flip_request_status(flip.remote_rule)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 717, in get_flip_request_status
    status = self.get_multiple_flip_request_status([remote_rule])
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 748, in get_multiple_flip_request_status
    rule_status, source, connection_list = utils.deserialize_request(flip)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 218, in deserialize_request
    deserialized_list = deserialize(request_str)
  File "/home/yujin/turtlebot/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 182, in deserialize
    return pickle.loads(str_msg)
ValueError: insecure string pickle
stonier commented 10 years ago

Do you have any other details on this. Was this a robot, or was it the concert? Did it happen on startup, runtime or on shutdown?

jihoonl commented 10 years ago

It was on the robot,and runtime. The concert was up and running more than 5mins.

Though it happened, public interface connections didn't get broke and were able to receive all commands.

stonier commented 10 years ago

Yeah, it would bring down the gateway node, but all existing socket connections would remain.

jihoonl commented 9 years ago

Happened in the robot.

[INFO] [WallTime: 1418812976.291401] Gateway : disengaged connection with the hub [xavier][192.168.10.32:6380]
Traceback (most recent call last):
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 78, in spin
    self._gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 132, in update_flipped_interface
    status = hub.get_flip_request_status(flip.remote_rule)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 717, in get_flip_request_status
    status = self.get_multiple_flip_request_status([remote_rule])
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 748, in get_multiple_flip_request_status
    rule_status, source, connection_list = utils.deserialize_request(flip)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 218, in deserialize_request
    deserialized_list = deserialize(request_str)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 182, in deserialize
    return pickle.loads(str_msg)
ValueError: insecure string pickle
piyushk commented 9 years ago

According to http://stackoverflow.com/questions/1746825/python-valueerror-insecure-string-pickle, it's probably a corrupted pickle. We should catch this error, and discard the original request being deserialized.

jihoonl commented 9 years ago

Do you have any suggestion to chase up this issue? where would be the best place to start to add debug message?

piyushk commented 9 years ago

I don't have a particularly good solution for this problem.

1) We should catch this error in the deserialize function, print a debug message, and return a deserialization error. 2) Everything in gateway_hub.py should catch the deserialize error and deal with it appropriately. Unfortunately, we'll have to look at each deserialize call one by one and see how we want to handle it if something goes wrong. 3) We then need to establish that when this error happens, have the values on the redis hub gotten corrupted (i.e. the debug message is printed over and over again or not). If the value on the redis hub has gotten corrupted, we'll need to discuss how we want to fix that. Maybe we can figure out why the value got corrupted in the first place. If the error message is only printed one off, then we should be fine as things will normalize in the next update.

jihoonl commented 9 years ago

Another log

[INFO] [WallTime: 1433148471.084004] Gateway : lost connection with hub, attempting to disconnect...
Traceback (most recent call last):
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 81, in spin
    self._gateway.spin()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 184, in update_flipped_interface
    status = hub.get_flip_request_status(flip.remote_rule)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 717, in get_flip_request_status
    status = self.get_multiple_flip_request_status([remote_rule])
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_hub.py", line 748, in get_multiple_flip_request_status
    rule_status, source, connection_list = utils.deserialize_request(flip)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 218, in deserialize_request
    deserialized_list = deserialize(request_str)
  File "/home/yujin/ros/turtlebot/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/utils.py", line 182, in deserialize
    return pickle.loads(str_msg)
ValueError: insecure string pickle
jihoonl commented 9 years ago

Looking at the three different logs, insecure string pickle error raised while deserializing flip_in request in all cases. Thus, flip_ins keys in hub is corrupted probably.