search

robotics-in-concert / rocon_multimaster

Key components for ros multimaster systems
12 stars 19 forks source link

Occurrence unreachable network problem in local clients at same time. #311

Closed dwlee closed 8 years ago

dwlee commented 9 years ago

Concert Master Side

[ INFO] [1436356577.599822714]: Zeroconf: service was removed [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[ INFO] [1436356583.680712591]: Zeroconf: discovered new service [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[ INFO] [1436356584.904560840]: Zeroconf: resolved service [xavier][_ros-multimaster-hub._tcp][local][2][ipv4][169.254.6.156:6380]
[ INFO] [1436356587.646353212]: Zeroconf: service was removed [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[ INFO] [1436356587.929152819]: Zeroconf: discovered new service [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[ INFO] [1436356589.155895992]: Zeroconf: resolved service [xavier][_ros-multimaster-hub._tcp][local][2][ipv4][192.168.10.32:6380]
[WARN] [WallTime: 1436356640.808056] Hub Watcher: gateway room_checkerc248c41308d74e38887aea02189dc266 has been unavailable for 60 seconds! Marking as unavailable.
[WARN] [WallTime: 1436356640.808586] Hub Watcher: gateway robosem_bridge496880fe7e8e47c8a895138ed97a14c0 has been unavailable for 60 seconds! Marking as unavailable.
[WARN] [WallTime: 1436356640.808974] Hub Watcher: gateway hue_bridgeedbaa732fbb84307bcaefb822a33b9e6 has been unavailable for 60 seconds! Marking as unavailable.
[WARN] [WallTime: 1436356640.809233] Hub Watcher: gateway ipcamerf964576e760f4b8980c57d653bd1ae55 has been unavailable for 60 seconds! Marking as unavailable.
[WARN] [WallTime: 1436356640.809491] Hub Watcher: gateway door_opener102793ed5f47439a8c1c25bc36d1cda1 has been unavailable for 60 seconds! Marking as unavailable.
[INFO] [WallTime: 1436356641.718569] Conductor : concert client transition [available->missing][robosem_bridge]
[INFO] [WallTime: 1436356641.719016] Conductor : concert client transition [available->missing][room_checker]
[INFO] [WallTime: 1436356641.719362] Conductor : concert client transition [available->missing][ipcamer]
[INFO] [WallTime: 1436356641.719708] Conductor : concert client transition [available->missing][hue_bridge]
[INFO] [WallTime: 1436356641.720094] Conductor : concert client transition [available->missing][door_opener]
[ INFO] [1436356641.792196416]: Remove subscriber of : door_opener
[ INFO] [1436356641.792221163]: Remove subscriber of : hue_bridge
[ INFO] [1436356641.792228679]: Remove subscriber of : robosem_bridge
[ INFO] [1436356641.792236108]: Remove subscriber of : room_checker
[WARN] [WallTime: 1436359502.948473] Hub Watcher: gateway younged72a11d7ceb457587911d33012aff56 has been unavailable for 60 seconds! Marking as unavailable.
[INFO] [WallTime: 1436359503.260744] Conductor : concert client transition [available->missing][young1]
[ INFO] [1436359503.292265499]: Remove subscriber of : ipcamer
[ INFO] [1436359503.292301355]: Remove subscriber of : young1
[WARN] [WallTime: 1436363781.140503] Hub Watcher: gateway room_checkerc248c41308d74e38887aea02189dc266 has been unavailable for 7200 seconds! Removing from hub.
[WARN] [WallTime: 1436363781.144138] Hub Watcher: gateway robosem_bridge496880fe7e8e47c8a895138ed97a14c0 has been unavailable for 7200 seconds! Removing from hub.
[WARN] [WallTime: 1436363781.147287] Hub Watcher: gateway hue_bridgeedbaa732fbb84307bcaefb822a33b9e6 has been unavailable for 7200 seconds! Removing from hub.
[WARN] [WallTime: 1436363781.149675] Hub Watcher: gateway ipcamerf964576e760f4b8980c57d653bd1ae55 has been unavailable for 7200 seconds! Removing from hub.
[WARN] [WallTime: 1436363781.150985] Hub Watcher: gateway door_opener102793ed5f47439a8c1c25bc36d1cda1 has been unavailable for 7200 seconds! Removing from hub.
[INFO] [WallTime: 1436363781.532711] Conductor : concert client transition [missing->gone][robosem_bridge]
[INFO] [WallTime: 1436363781.534032] Conductor : concert client transition [missing->gone][room_checker]
[INFO] [WallTime: 1436363781.534863] Conductor : concert client transition [missing->gone][ipcamer]
[INFO] [WallTime: 1436363781.535745] Conductor : concert client transition [missing->gone][hue_bridge]
[INFO] [WallTime: 1436363781.536766] Conductor : concert client transition [missing->gone][door_opener]
[WARN] [WallTime: 1436363781.547502] Scheduler : lost allocated concert client [robosem_bridge]
[WARN] [WallTime: 1436363781.550415] Scheduler : lost allocated concert client [room_checker]
[WARN] [WallTime: 1436363781.552995] Scheduler : lost allocated concert client [hue_bridge]
[WARN] [WallTime: 1436363781.556007] Scheduler : lost allocated concert client [door_opener]

Client Side

[ INFO] [1436331603.619227294]: Zeroconf: resolved service [dwlee][_ros-multimaster-hub._tcp][local][2][ipv4][192.168.10.24:6380]
[WARN] [WallTime: 1436331604.108386] Gateway : failed to register gateway with the hub [192.168.10.24:6380][15][hub/ip not in non-empty whitelist [dwlee, 192.168.10.24:6380][xavier]]
[INFO] [WallTime: 1436331604.109495] Gateway : blacklisting hub [hub/ip not in non-empty whitelist [dwlee, 192.168.10.24:6380][xavier]]
[ INFO] [1436334956.313777459]: Zeroconf: service was removed [dwlee][_ros-multimaster-hub._tcp][local][2][ipv4]
[INFO] [WallTime: 1436334956.471087] Gateway : remove lost service[192.168.10.24:6380] from discovered hub list
[ INFO] [1436335150.602210937]: Zeroconf: discovered new service [dwlee][_ros-multimaster-hub._tcp][local][2][ipv4]
[ INFO] [1436335151.102889506]: Zeroconf: resolved service [dwlee][_ros-multimaster-hub._tcp][local][2][ipv4][192.168.10.24:6380]
[WARN] [WallTime: 1436335151.378356] Gateway : failed to register gateway with the hub [192.168.10.24:6380][15][hub/ip not in non-empty whitelist [dwlee, 192.168.10.24:6380][xavier]]
[INFO] [WallTime: 1436335151.379483] Gateway : blacklisting hub [hub/ip not in non-empty whitelist [dwlee, 192.168.10.24:6380][xavier]]
[ INFO] [1436335197.914853139]: Zeroconf: service was removed [dwlee][_ros-multimaster-hub._tcp][local][2][ipv4]
[INFO] [WallTime: 1436335197.984942] Gateway : remove lost service[192.168.10.24:6380] from discovered hub list
[WARN] [WallTime: 1436345348.655304] Robosem Bridge : Exception error in receive header data. Reason:[Errno 104] Connection reset by peer
[ INFO] [1436356577.599769920]: Zeroconf: service was removed [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[ERROR] [WallTime: 1436356578.116917] Gateway : couldn't get system state from the master [did you set your master uri to a wireless IP that just went down?]
Traceback (most recent call last):
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 81, in spin
    self._gateway.spin()
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 75, in start
    self._gateway.update_flipped_interface(connections, remote_gateway_hub_index)
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 146, in update_flipped_interface
    local_connection_index, remote_gateway_hub_index, self._unique_name, self.master)
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/flipped_interface.py", line 94, in update
    new_flips, removed_flips, flipped = self._prepare_flips(connections, remote_gateways, unique_name, master)  # Totally regenerate a new flipped interface, compare with old
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/flipped_interface.py", line 261, in _prepare_flips
    matched_flip_rules = self._generate_flips(connection.rule.type, connection.rule.name, connection.rule.node, remote_gateways, unique_name, master)
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/flipped_interface.py", line 239, in _generate_flips
    matched_flip.rule.node = "%s,%s"%(node, master.lookupNode(node)) # just in case we used a regex
  File "/opt/ros/indigo/lib/python2.7/dist-packages/rosgraph/masterapi.py", line 433, in lookupNode
    return self._succeed(self.handle.lookupNode(self.caller_id, node_name))        
  File "/usr/lib/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1292, in single_request
    self.send_content(h, request_body)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1439, in send_content
    connection.endheaders(request_body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
socket.error: [Errno 101] Network is unreachable
[ INFO] [1436356583.681401217]: Zeroconf: discovered new service [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[ INFO] [1436356584.904898185]: Zeroconf: resolved service [xavier][_ros-multimaster-hub._tcp][local][2][ipv4][169.254.6.156:6380]
[ INFO] [1436356587.646089077]: Zeroconf: service was removed [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[ INFO] [1436356587.929208366]: Zeroconf: discovered new service [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
dwlee commented 8 years ago

The same error was happened in concert master.

[ INFO] [1437124438.102517831]: Zeroconf: service was removed [xavier][_ros-multimaster-hub._tcp][local][2][ipv4]
[INFO] [WallTime: 1437124440.108159] Gateway : unflipping received flip [hue_bridgefbc603a493ee435497e6be61e6a7f751][subscriber][/set_hue][/hue_bridge/rocon_hue_bridge_server]
Traceback (most recent call last):
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/scripts/gateway.py", line 22, in <module>
    gateway.spin()
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway_node.py", line 81, in spin
    self._gateway.spin()
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 78, in spin
    self.watcher_thread.start()
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/watcher_thread.py", line 79, in start
    self._gateway.update_flipped_in_interface(registrations, remote_gateway_hub_index)
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/gateway.py", line 378, in update_flipped_in_interface
    self.master.unregister(local_registration)
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/master_api.py", line 478, in unregister
    node_master, registration.connection.xmlrpc_uri, registration.connection.rule.name)
  File "/home/yujin/ros/yujinrnd/src/rocon_multimaster/rocon_gateway/src/rocon_gateway/master_api.py", line 548, in _unregister_subscriber
    xmlrpcapi(xmlrpc_uri).publisherUpdate('/master', name, [])
  File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1587, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.7/xmlrpclib.py", line 1273, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1301, in single_request
    self.send_content(h, request_body)
  File "/usr/lib/python2.7/xmlrpclib.py", line 1448, in send_content
    connection.endheaders(request_body)
  File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 797, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 778, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python2.7/socket.py", line 571, in create_connection
    raise err
socket.error: [Errno 101] Network is unreachable