search

rcbops / chef-cookbooks

RCB OPS - Chef Cookbooks
Other
118 stars 102 forks source link

rpcdaemon fails with a python stack trace on grizzly #874

Open breu opened 10 years ago

breu commented 10 years ago

RHEL python 2.6

2014-03-12 17:22:13  l3agent    WARNING  dal-appblx001-09.prod.walmart.com/L3 agent [23309caf-f3cf-493f-ab21-ed68ecdcd794]: is now down.
2014-03-12 17:22:13  l3agent    INFO     Removing router 7096df0b-e981-4d85-b941-2817546271d3 from dal-appblx001-09.prod.walmart.com/L3 agent [23309caf-f3cf-493f-ab21-ed68ecdcd794]
2014-03-12 17:22:13  l3agent    WARNING  Retry 1: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:16  l3agent    WARNING  Retry 2: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:19  l3agent    WARNING  Retry 3: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:22  l3agent    WARNING  Retry 4: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:25  l3agent    WARNING  Retry 5: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:28  l3agent    WARNING  Retry 6: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:31  l3agent    WARNING  Retry 7: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:34  l3agent    WARNING  Retry 8: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:37  l3agent    WARNING  Retry 9: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:40  l3agent    WARNING  Retry 10: The router 7096df0b-e981-4d85-b941-2817546271d3 is not hosted by L3 agent 23309caf-f3cf-493f-ab21-ed68ecdcd794.
2014-03-12 17:22:43  rpcdaemon  INFO     Stopping worker...
2014-03-12 17:22:48  rpcdaemon  WARNING  Error stopping worker. Shutting down uncleanly.
2014-03-12 17:22:48  rpcdaemon  INFO     Stopped.
Traceback (most recent call last):
  File "/usr/bin/rpcdaemon", line 9, in <module>
    load_entry_point('RPCDaemon==1.0.1', 'console_scripts', 'rpcdaemon')()
  File "/usr/lib/python2.6/site-packages/rpcdaemon/__init__.py", line 166, in main
    monitor.check()
  File "/usr/lib/python2.6/site-packages/rpcdaemon/__init__.py", line 145, in check
    plugin.check()
  File "/usr/lib/python2.6/site-packages/rpcdaemon/lib/neutronagent.py", line 151, in check
    self.handle(agent, False)
  File "/usr/lib/python2.6/site-packages/rpcdaemon/plugins/l3agent.py", line 87, in handle
    lambda: self.client.remove_router_from_l3_agent(agent['id'],
  File "/usr/lib/python2.6/site-packages/rpcdaemon/lib/neutronagent.py", line 186, in retryable
    on_fail(exc_info)
  File "/usr/lib/python2.6/site-packages/rpcdaemon/lib/neutronagent.py", line 189, in fail
    self.logger.debug('Retry exception info: {} {} {}'.format(exc_info[0], exc_info[1], exc_info[2]))
ValueError: zero length field name in format
breu commented 10 years ago

This is after the following:

NodeA and NodeB are up Stop quantum-l3-agent on NodeB and move all routers to Node A Start quantum-l3-agent on NodeB ...wait... Stop quantum-l3-agent on NodeA stack trace