search

F5Networks / f5-openstack-agent

The F5 Agent for OpenStack Neutron allows you to deploy BIG-IP services in an OpenStack environment.
http://clouddocs.f5.com/products/openstack/agent/latest
Apache License 2.0
14 stars 38 forks source link

LBaaSv2 agent not registering with neutron -- Critical errors shown in log #893

Closed pjbreaux closed 7 years ago

pjbreaux commented 7 years ago

OpenStack Release

mitaka

Description

The lbaasv2 agent is not registering with neutron in the neutron agent-list upon installation and start of the process. The service status looks as though it is active:

[root@host-145 testlab(keystone_admin)]# systemctl status f5-openstack-agent
● f5-openstack-agent.service - F5 LBaaSv2 BIG-IP Agent
   Loaded: loaded (/usr/lib/systemd/system/f5-openstack-agent.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-08-29 15:39:28 MDT; 1s ago
 Main PID: 24924 (coverage)
   CGroup: /system.slice/f5-openstack-agent.service
           └─24924 /usr/bin/python2 /usr/bin/coverage run --rcfile=/home/testlab/coverage/.coveragerc /usr/bin/f5-oslbaasv2-agent --log-file /var/log/neutron/f5-openstack-agent.log --con...

Aug 29 15:39:28 host-145.int.lineratesystems.com systemd[1]: Started F5 LBaaSv2 BIG-IP Agent.
Aug 29 15:39:28 host-145.int.lineratesystems.com systemd[1]: Starting F5 LBaaSv2 BIG-IP Agent...
Aug 29 15:39:29 host-145.int.lineratesystems.com coverage[24924]: Option "verbose" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
[root@host-145 testlab(keystone_admin)]#

But there is no entry for agent in the neutron agent-list:

[testlab@host-145 ~(keystone_admin)]$ neutron agent-list
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                             | availability_zone | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+----------------+---------------------------+
| 15d87102-07c7-4452-b349-19faa4c72bc6 | Loadbalancer agent | host-145.int.lineratesystems.com |                   | xxx   | True           | neutron-lbaas-agent       |
| 214cc650-f270-4e63-b723-f9f2008409bd | Open vSwitch agent | host-144.int.lineratesystems.com |                   | :-)   | True           | neutron-openvswitch-agent |
| 2980729e-66f0-439a-ba83-09c45ac3370a | Open vSwitch agent | host-145.int.lineratesystems.com |                   | :-)   | True           | neutron-openvswitch-agent |
| 412bf0dc-2003-4bcc-92a7-d8105c5536d0 | DHCP agent         | host-145.int.lineratesystems.com | nova              | :-)   | True           | neutron-dhcp-agent        |
| 6de0ccfe-da29-4111-96cf-2aa96b76bcc2 | L3 agent           | host-145.int.lineratesystems.com | nova              | :-)   | True           | neutron-l3-agent          |
| 92228f9b-aa37-41de-a062-f9c4e34cbf0e | Open vSwitch agent | host-143.int.lineratesystems.com |                   | :-)   | True           | neutron-openvswitch-agent |
| f92ab826-0b3e-47e6-aafd-1f8f9fa1fe9f | Metadata agent     | host-145.int.lineratesystems.com |                   | :-)   | True           | neutron-metadata-agent    |
+--------------------------------------+--------------------+----------------------------------+-------------------+-------+----------------+---------------------------+

In the agent log, I see repeated critical and error messages indicating no lbdriver attribute on the LbaasAgentManager object. These error messages repeat for a time, then disappear.

2017-08-29 15:33:51.435 22270 INFO neutron.common.config [-] Logging enabled!
2017-08-29 15:33:51.436 22270 INFO neutron.common.config [-] /usr/bin/f5-oslbaasv2-agent version 8.3.0
2017-08-29 15:33:51.436 22270 DEBUG neutron.common.config [-] command line: /usr/bin/f5-oslbaasv2-agent --log-file /var/log/neutron/f5-openstack-agent.log --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/services/f5/f5-openstack-agent.ini setup_logging /usr/lib/python2.7/site-packages/neutron/common/config.py:269
2017-08-29 15:33:51.437 22270 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] Initializing LbaasAgentManager __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:233
2017-08-29 15:33:51.437 22270 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] runtime environment: 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:234
2017-08-29 15:33:51.438 22270 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-d82e4b59-7dc1-4443-994a-f5c7d0d0d5b8 - - - - -] Initializing LogicalServiceCache __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:164
2017-08-29 15:33:51.438 22270 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-d82e4b59-7dc1-4443-994a-f5c7d0d0d5b8 - - - - -] setting service resync intervl to 300 seconds __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:255
2017-08-29 15:33:51.439 22270 CRITICAL neutron [req-d82e4b59-7dc1-4443-994a-f5c7d0d0d5b8 - - - - -] AttributeError: 'LbaasAgentManager' object has no attribute 'lbdriver'
2017-08-29 15:33:51.439 22270 ERROR neutron Traceback (most recent call last):
2017-08-29 15:33:51.439 22270 ERROR neutron   File "/usr/bin/f5-oslbaasv2-agent", line 11, in <module>
2017-08-29 15:33:51.439 22270 ERROR neutron     load_entry_point('f5-openstack-agent==9.3.3b2', 'console_scripts', 'f5-oslbaasv2-agent')()
2017-08-29 15:33:51.439 22270 ERROR neutron   File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent.py", line 80, in main
2017-08-29 15:33:51.439 22270 ERROR neutron     mgr = manager.LbaasAgentManager(cfg.CONF)
2017-08-29 15:33:51.439 22270 ERROR neutron   File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py", line 266, in __init__
2017-08-29 15:33:51.439 22270 ERROR neutron     '.' + self.lbdriver.hostnames[0])
2017-08-29 15:33:51.439 22270 ERROR neutron AttributeError: 'LbaasAgentManager' object has no attribute 'lbdriver'
2017-08-29 15:33:51.439 22270 ERROR neutron
2017-08-29 15:33:53.329 22356 INFO neutron.common.config [-] Logging enabled!
2017-08-29 15:33:53.330 22356 INFO neutron.common.config [-] /usr/bin/f5-oslbaasv2-agent version 8.3.0
2017-08-29 15:33:53.330 22356 DEBUG neutron.common.config [-] command line: /usr/bin/f5-oslbaasv2-agent --log-file /var/log/neutron/f5-openstack-agent.log --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/services/f5/f5-openstack-agent.ini setup_logging /usr/lib/python2.7/site-packages/neutron/common/config.py:269
2017-08-29 15:33:53.330 22356 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] Initializing LbaasAgentManager __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:233
2017-08-29 15:33:53.331 22356 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] runtime environment: 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:234
2017-08-29 15:33:53.331 22356 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-32eca3f5-86af-402c-99ef-a268d7b3333f - - - - -] Initializing LogicalServiceCache __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:164
2017-08-29 15:33:53.332 22356 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-32eca3f5-86af-402c-99ef-a268d7b3333f - - - - -] setting service resync intervl to 300 seconds __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:255
2017-08-29 15:33:53.332 22356 CRITICAL neutron [req-32eca3f5-86af-402c-99ef-a268d7b3333f - - - - -] AttributeError: 'LbaasAgentManager' object has no attribute 'lbdriver'
2017-08-29 15:33:53.332 22356 ERROR neutron Traceback (most recent call last):
2017-08-29 15:33:53.332 22356 ERROR neutron   File "/usr/bin/f5-oslbaasv2-agent", line 11, in <module>
2017-08-29 15:33:53.332 22356 ERROR neutron     load_entry_point('f5-openstack-agent==9.3.3b2', 'console_scripts', 'f5-oslbaasv2-agent')()
2017-08-29 15:33:53.332 22356 ERROR neutron   File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent.py", line 80, in main
2017-08-29 15:33:53.332 22356 ERROR neutron     mgr = manager.LbaasAgentManager(cfg.CONF)
2017-08-29 15:33:53.332 22356 ERROR neutron   File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py", line 266, in __init__
2017-08-29 15:33:53.332 22356 ERROR neutron     '.' + self.lbdriver.hostnames[0])
2017-08-29 15:33:53.332 22356 ERROR neutron AttributeError: 'LbaasAgentManager' object has no attribute 'lbdriver'
2017-08-29 15:33:53.332 22356 ERROR neutron
2017-08-29 15:33:55.102 22420 INFO neutron.common.config [-] Logging enabled!
2017-08-29 15:33:55.102 22420 INFO neutron.common.config [-] /usr/bin/f5-oslbaasv2-agent version 8.3.0
2017-08-29 15:33:55.103 22420 DEBUG neutron.common.config [-] command line: /usr/bin/f5-oslbaasv2-agent --log-file /var/log/neutron/f5-openstack-agent.log --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/services/f5/f5-openstack-agent.ini setup_logging /usr/lib/python2.7/site-packages/neutron/common/config.py:269
2017-08-29 15:33:55.103 22420 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] Initializing LbaasAgentManager __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:233
2017-08-29 15:33:55.103 22420 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] runtime environment: 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:234
2017-08-29 15:33:55.104 22420 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-f46fdc9c-884a-4881-a20b-d19f5e3cfaba - - - - -] Initializing LogicalServiceCache __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:164
2017-08-29 15:33:55.104 22420 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-f46fdc9c-884a-4881-a20b-d19f5e3cfaba - - - - -] setting service resync intervl to 300 seconds __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:255
2017-08-29 15:33:55.105 22420 CRITICAL neutron [req-f46fdc9c-884a-4881-a20b-d19f5e3cfaba - - - - -] AttributeError: 'LbaasAgentManager' object has no attribute 'lbdriver'
2017-08-29 15:33:55.105 22420 ERROR neutron Traceback (most recent call last):
2017-08-29 15:33:55.105 22420 ERROR neutron   File "/usr/bin/f5-oslbaasv2-agent", line 11, in <module>
2017-08-29 15:33:55.105 22420 ERROR neutron     load_entry_point('f5-openstack-agent==9.3.3b2', 'console_scripts', 'f5-oslbaasv2-agent')()
2017-08-29 15:33:55.105 22420 ERROR neutron   File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent.py", line 80, in main
2017-08-29 15:33:55.105 22420 ERROR neutron     mgr = manager.LbaasAgentManager(cfg.CONF)
2017-08-29 15:33:55.105 22420 ERROR neutron   File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py", line 266, in __init__
2017-08-29 15:33:55.105 22420 ERROR neutron     '.' + self.lbdriver.hostnames[0])
2017-08-29 15:33:55.105 22420 ERROR neutron AttributeError: 'LbaasAgentManager' object has no attribute 'lbdriver'
2017-08-29 15:33:55.105 22420 ERROR neutron

After the errors disappear, the agent log looks like such. Note that the /usr/bin/f5-oslbaasv2-agent version is 8.3.0. This seems to be a reference to the neutron-lbaas version, which is 8.3.0, not of our own agent version, which differs from OpenStack by one.

2017-08-29 15:44:24.130 26843 INFO neutron.common.config [-] Logging enabled!
2017-08-29 15:44:24.131 26843 INFO neutron.common.config [-] /usr/bin/f5-oslbaasv2-agent version 8.3.0
2017-08-29 15:44:24.131 26843 DEBUG neutron.common.config [-] command line: /usr/bin/f5-oslbaasv2-agent --log-file /var/log/neutron/f5-openstack-agent.log --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/services/f5/f5-openstack-agent.ini setup_logging /usr/lib/python2.7/site-packages/neutron/common/config.py:269
2017-08-29 15:44:24.131 26843 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] Initializing LbaasAgentManager __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:233
2017-08-29 15:44:24.132 26843 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] runtime environment: 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:234
2017-08-29 15:44:24.132 26843 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-2b691f19-32a7-4afd-8d64-dc805da58923 - - - - -] Initializing LogicalServiceCache __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:164
2017-08-29 15:44:24.133 26843 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-2b691f19-32a7-4afd-8d64-dc805da58923 - - - - -] setting service resync intervl to 300 seconds __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:255
2017-08-29 15:44:26.380 26858 INFO neutron.common.config [-] Logging enabled!
2017-08-29 15:44:26.381 26858 INFO neutron.common.config [-] /usr/bin/f5-oslbaasv2-agent version 8.3.0
2017-08-29 15:44:26.381 26858 DEBUG neutron.common.config [-] command line: /usr/bin/f5-oslbaasv2-agent --log-file /var/log/neutron/f5-openstack-agent.log --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/services/f5/f5-openstack-agent.ini setup_logging /usr/lib/python2.7/site-packages/neutron/common/config.py:269
2017-08-29 15:44:26.382 26858 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] Initializing LbaasAgentManager __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:233
2017-08-29 15:44:26.382 26858 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] runtime environment: 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:234
2017-08-29 15:44:26.383 26858 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-1689a240-00f5-4914-b65d-d2fb494d9d13 - - - - -] Initializing LogicalServiceCache __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:164
2017-08-29 15:44:26.383 26858 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-1689a240-00f5-4914-b65d-d2fb494d9d13 - - - - -] setting service resync intervl to 300 seconds __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:255
2017-08-29 15:44:28.585 26871 INFO neutron.common.config [-] Logging enabled!
2017-08-29 15:44:28.585 26871 INFO neutron.common.config [-] /usr/bin/f5-oslbaasv2-agent version 8.3.0
2017-08-29 15:44:28.586 26871 DEBUG neutron.common.config [-] command line: /usr/bin/f5-oslbaasv2-agent --log-file /var/log/neutron/f5-openstack-agent.log --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/services/f5/f5-openstack-agent.ini setup_logging /usr/lib/python2.7/site-packages/neutron/common/config.py:269
2017-08-29 15:44:28.586 26871 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] Initializing LbaasAgentManager __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:233
2017-08-29 15:44:28.587 26871 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [-] runtime environment: 2.7.5 (default, Nov 20 2015, 02:00:19)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:234
2017-08-29 15:44:28.587 26871 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-7a1ebffa-dc8b-4b7b-9ede-60bdb2ad19cf - - - - -] Initializing LogicalServiceCache __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:164
2017-08-29 15:44:28.588 26871 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.agent_manager [req-7a1ebffa-dc8b-4b7b-9ede-60bdb2ad19cf - - - - -] setting service resync intervl to 300 seconds __init__ /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/agent_manager.py:255

Agent version and neutron-lbaas version. As far as I know, this problem has always existed though.

f5-icontrol-rest (1.3.0)
f5-openstack-agent (9.3.3b2)
f5-openstack-lbaasv2-driver (9.3.3b2)
f5-sdk (2.3.3)
[root@host-145 testlab(keystone_admin)]# pip list | grep neutorn
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
[root@host-145 testlab(keystone_admin)]# pip list | grep neutron
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
neutron (8.3.0)
neutron-lbaas (8.3.0)
neutron-lib (0.0.3)
python-neutronclient (4.1.2)
[root@host-145 testlab(keystone_admin)]#

I do see the driver initialization in the neutron server.log, but of course there's no RPC communication with the agent.

2017-08-29 15:33:31.799 21904 DEBUG neutron_lbaas.drivers.f5.driver_v2 [-] F5LBaaSV2Driver: initializing, version=9.1.0, impl=9.3.3.b2, env=Project __init__ /usr/lib/python2.7/site-packages/neutron_lbaas/drivers/f5/driver_v2.py:53
2017-08-29 15:33:31.800 21904 DEBUG neutron.callbacks.manager [-] Subscribe: <function post_fork_callback_Project at 0x5eea848> process after_create subscribe /usr/lib/python2.7/site-packages/neutron/callbacks/manager.py:41
2017-08-29 15:33:31.800 21904 DEBUG neutron.services.service_base [-] Loaded 'neutron_lbaas.drivers.f5.driver_v2.F5LBaaSV2Driver' provider for service LOADBALANCERV2 load_drivers /usr/lib/python2.7/site-packages/neutron/services/service_base.py:80

Agent Version

mitaka

Operating System

CentOS 7

OpenStack Release

mitaka

dflanigan commented 7 years ago

PR #894 reverted the code that introduced this issue. Closing.