search

F5Networks / f5-openstack-agent

The F5 Agent for OpenStack Neutron allows you to deploy BIG-IP services in an OpenStack environment.
http://clouddocs.f5.com/products/openstack/agent/latest
Apache License 2.0
14 stars 38 forks source link

lbaas loadbalancer error state after update in pool's --lb-algorithm option #535

Open ssorenso opened 7 years ago

ssorenso commented 7 years ago

Agent Version

$ pip list | grep f5 DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning. f5-icontrol-rest (1.1.0) f5-openstack-agent (9.2.0b1) f5-openstack-lbaasv2-driver (9.2.0b1) f5-sdk (2.1.0)

Operating System

3.10.0-327.10.1.el7.x86_64 #1 SMP Tue Feb 16 17:03:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

OpenStack Release

Neutron/Mitaka

Description

The issue here is recreated with the following:

  1. Create a subnet of normal size with the same tenant ID as network
  2. Create a loadbalancer on that subnet
  3. Create a listener on that subnet
  4. Create a pool using that listener and loadbalancer with --lb-algorithm ROUND_ROBIN
  5. Create members on the pool that has the same IP assignment as the loadbalancer

NOTE: the loadbalancer's IP appears to be random upon creation within the pool. I'm sure it's not really 'random'; however, the first time I recreated this issue, it was assigned to .3, and the second it was assigned to .4 for the last octet in the subnet. The discovery of the traceback made this clear that it was an unhandled exception for the REST API (see comments).

The second re-creation of this issue prompted me to create another bug that I've now closed as a duplicate (issue 536). The attached data for that duplication is: lbaas_loadbalancer_update_bug.zip

Recovery:

  1. Delete the pool
  2. List the loadbalancers

You should see that the loadbalancer has returned to an Active state.

Deployment

General Test lab with Mitaka, node count 2, and undercloud.

ssorenso commented 7 years ago

The following traceback was found: 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver Traceback (most recent call last): 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/icontrol_driver.py", line 1213, in _common_service_handler 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver all_subnet_hints) 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py", line 69, in assure_service 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver self._assure_members(service, all_subnet_hints) 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py", line 255, in _assure_members 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver raise f5_ex.MemberCreationException(err.message) 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver MemberCreationException: 400 Unexpected Error: Bad Request for uri: https://10.190.7.138:443/mgmt/tm/ltm/pool/~Project_05fdde8c070441e9abd8c629cb229a05~Project_05237fec-30dd-405e-a241-7ffa297ec770/members/ 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver Text: u'{"code":400,"message":"01070080:3: The requested pool member is already in use as a self IP address (10.33.32.4%1).","errorStack":[]}' 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver 2017-01-16 13:22:11.892 22591 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver [req-f55b6198-98c1-4744-8c58-aaa2dc44cc5f c7828ba8aed44578b67126c743c4bb0c 05fdde8c070441e9abd8c629cb229a05 - - -] _update_service_status _update_service_status /usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/icontrol_driver.py:1237 2017-01-16 13:22:11.893 22591 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver [req-f55b6198-98c1-4744-8c58-aaa2dc44cc5f c7828ba8aed44578b67126c743c4bb0c 05fdde8c070441e9abd8c629cb229a05 - - -] f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver.iControlDriver method _update_pool_status called with arguments ([{u'lb_algorithm': u'ROUND_ROBIN', u'listener_id': u'ab0adf48-b498-4771-9efb-8e2601cef59c', u'protocol': u'HTTPS', u'description': u'', u'provisioning_status': u'ACTIVE', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'l7_policies': [], u'loadbalancer_id': u'08bde970-ea8b-4f0f-9fc2-8790c7019048', u'healthmonitor_id': None, u'listeners': [{u'id': u'ab0adf48-b498-4771-9efb-8e2601cef59c'}], u'members': [{u'id': u'1d449c59-e2ac-4cb6-b6d9-6098ecad3bce'}, {u'id': u'c4e00f1d-01ae-4edf-890b-519b3a72d16d'}, {u'id': u'66af9b62-0ab8-4f19-b7e3-f3508934067a'}], u'admin_state_up': True, u'sessionpersistence': None, u'id': u'05237fec-30dd-405e-a241-7ffa297ec770', u'operating_status': u'ONLINE', u'name': u'test_pool'}],) {} wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:45 2017-01-16 13:22:11.894 22591 DEBUG f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver [req-f55b6198-98c1-4744-8c58-aaa2dc44cc5f c7828ba8aed44578b67126c743c4bb0c 05fdde8c070441e9abd8c629cb229a05 - - -] f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver.iControlDriver method _update_listener_status called with arguments ({u'subnets': {u'4cfecb9c-9059-42b8-b494-4223f75c6b5f': {u'description': u'', u'enable_dhcp': True, u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'created_at': u'2017-01-16T20:21:17', u'dns_nameservers': [], u'updated_at': u'2017-01-16T20:21:17', u'gateway_ip': u'10.33.32.1', u'ipv6_ra_mode': None, u'allocation_pools': [{u'start': u'10.33.32.2', u'end': u'10.33.32.5'}], u'host_routes': [], u'shared': False, u'ip_version': 4, u'ipv6_address_mode': None, u'cidr': u'10.33.32.0/29', u'id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'subnetpool_id': None, u'name': u'test_net'}}, u'listeners': [{u'l7_policies': [], u'protocol': u'HTTPS', u'description': u'', u'default_tls_container_id': None, 'snat_pool_name': u'Project_05fdde8c070441e9abd8c629cb229a05', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'admin_state_up': True, u'connection_limit': -1, u'id': u'ab0adf48-b498-4771-9efb-8e2601cef59c', 'use_snat': True, u'protocol_port': 8080, u'sni_containers': [], u'provisioning_status': u'ACTIVE', u'default_pool_id': u'05237fec-30dd-405e-a241-7ffa297ec770', u'loadbalancer_id': u'08bde970-ea8b-4f0f-9fc2-8790c7019048', u'operating_status': 'ONLINE', u'name': u'test_listener'}], u'healthmonitors': [], u'members': [{u'name': u'', u'weight': 1, u'admin_state_up': True, u'subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'provisioning_status': u'ACTIVE', u'pool_id': u'05237fec-30dd-405e-a241-7ffa297ec770', u'id': u'1d449c59-e2ac-4cb6-b6d9-6098ecad3bce', u'gre_vteps': [], u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'address': u'10.33.32.4%1', u'protocol_port': 8080, u'port': {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2017-01-16T20:21:26', u'device_owner': u'network:f5lbaasv2', u'binding:profile': {}, u'fixed_ips': [{u'subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'ip_address': u'10.33.32.4'}], u'id': u'afd8a23a-c0f2-4868-851e-4717079c0cb8', u'security_groups': [], u'binding:vif_details': {}, u'binding:vif_type': u'binding_failed', u'mac_address': u'fa:16:3e:90:f5:61', u'status': u'ACTIVE', u'binding:host_id': u'host-141.int.lineratesystems.com:8c8ace09-59e1-5105-a364-ed3e10f9287b', u'description': None, u'device_id': u'8e8cb29f-d011-5493-8cdc-93397076ac90', u'name': u'local-bigip1-4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'admin_state_up': True, u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'dns_name': None, u'created_at': u'2017-01-16T20:21:26', u'binding:vnic_type': u'normal', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05'}, u'operating_status': u'ONLINE', u'vxlan_vteps': [u'201.0.139.10']}, {u'name': u'', u'weight': 1, u'admin_state_up': True, u'subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'provisioning_status': u'ACTIVE', u'pool_id': u'05237fec-30dd-405e-a241-7ffa297ec770', u'id': u'66af9b62-0ab8-4f19-b7e3-f3508934067a', u'gre_vteps': [], u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'address': u'10.33.32.3%1', u'protocol_port': 8080, u'port': {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2017-01-16T20:21:25', u'device_owner': u'network:f5lbaasv2', u'binding:profile': {}, u'fixed_ips': [{u'subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'ip_address': u'10.33.32.3'}], u'id': u'7ced9869-0b53-48d1-9aa5-6825464b50fb', u'security_groups': [u'b1a846c7-65cd-4954-9db6-435d52bb1dc4'], u'binding:vif_details': {}, u'binding:vif_type': u'binding_failed', u'mac_address': u'fa:16:3e:84:be:4e', u'status': u'DOWN', u'binding:host_id': u'host-141.int.lineratesystems.com:8c8ace09-59e1-5105-a364-ed3e10f9287b', u'description': None, u'device_id': u'8e8cb29f-d011-5493-8cdc-93397076ac90', u'name': u'loadbalancer-08bde970-ea8b-4f0f-9fc2-8790c7019048', u'admin_state_up': True, u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'dns_name': None, u'created_at': u'2017-01-16T20:21:24', u'binding:vnic_type': u'normal', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05'}, u'operating_status': u'ONLINE', u'vxlan_vteps': [u'201.0.139.10']}, {u'name': u'', u'weight': 1, u'admin_state_up': True, u'subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'provisioning_status': u'ACTIVE', u'pool_id': u'05237fec-30dd-405e-a241-7ffa297ec770', u'id': u'c4e00f1d-01ae-4edf-890b-519b3a72d16d', u'gre_vteps': [], u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'address': u'10.33.32.5%1', u'protocol_port': 8080, u'port': {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2017-01-16T20:21:27', u'device_owner': u'network:f5lbaasv2', u'binding:profile': {}, u'fixed_ips': [{u'subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'ip_address': u'10.33.32.5'}], u'id': u'cfb698a9-b968-4286-a13e-a409de3d6417', u'security_groups': [], u'binding:vif_details': {}, u'binding:vif_type': u'binding_failed', u'mac_address': u'fa:16:3e:28:e7:a4', u'status': u'ACTIVE', u'binding:host_id': u'host-141.int.lineratesystems.com:8c8ace09-59e1-5105-a364-ed3e10f9287b', u'description': None, u'device_id': u'8e8cb29f-d011-5493-8cdc-93397076ac90', u'name': u'snat-traffic-group-local-only-4cfecb9c-9059-42b8-b494-4223f75c6b5f_0', u'admin_state_up': True, u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'dns_name': None, u'created_at': u'2017-01-16T20:21:27', u'binding:vnic_type': u'normal', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05'}, u'operating_status': u'ONLINE', u'vxlan_vteps': [u'201.0.139.10']}], u'l7policy_rules': [], u'pools': [{u'lb_algorithm': u'ROUND_ROBIN', u'listener_id': u'ab0adf48-b498-4771-9efb-8e2601cef59c', u'protocol': u'HTTPS', u'description': u'', u'provisioning_status': u'ACTIVE', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'l7_policies': [], u'loadbalancer_id': u'08bde970-ea8b-4f0f-9fc2-8790c7019048', u'healthmonitor_id': None, u'listeners': [{u'id': u'ab0adf48-b498-4771-9efb-8e2601cef59c'}], u'members': [{u'id': u'1d449c59-e2ac-4cb6-b6d9-6098ecad3bce'}, {u'id': u'c4e00f1d-01ae-4edf-890b-519b3a72d16d'}, {u'id': u'66af9b62-0ab8-4f19-b7e3-f3508934067a'}], u'admin_state_up': True, u'sessionpersistence': None, u'id': u'05237fec-30dd-405e-a241-7ffa297ec770', u'operating_status': u'ONLINE', u'name': u'test_pool'}], 'traffic_group': u'/Common/traffic-group-1', u'l7policies': [], u'networks': {u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350': {u'provider:physical_network': None, u'updated_at': u'2017-01-16T17:18:00', u'mtu': 1450, u'id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'router:external': False, u'availability_zone_hints': [], u'availability_zones': [u'nova'], u'ipv4_address_scope': None, u'shared': False, u'status': u'ACTIVE', u'subnets': [u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'811206b0-f18c-467a-802e-1942056844b3', u'60beaede-3b02-4db2-9763-16f0a830990b'], u'description': u'', u'tags': [], u'ipv6_address_scope': None, 'route_domain_id': 1, u'provider:segmentation_id': 64, u'name': u'testlab-client-network', u'admin_state_up': True, u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'created_at': u'2017-01-16T17:18:00', u'provider:network_type': u'vxlan', u'vlan_transparent': None}}, u'loadbalancer': {u'vxlan_vteps': [u'201.0.140.1', u'201.0.141.1', u'201.0.142.1', u'201.0.139.10'], u'name': u'foob', u'provisioning_status': 'ERROR', u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05', u'admin_state_up': True, u'provider': u'f5networks', u'id': u'08bde970-ea8b-4f0f-9fc2-8790c7019048', u'gre_vteps': [], u'pools': [{u'id': u'05237fec-30dd-405e-a241-7ffa297ec770'}], u'listeners': [{u'id': u'ab0adf48-b498-4771-9efb-8e2601cef59c'}], u'vip_port_id': u'7ced9869-0b53-48d1-9aa5-6825464b50fb', u'vip_address': u'10.33.32.3%1', u'vip_subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', 'traffic_group': u'traffic-group-1', u'vip_port': {u'allowed_address_pairs': [], u'extra_dhcp_opts': [], u'updated_at': u'2017-01-16T20:21:25', u'device_owner': u'network:f5lbaasv2', u'binding:profile': {}, u'fixed_ips': [{u'subnet_id': u'4cfecb9c-9059-42b8-b494-4223f75c6b5f', u'ip_address': u'10.33.32.3'}], u'id': u'7ced9869-0b53-48d1-9aa5-6825464b50fb', u'security_groups': [u'b1a846c7-65cd-4954-9db6-435d52bb1dc4'], u'binding:vif_details': {}, u'binding:vif_type': u'binding_failed', u'mac_address': u'fa:16:3e:84:be:4e', u'status': u'DOWN', u'binding:host_id': u'host-141.int.lineratesystems.com:8c8ace09-59e1-5105-a364-ed3e10f9287b', u'description': None, u'device_id': u'8e8cb29f-d011-5493-8cdc-93397076ac90', u'name': u'loadbalancer-08bde970-ea8b-4f0f-9fc2-8790c7019048', u'admin_state_up': True, u'network_id': u'38f89e4e-ca17-4de0-89dd-7e6af4e5a350', u'dns_name': None, u'created_at': u'2017-01-16T20:21:24', u'binding:vnic_type': u'normal', u'tenant_id': u'05fdde8c070441e9abd8c629cb229a05'}, u'operating_status': u'OFFLINE', u'description': u''}},) {} wrapper /usr/lib/python2.7/site-packages/oslo_log/helpers.py:45

Looking at specifically: 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver MemberCreationException: 400 Unexpected Error: Bad Request for uri: https://10.190.7.138:443/mgmt/tm/ltm/pool/~Project_05fdde8c070441e9abd8c629cb229a05~Project_05237fec-30dd-405e-a241-7ffa297ec770/members/ 2017-01-16 13:22:11.891 22591 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver Text: u'{"code":400,"message":"01070080:3: The requested pool member is already in use as a self IP address (10.33.32.4%1).","errorStack":[]}'

This would imply that:

ssorenso commented 7 years ago

This issue makes it so that auto-population from a script written by a user cannot be executed due to the fact that members have to be manually assigned an IP address at creation, and the loadbalancer is auto-assigned an IP address pseudo-randomly. Thus, a user would have to create the logic to assign the members IP addresses differently post-creation of the loadbalancer...