[STACK-2313] backup plug interface failed because of reloading

Description

If Feature Addition:

Required: Related user story

If Bug Fix:

Required: Severity Level High
Required: Issue Description
1. check the journal log on QA's server, when a10-octavia try to plug interface and reload on backup vthunder (i.e. AmphoraePostVIPPlug()), the backup vthunder is not up (maybe reloaded by configuration sync. or vcs). So, acos-client failed to connect ot vthunder and raise ConnectionError.

Journal log:

May 10 18:22:50 qa-109 a10-octavia-worker[28974]: DEBUG acos_client.v30.axapi_http [-] axapi_http: POST url = /axapi/v3/auth {{(pid=29014) request_impl /opt/stack/acos-client/acos_client/v30/axapi_http.py:76}}
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: DEBUG acos_client.v30.axapi_http [-] axapi_http: params = {
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:     "credentials": {
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:         "username": "********",
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:         "password": "********"
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:     }
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: } {{(pid=29014) request_impl /opt/stack/acos-client/acos_client/v30/axapi_http.py:77}}
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: DEBUG acos_client.v30.axapi_http [-] axapi_http: params + axapi_args = {
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:     "credentials": {
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:         "username": "********",
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:         "password": "********"
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:     }
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: } {{(pid=29014) request_impl /opt/stack/acos-client/acos_client/v30/axapi_http.py:86}}
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: DEBUG acos_client.v30.axapi_http [-] axapi_http: params_all = {'credentials': {'username': '********', 'password': '********'}} {{(pid=29014) request_impl /opt/stack/acos-client/acos_clie
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: DEBUG acos_client.v30.axapi_http [-] axapi_http: headers = {
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:     "Content-type": "application/json",
May 10 18:22:50 qa-109 a10-octavia-worker[28974]:     "User-Agent": "ACOS-Client-AGENT-2.6.1"
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: } {{(pid=29014) request_impl /opt/stack/acos-client/acos_client/v30/axapi_http.py:108}}
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: WARNING urllib3.connectionpool [-] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.Verif
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: WARNING urllib3.connectionpool [-] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.Verif
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: WARNING urllib3.connectionpool [-] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.Verif
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR acos_client.v30.axapi_http [-] acos_client failing with error ConnectionError after 3 retries: ConnectionError: HTTPSConnectionPool(host='192.168.0.228', port=443): Max retries exce
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks [-] Failed to save configuration and reboot on vThunder for amphora id: c39202ad-da29-436b-8697-14d1332d8910: ConnectionError: HTT
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks Traceback (most recent call last):
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/a10-octavia/a10_octavia/controller/worker/tasks/vthunder_tasks.py", line 117, in execute
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     self.axapi_client.system.action.write_memory()
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/action.py", line 37, in write_memory
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     self._post("/write/memory/", payload, **kwargs)
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/base.py", line 73, in _post
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     axapi_args=axapi_args, **kwargs)
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/base.py", line 44, in _request
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     return self.client.http.request(method, self.url(action), params, self.auth_header,
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/base.py", line 35, in url
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     self.auth_header['Authorization'] = "A10 %s" % self.client.session.id
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/session.py", line 28, in id
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     self.authenticate(self.username, self.password)
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/session.py", line 43, in authenticate
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     r = self.http.post(url, payload)
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/axapi_http.py", line 195, in post
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     timeout=timeout, axapi_args=axapi_args, **kwargs)
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks   File "/opt/stack/acos-client/acos_client/v30/axapi_http.py", line 187, in request
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks     raise e
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks ConnectionError: HTTPSConnectionPool(host='192.168.0.228', port=443): Max retries exceeded with url: /axapi/v3/auth (Caused by New
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: ERROR a10_octavia.controller.worker.tasks.vthunder_tasks
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: WARNING a10_octavia.controller.worker.controller_worker [-] Task 'amphorae-post-vip-plug-for-backup' (2a0fc24e-1ed7-4e52-8f31-26218992d275) transitioned into state 'FAILURE' from state 'R
May 10 18:22:50 qa-109 a10-octavia-worker[28974]: 128 predecessors (most recent first):

Jira Ticket

https://a10networks.atlassian.net/browse/STACK-2313

Technical Approach

a10-octavia side PR: https://github.com/a10networks/a10-octavia/pull/397 In acos-client side also make the axapi request more reliable. And retry on ConnectionError.

Config Changes

[vthunder]
default_vthunder_username = "admin"
default_vthunder_password = "a10"
default_axapi_version = "30"
#default_axapi_timeout = 600

[a10_controller_worker]
amp_image_owner_id = 99c9c2304f114685a32db30769c8a7e2
amp_secgroup_list = 2d0a3480-7f08-4184-b96f-c39bb444dd38
amp_flavor_id = 69995a83-c47a-427f-bf70-e9d9752aa915
amp_boot_network_list = 7f8ddd7f-c5c3-463a-b1c3-596f988480a4
#amp_boot_network_list = 7f8ddd7f-c5c3-463a-b1c3-596f988480a4, 61305531-6e3a-44b4-8e67-05e93363dddc, a66685db-31ee-46c6-928a-ac034d87dc63, ec33a88d-3290-4ad6-95cc-226fcd2a4458
amp_ssh_key_name = octavia_ssh_key
network_driver = a10_octavia_neutron_driver
workers = 2
amp_active_retries = 150
amp_active_wait_sec = 10
amp_busy_wait_sec = 0
amp_image_id = 0a845873-bd91-4f8c-ad58-cf5d5afde09c
#loadbalancer_topology = SINGLE
loadbalancer_topology = ACTIVE_STANDBY

[a10_health_manager]
udp_server_ip_address = 10.64.28.68
bind_port = 5550
bind_ip = 10.64.28.68
heartbeat_interval = 5
heartbeat_key = insecure
heartbeat_timeout = 90
health_check_interval = 3
failover_timeout = 600
health_check_timeout = 3
health_check_max_retries = 5

[a10_house_keeping]
load_balancer_expiry_age = 3600
amphorae_expiry_age = 3600
#use_periodic_write_memory = 'enable'
#write_mem_interval = 300

[a10_global]
network_type = "flat"

Test Cases

pdb before backup AmphoraePostVIPPlug(), and reboot backup vthunder before continue pdb. (to simulate QA's situation)

commands:

openstack loadbalancer create --vip-subnet-id tp91 --name vip1
openstack loadbalancer create --vip-subnet-id tp92 --name vip2

Manual Testing

LOG:

stack@openstack-4:~/source/a10-octavia/vThunder$ openstack loadbalancer list
+--------------------------------------+------+----------------------------------+----------------+---------------------+----------+
| id                                   | name | project_id                       | vip_address    | provisioning_status | provider |
+--------------------------------------+------+----------------------------------+----------------+---------------------+----------+
| aed9caab-0f35-45d8-b2e0-45d54e962186 | vip1 | 99c9c2304f114685a32db30769c8a7e2 | 192.168.91.124 | ACTIVE              | a10      |
| fdf543c6-321b-4883-ba73-5e6032735cfb | vip2 | 99c9c2304f114685a32db30769c8a7e2 | 192.168.92.122 | ACTIVE              | a10      |
+--------------------------------------+------+----------------------------------+----------------+---------------------+----------+

a10networks / acos-client

[STACK-2313] backup plug interface failed because of reloading #345