[STACK-2301] axapi retry when ACOS System busy

[ytsai-a10]

Description

• Severity Level Critical
• Issue Description In Active-Standby mode, ACOS device may busy when vcs is doing configuration sync, negotiation or reloading. And this may happens at any time, Therefore, there is no way to prevent it in a10-octavia. So, it is better we handle "System is busy" issue in acos-client.

Jira Ticket

https://a10networks.atlassian.net/browse/STACK-2301

Technical Approach

In active-standby mode, after reload/reboot, vcs reload or vcs configuration sync. check device is up and vcs ready after sleep.

Config Changes

[vthunder]
default_vthunder_username = "admin"
default_vthunder_password = "a10"
default_axapi_version = "30"
#default_axapi_timeout = 600

[a10_controller_worker]
amp_image_owner_id = 99c9c2304f114685a32db30769c8a7e2
amp_secgroup_list = 2d0a3480-7f08-4184-b96f-c39bb444dd38
amp_flavor_id = 69995a83-c47a-427f-bf70-e9d9752aa915
amp_boot_network_list = 7f8ddd7f-c5c3-463a-b1c3-596f988480a4
amp_ssh_key_name = octavia_ssh_key
network_driver = a10_octavia_neutron_driver
workers = 2
amp_active_retries = 150
amp_active_wait_sec = 10
amp_busy_wait_sec = 0
amp_image_id = 0a845873-bd91-4f8c-ad58-cf5d5afde09c
#loadbalancer_topology = SINGLE
loadbalancer_topology = ACTIVE_STANDBY

[a10_health_manager]
udp_server_ip_address = 10.64.28.68
bind_port = 5550
bind_ip = 10.64.28.68
heartbeat_interval = 5
heartbeat_key = insecure
heartbeat_timeout = 90
health_check_interval = 3
failover_timeout = 600
health_check_timeout = 3
health_check_max_retries = 5

[a10_house_keeping]
load_balancer_expiry_age = 3600
amphorae_expiry_age = 3600
#use_periodic_write_memory = 'enable'
#write_mem_interval = 300

[a10_global]
network_type = "flat"
#network_type = "vlan"
vrid_floating_ip = "dhcp"

Test Cases

• create 2 LB with different subnet at the same time

Manual Testing

openstack loadbalancer create --vip-subnet-id tp91 --name vip1
openstack loadbalancer create --vip-subnet-id tp92 --name vip2