Closed pjbreaux closed 7 years ago
It looks like the failure of this test leads into the failure of tests that are run after it. We must be more aggressive on the teardown of the tenant. Likely, this is due to a policy not being torn down. If we remove the policy, then we should be able to remove the tenant in most cases.
@dflanigan: this was the issue which is causing further test failures and I was wrong--it's not a test bug. I'm looking at the issue currently to provide a more well-rounded reconstruction of the problem.
For actual test issue, it appears that we are attempting to delete the policy before it has been disassociated from the virtual:
2017-02-07 22:19:26.758 55140 DEBUG root [req-f4036b27-b722-40fa-938a-90675ba5a332 833309c56e064131898d2e198425bd56 61345078b7414f9cad738455e6042c2e - - -] delete WITH uri: https://10.190.3.22:443/mgmt/tm/ltm/policy/~Project_61345078b7414f9cad738455e6042c2e~wrapper_policy/ AND suffix: AND kwargs: {} wrapper /usr/lib/python2.7/site-packages/icontrol/session.py:257
2017-02-07 22:19:26.768 55140 DEBUG root [req-f4036b27-b722-40fa-938a-90675ba5a332 833309c56e064131898d2e198425bd56 61345078b7414f9cad738455e6042c2e - - -] RESPONSE::STATUS: 400 Content-Type: application/json Content-Encoding: None
Text: u'{"code":400,"message":"01070635:3: The policy (/Project_61345078b7414f9cad738455e6042c2e/wrapper_policy) is referenced by one or more virtuals.","errorStack":[]}' wrapper /usr/lib/python2.7/site-packages/icontrol/session.py:265
2017-02-07 22:19:26.769 55140 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver [req-f4036b27-b722-40fa-938a-90675ba5a332 833309c56e064131898d2e198425bd56 61345078b7414f9cad738455e6042c2e - - -] 400 Unexpected Error: Bad Request for uri: https://10.190.3.22:443/mgmt/tm/ltm/policy/~Project_61345078b7414f9cad738455e6042c2e~wrapper_policy/
Text: u'{"code":400,"message":"01070635:3: The policy (/Project_61345078b7414f9cad738455e6042c2e/wrapper_policy) is referenced by one or more virtuals.","errorStack":[]}'
@jlongstaf: tested against 11.5.4 and did not see this issue. Could this be related to https://github.com/F5Networks/f5-openstack-agent/issues/577?
You said you tested against 11.6.0 and it failed?
This error is two fold, I believe. It is associated with the issue linked above in the agent. It is primarily failing due to the error below, when deploying a condition that has similar operands as an existing condition:
2017-05-23 18:00:23.602 22387 DEBUG root [req-f52b770e-be8d-4c16-8cd9-9e8377aea71d a76cc34e2a3b40e29fcefb123f818b54 cfceb9cf401544359c553b855eba5a46 - - -] patch WITH uri: https://10.190.7.238:443/mgmt/tm/ltm/policy/~Project_cfceb9cf40154
4359c553b855eba5a46~wrapper_policy/ AND suffix: AND kwargs: {'json': {'name': 'wrapper_policy', 'rules': [{'ordinal': 1, 'conditions': [{'name': '0', 'httpUri': True, 'request': True, 'endsWith': True, 'values': [u'real'], 'path': True},
{'name': '1', 'contains': True, 'request': True, 'tmName': u'X-HEADER', 'values': [u'es'], 'httpHeader': True}, {'startsWith': True, 'name': '2', 'httpUri': True, 'request': True, 'values': [u're'], 'path': True}, {'startsWith': True, 'n
ame': '3', 'request': True, 'host': True, 'values': [u'real'], 'httpHost': True}, {'name': '4', 'httpUri': True, 'contains': True, 'request': True, 'values': [u'al'], 'path': True}, {'name': '5', 'httpUri': True, 'contains': True, 'reques
t': True, 'values': [u'l'], 'path': True}], 'name': u'reject_1', 'actions': [{'reset': True, 'forward': True, 'request': True, 'name': '0'}]}], 'partition': u'Project_cfceb9cf401544359c553b855eba5a46', 'controls': ['forwarding'], 'strateg
y': 'first-match', 'requires': ['http']}} wrapper /usr/lib/python2.7/site-packages/icontrol/session.py:257
2017-05-23 18:00:23.635 22387 DEBUG root [req-f52b770e-be8d-4c16-8cd9-9e8377aea71d a76cc34e2a3b40e29fcefb123f818b54 cfceb9cf401544359c553b855eba5a46 - - -] RESPONSE::STATUS: 400 Content-Type: application/json Content-Encoding: None
Text: u'{"code":400,"message":"010716d8:3: Compilation of ltm policies for virtual server /Project_cfceb9cf401544359c553b855eba5a46/Project_2df9f8a8-ec9e-4cb5-a2f6-2953a38e1b5c failed; Failed to compile the combined policies.","errorStack
":[]}' wrapper /usr/lib/python2.7/site-packages/icontrol/session.py:265
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver [req-f52b770e-be8d-4c16-8cd9-9e8377aea71d a76cc34e2a3b40e29fcefb123f818b54 cfceb9cf401544359c553b855eba5a46 - - -] 400 Unexpected Error: Bad Requ
est for uri: https://10.190.7.238:443/mgmt/tm/ltm/policy/~Project_cfceb9cf401544359c553b855eba5a46~wrapper_policy/
Text: u'{"code":400,"message":"010716d8:3: Compilation of ltm policies for virtual server /Project_cfceb9cf401544359c553b855eba5a46/Project_2df9f8a8-ec9e-4cb5-a2f6-2953a38e1b5c failed; Failed to compile the combined policies.","errorStack
":[]}'
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver Traceback (most recent call last):
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/icontrol_driver.py", line 1250, in _common_service_handler
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver all_subnet_hints)
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py", line 63, in assure_service
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver self._assure_l7policies_created(service)
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver File "/usr/lib/python2.7/site-packages/f5_openstack_agent/lbaasv2/drivers/bigip/lbaas_builder.py", line 462, in _assure_l7policies_created
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver raise f5_ex.L7PolicyCreationException(err.message)
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver L7PolicyCreationException: 400 Unexpected Error: Bad Request for uri: https://10.190.7.238:443/mgmt/tm/ltm/policy/~Project_cfceb9cf401544359c553b
855eba5a46~wrapper_policy/
2017-05-23 18:00:23.636 22387 ERROR f5_openstack_agent.lbaasv2.drivers.bigip.icontrol_driver Text: u'{"code":400,"message":"010716d8:3: Compilation of ltm policies for virtual server /Project_cfceb9cf401544359c553b855eba5a46/Project_2df9f
8a8-ec9e-4cb5-a2f6-2953a38e1b5c failed; Failed to compile the combined policies.","errorStack":[]}'
This is an issue in the BIG-IP 11.6.0 version we are testing against (BIGIP-11.6.0.0.0.401). A hotfix is available that addresses my second point above. The recommendation to users is to apply the hotfix, or bump up their BIG-IP version to 11.6.1.
Here's the release notes for the latest hotfix related that issue:
447043-3 : Cannot have 2 distinct 'contains' conditions on the same LTM policy operand
Component: Local Traffic Manager
Symptoms:
Cannot express conditions such as 'user-agent contains 'Android' AND 'Mobile'. LTM policies have operands that can be matched against a set of values, causing a match when the operand matches one of these values. There is no way to use current functionality to match all of the values. One specific situation in which this is needed is to configure 'contains'.
Conditions:
Specify an ltm rule with 2 conditions with the same operand and match type, for example:
conditions {
0 {
http-header
name User-Agent
contains
values { Android }
}
1 {
http-header
name User-Agent
contains
values { Mobile }
}
Impact:
The policy does not work. The system posts an error message similar to the following: Failed to compile the combined policies.
Fix:
LTM policies now allow for rules to have multiple conditions on the same operand and same match type so that 'user-agent contains 'Android' AND 'Mobile' can now be expressed by specifying:
conditions {
0 {
http-header
name User-Agent
contains
values { Android }
}
1 {
http-header
name User-Agent
contains
values { Mobile }
}
I believe this is a bug and not a test issue.
This is a reproducible issue against BIG-IP 11.5.4.
Agent Version
9.2.0
OpenStack Release
Mitaka
Description
The test
api/test_l7policy_rules.py::L7PolicyJSONReject::test_policy_reject_many_rules
creates many rules that vary the comparison type and the value, all wrapped in a reject policy. The REST calls and the tempest failure is given below. It appears that the policy still exists even after all rules are deleted, which should result in the empty policy being deleted (called policy_wrapper).Snapshot from the agent log: