Azure / azure-cli

Azure Command-Line Interface
MIT License
4.01k stars 2.98k forks source link

Investigate security test flakiness #9406

Open tjprescott opened 5 years ago

tjprescott commented 5 years ago

The test test_security_alerts is behaving flaky on ADO. Possibly related to #9385.

Occasional ADO stack trace:

======================================================================
FAIL: test_security_alerts (azure.cli.command_modules.security.tests.latest.test_alerts_scenario.SecurityCenterAlertsTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 251, in _in_process_execute
    self.exit_code = cli_ctx.invoke(shlex.split(command), out_file=stdout_buf) or 0
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/knack/cli.py", line 218, in invoke
    exit_code = self.exception_handler(ex)
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/core/__init__.py", line 98, in exception_handler
    return handle_exception(ex)
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/patches.py", line 33, in _handle_main_exception
    raise ex
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/knack/cli.py", line 206, in invoke
    cmd_result = self.invocation.execute(args)
vcr.errors.CannotOverwriteExistingCassetteException: No match for the request (<Request (GET) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli_role_assign000001/providers/Microsoft.Security/alerts?api-version=2015-06-01-preview>) was found. Can't overwrite existing cassette ('/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/recordings/test_security_alerts.yaml') in your current record mode ('once').

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure_devtools/scenario_tests/decorators.py", line 40, in _preparer_wrapper
    fn(test_class_instance, **kwargs)
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/test_alerts_scenario.py", line 16, in test_security_alerts
    alerts = self.cmd('az security alert list').get_output_in_json()
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 161, in cmd
    return execute(self.cli_ctx, command, expect_failure=expect_failure).assert_with_checks(checks)
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 200, in __init__
    self._in_process_execute(cli_ctx, command, expect_failure=expect_failure)
  File "/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/testsdk/base.py", line 256, in _in_process_execute
    raise AssertionError(ex)
AssertionError: No match for the request (<Request (GET) https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/cli_role_assign000001/providers/Microsoft.Security/alerts?api-version=2015-06-01-preview>) was found. Can't overwrite existing cassette ('/opt/hostedtoolcache/Python/3.6.8/x64/lib/python3.6/site-packages/azure/cli/command_modules/security/tests/latest/recordings/test_security_alerts.yaml') in your current record mode ('once').

When run locally, these tests pass consistently.

AddySanto commented 3 years ago

Looking for the relevant contact person on the team.

yonzhan commented 3 years ago

security