[BUG] security plugin integration tests for 2.0.1 and 2.1.0 flaky

[dblock]

What is the bug?

Integration tests seem to fail often with 2.0.1 and 2.1.0 in security plugin tests. Server responds with 500s.

How can one reproduce the bug?

https://github.com/opensearch-project/opensearch-py/actions/runs/6947817680/job/18902465761 https://github.com/opensearch-project/opensearch-py/actions/runs/6949254612/job/18906963187

test_opensearchpy/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_update_audit_configuration FAILED [ 45%]
test_opensearchpy/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_update_role FAILED [ 48%]
test_opensearchpy/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_update_user FAILED [ 51%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_create_role FAILED [ 54%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_create_role_with_body_param_empty PASSED [ 58%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_create_user FAILED [ 61%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_create_user_with_body_param_empty PASSED [ 64%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_create_user_with_role FAILED [ 67%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_delete_role FAILED [ 70%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_delete_user FAILED [ 74%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_get_role FAILED [ 77%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_get_user FAILED [ 80%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_health PASSED [ 83%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_health_check PASSED [ 87%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_update_audit_config FAILED [ 90%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_update_audit_configuration FAILED [ 93%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_update_role FAILED [ 96%]
test_opensearchpy/test_async/test_server_secured/test_security_plugin.py::TestSecurityPlugin::test_update_user FAILED [100%]

______________ TestSecurityPlugin.test_update_audit_configuration ______________

self = <test_opensearchpy.test_server_secured.test_security_plugin.TestSecurityPlugin testMethod=test_update_audit_configuration>

    def test_update_audit_configuration(self) -> None:
>       response = self.client.security.update_audit_configuration(
            body=self.AUDIT_CONFIG_SETTINGS
        )

test_opensearchpy/test_server_secured/test_security_plugin.py:214: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
opensearchpy/client/utils.py:181: in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
opensearchpy/client/security.py:976: in update_audit_configuration
    return self.transport.perform_request(
opensearchpy/transport.py:448: in perform_request
    raise e
opensearchpy/transport.py:409: in perform_request
    status, headers_response, data = connection.perform_request(
opensearchpy/connection/http_urllib3.py:290: in perform_request
    self._raise_error(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Urllib3HttpConnection: https://instance:9200>, status_code = 500
raw_data = '{"status":"INTERNAL_SERVER_ERROR","message":"Error [instance][172.18.0.2:9300][indices:data/write/bulk[s][p]]"}'

What is the expected behavior?

Integration tests to be reliable.

[peternied]

I looked at the run, can't see the contents of the opensearch.log - that would be where the root cause of the issue could be found as it will include exception information for troubleshooting. Can the test runs be updated to capture that information?

[dblock]

Might be related to https://github.com/opensearch-project/security/issues/1961. Will look at extracting the OpenSearch logs, thanks @peternied.