DamianNud commented 1 month ago

Description

During the release testing (https://github.com/wazuh/wazuh/issues/26061) we detected that the test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/configuration/validation API integration test failed.

Errors:
E   tavern.util.exceptions.TestFailError: Test 'Request manager configuration validation (Allow)' failed:
    - Status code was 500, expected 200
    - Structure of returned data was different than expected  - Extra keys in response: {'detail', 'title', 'dapi_errors'} - Keys missing from response: {'data'} (expected = '{'error': <tavern.util.loader.IntSentinel object at 0x7fa6bb268a00>, 'data': {'affected_items': <tavern.util.loader.AnythingSentinel object at 0x7fa6bbaa8790>, 'total_failed_items': 0}}' (type = <class 'dict'>), actual = '{'title': 'Wazuh Internal Error', 'detail': 'Timeout executing API request', 'dapi_errors': {'master-node': {'error': 'Timeout executing API request', 'logfile': 'WAZUH_HOME/logs/api.log'}}, 'error': 3021}' (type = <class 'dict'>))

Artifacts.zip

This issue aims to investigate the root cause of the problem and fix the mentioned test case.

Checks

The following elements have been updated or reviewed (should also be checked if no modification is required):

[x] Tests (unit tests, API integration tests).
[x] Changelog.
[x] Documentation.
[x] Integration test mapping (using api/test/integration/mapping/_test_mapping.py).

RamosFe commented 1 month ago

Update

Testing the API integration test locally, no errors where founds. After analyzing the error, I found out that a timeout causes the error. I reached the conclussion that the failed test was due to the environment and not an error related to the API itself. To validate this hypothesis, I added the wait_to_complete parameter to the endpoint:

test_name: GET /manager/configuration/validation

stages:

  - name: Request manager configuration validation (Allow)
    request:
      verify: False
      url: "{protocol:s}://{host:s}:{port:d}/manager/configuration/validation"
      method: GET
      headers:
        Authorization: "Bearer {test_login_token}"
      params:
        wait_for_complete: true
    response:
      <<: *permission_allowed

After this change, the test passed as shown in the following executions.

Local execution

sudo -E ENV_MODE=standalone ../../../env/bin/python3 -m pytest test_rbac_black_manager_endpoints.tavern.yaml
========================================================================================================== test session starts ==========================================================================================================
platform linux -- Python 3.10.12, pytest-7.3.1, pluggy-1.5.0
rootdir: /home/federamos/Documents/Wazuh/Repositories/wazuh/api/test/integration
configfile: pytest.ini
plugins: asyncio-0.18.1, html-2.1.1, metadata-3.1.1, cov-4.1.0, tavern-1.23.5, anyio-4.1.0, trio-0.8.0
asyncio: mode=auto
collected 16 items
test_rbac_black_manager_endpoints.tavern.yaml ................                                                                                                                                                                    [100%]
=========================================================================================================== warnings summary ============================================================================================================
../../../env/lib/python3.10/site-packages/_pytest/nodes.py:642
  /home/federamos/Documents/Wazuh/Repositories/wazuh/env/lib/python3.10/site-packages/_pytest/nodes.py:642: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to YamlFile is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    return super().from_parent(parent=parent, fspath=fspath, path=path, **kw)
test_rbac_black_manager_endpoints.tavern.yaml: 16 warnings
  <frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================================== 16 passed, 17 warnings in 449.94s (0:07:29) ==============================================================================================

Test results in pipeline

============================= test session starts ==============================
platform linux -- Python 3.10.6, pytest-7.3.1, pluggy-1.2.0 -- /usr/bin/python3
cachedir: .pytest_cache
metadata: {'Python': '3.10.6', 'Platform': 'Linux-5.19.0-1028-aws-x86_64-with-glibc2.35', 'Packages': {'pytest': '7.3.1', 'pluggy': '1.2.0'}, 'Plugins': {'asyncio': '0.18.1', 'html': '2.1.1', 'trio': '0.7.0', 'metadata': '3.0.0', 'tavern': '1.23.5', 'aiohttp': '1.0.4'}}
rootdir: /tmp/Test_integration_endpoints_B4780_test_files/api/test/integration
configfile: pytest.ini
plugins: asyncio-0.18.1, html-2.1.1, trio-0.7.0, metadata-3.0.0, tavern-1.23.5, aiohttp-1.0.4
asyncio: mode=auto
collecting ... collected 16 items

test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/configuration PASSED [  6%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/configuration/validation PASSED [ 12%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/configuration/{component}/{configuration} PASSED [ 18%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/info PASSED  [ 25%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/logs PASSED  [ 31%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/logs/summary PASSED [ 37%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/daemons/stats PASSED [ 43%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/stats PASSED [ 50%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/stats/analysisd PASSED [ 56%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/stats/hourly PASSED [ 62%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/stats/remoted PASSED [ 68%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/stats/weekly PASSED [ 75%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/status PASSED [ 81%]
test_rbac_black_manager_endpoints.tavern.yaml::GET /manager/api/config PASSED [ 87%]
test_rbac_black_manager_endpoints.tavern.yaml::PUT /manager/configuration PASSED [ 93%]
test_rbac_black_manager_endpoints.tavern.yaml::PUT /manager/restart PASSED [100%]

=============================== warnings summary ===============================
../../../../../home/ubuntu/.local/lib/python3.10/site-packages/_pytest/nodes.py:642
  /home/ubuntu/.local/lib/python3.10/site-packages/_pytest/nodes.py:642: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to YamlFile is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    return super().from_parent(parent=parent, fspath=fspath, path=path, **kw)

test_rbac_black_manager_endpoints.tavern.yaml: 16 warnings
  <frozen importlib._bootstrap>:283: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
- generated html file: file:///tmp/Test_integration_endpoints_B4780_test_files/api/test/integration/Test_integration_endpoints_B4780_test_rbac_black_manager_endpoints.html -
================= 16 passed, 17 warnings in 383.13s (0:06:23) ==================

javiersanchz commented 1 month ago

Review

As mentioned, the error originates from a Timeout, which can be verified in the api.log:

- 2024/09/30 18:05:12 ERROR: Timeout executing API request

This error comes from the testing environment, exceeding the time limit for the API call.
As seen in the provided tests, the test passes correctly (it was rechecked, resulting in a successful outcome)

LGTM !

fdalmaup commented 1 month ago

Review

The testing environment used for the tests had several changes related to the instances where these are run. The resources in certain builds where the complete test suite is executed may be limited but since the whole infrastructure is being reviewed for the upcoming 5.0 and the server itself and the test are not the cause of the error, we can determine that is part of the flakiness we have been seeing in the last release testing issues (https://github.com/wazuh/wazuh/issues/25891 and https://github.com/wazuh/wazuh/issues/25367#issuecomment-2309551026 for example).

LGTM!

wazuh / wazuh

Failed test in test_rbac_black_manager_endpoints #26082

Description

Checks

Update

Local execution

Test results in pipeline

Review

Review