Selutario commented 5 months ago

Component
Wazuh API

Description

We need a new development, similar to what was done for the <limits> and <indexer> blocks, to prevent users from using Public VirustTotal API keys in the <integration><name>virustotal</name> block (link)

<integration>
  <name>virustotal</name>
  <api_key><API_KEY></api_key>
  <group>syscheck</group>
  <alert_format>json</alert_format>
</integration>

Using or not the Public API key shown above should only be allowed when explicitly permitted in the api.yaml (it will be allowed by default). Therefore, a new configuration must be added:

# Uploadable Wazuh configuration sections
 upload_configuration:
   integrations:
     virustotal:
       public_key:
         allow: yes
         minimum_quota: 240

Checks

The following elements have been updated or reviewed (should also be checked if no modification is required):

[x] Tests (unit tests, API integration tests, API configuration IT).
[x] Changelog.
[x] Documentation.
[x] Integration test mapping (using api/test/integration/mapping/_test_mapping.py).

[!NOTE] Related issues:

https://github.com/wazuh/wazuh/issues/16520

https://github.com/wazuh/wazuh/issues/22656

fdalmaup commented 5 months ago

Issue Update

The required modifications have been applied and manually tested with the PUT /manager/configuration endpoint.

Wazuh configuration update with a public API key when not allowed:

{
"data": {
"affected_items": [],
"total_affected_items": 0,
"total_failed_items": 1,
"failed_items": [
  {
    "error": {
      "code": 1130,
      "message": "Public Virus Total API Key detected: integrations > virustotal",
      "remediation": "To solve this, either use a premium VirusTotal API key or disable the public key protection in the API settings: https://documentation.wazuh.com/current/user-manual/api/configuration.html"
    },
    "id": [
      "wd_master_1"
    ]
  }
]
},
"message": "Could not update configuration in specified node",
"error": 1
}

Wazuh configuration update results in a request error when communicating with the VirusTotal API:

{
"data": {
"affected_items": [],
"total_affected_items": 0,
"total_failed_items": 1,
"failed_items": [
  {
    "error": {
      "code": 1131,
      "message": "Virus Total API request error",
      "remediation": "The use of Virus Total Public API keys is disabled but could not be checked. To solve this, check your connection to the Virus Total API or disable the public key protection in the API settings: https://documentation.wazuh.com/current/user-manual/api/configuration.html"
    },
    "id": [
      "wd_master_1"
    ]
  }
]
},
"message": "Could not update configuration in specified node",
"error": 1
}

Next tasks:

Update Unit tests
Update API ITs
Update documentation

fdalmaup commented 5 months ago

Issue Update

Rebased to 4.8.0.
UTs update

API

```console $ pytest api/api =========================================================================================== test session starts =========================================================================================== platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-1.3.0 rootdir: /home/fdalmau/git/wazuh/api/api configfile: pytest.ini plugins: metadata-3.0.0, cov-3.0.0, aiohttp-1.0.4, anyio-4.1.0, trio-0.7.0, tavern-1.23.5, asyncio-0.18.1, html-2.1.1 asyncio: mode=auto collected 575 items api/api/controllers/test/test_active_response_controller.py . [ 0%] api/api/controllers/test/test_agent_controller.py ............................................ [ 7%] api/api/controllers/test/test_cdb_list_controller.py ...... [ 8%] api/api/controllers/test/test_ciscat_controller.py . [ 9%] api/api/controllers/test/test_cluster_controller.py ........................ [ 13%] api/api/controllers/test/test_decoder_controller.py ....... [ 14%] api/api/controllers/test/test_default_controller.py . [ 14%] api/api/controllers/test/test_event_controller.py . [ 14%] api/api/controllers/test/test_experimental_controller.py ............... [ 17%] api/api/controllers/test/test_manager_controller.py ..................... [ 21%] api/api/controllers/test/test_mitre_controller.py ....... [ 22%] api/api/controllers/test/test_overview_controller.py . [ 22%] api/api/controllers/test/test_rootcheck_controller.py .... [ 23%] api/api/controllers/test/test_rule_controller.py ........ [ 24%] api/api/controllers/test/test_sca_controller.py .. [ 24%] api/api/controllers/test/test_security_controller.py ................................................... [ 33%] api/api/controllers/test/test_syscheck_controller.py .... [ 34%] api/api/controllers/test/test_syscollector_controller.py ......... [ 36%] api/api/controllers/test/test_task_controller.py . [ 36%] api/api/models/test/test_model.py .............................. [ 41%] api/api/test/test_alogging.py ...................... [ 45%] api/api/test/test_authentication.py ........... [ 47%] api/api/test/test_configuration.py ................................................... [ 56%] api/api/test/test_encoder.py ... [ 56%] api/api/test/test_middlewares.py ............... [ 59%] api/api/test/test_signals.py ......... [ 60%] api/api/test/test_uri_parser.py ... [ 61%] api/api/test/test_util.py ............................................... [ 69%] api/api/test/test_validator.py .................................................................................................................................................................... [ 97%] ............ [100%] ===================================================================================== 575 passed, 1 warning in 4.64s ====================================================================================== ```

Framework

```console $ pytest framework =========================================================================================== test session starts =========================================================================================== platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-1.3.0 rootdir: /home/fdalmau/git/wazuh/framework configfile: pytest.ini plugins: metadata-3.0.0, cov-3.0.0, aiohttp-1.0.4, anyio-4.1.0, trio-0.7.0, tavern-1.23.5, asyncio-0.18.1, html-2.1.1 asyncio: mode=auto collected 2215 items framework/scripts/tests/test_agent_groups.py .............. [ 0%] framework/scripts/tests/test_agent_upgrade.py ............... [ 1%] framework/scripts/tests/test_cluster_control.py ...... [ 1%] framework/scripts/tests/test_rbac_control.py ......... [ 1%] framework/scripts/tests/test_wazuh_clusterd.py ....... [ 2%] framework/scripts/tests/test_wazuh_logtest.py ...................... [ 3%] framework/wazuh/core/cluster/dapi/tests/test_dapi.py ................................ [ 4%] framework/wazuh/core/cluster/tests/test_client.py ................ [ 5%] framework/wazuh/core/cluster/tests/test_cluster.py ................................... [ 7%] framework/wazuh/core/cluster/tests/test_common.py ...................................................................................... [ 10%] framework/wazuh/core/cluster/tests/test_control.py ...... [ 11%] framework/wazuh/core/cluster/tests/test_local_client.py .............. [ 11%] framework/wazuh/core/cluster/tests/test_local_server.py ........................ [ 12%] framework/wazuh/core/cluster/tests/test_master.py ................................................ [ 15%] framework/wazuh/core/cluster/tests/test_server.py ............................. [ 16%] framework/wazuh/core/cluster/tests/test_utils.py ................ [ 17%] framework/wazuh/core/cluster/tests/test_worker.py .................................. [ 18%] framework/wazuh/core/tests/test_active_response.py .................... [ 19%] framework/wazuh/core/tests/test_agent.py ..................................................................................................................................................... [ 26%] framework/wazuh/core/tests/test_cdb_list.py ...................................... [ 27%] framework/wazuh/core/tests/test_common.py ......... [ 28%] framework/wazuh/core/tests/test_configuration.py ........................................................................................ [ 32%] framework/wazuh/core/tests/test_database.py ............. [ 32%] framework/wazuh/core/tests/test_decoder.py ................ [ 33%] framework/wazuh/core/tests/test_exception.py .......... [ 34%] framework/wazuh/core/tests/test_input_validator.py ... [ 34%] framework/wazuh/core/tests/test_logtest.py .. [ 34%] framework/wazuh/core/tests/test_manager.py ............................... [ 35%] framework/wazuh/core/tests/test_mitre.py ............. [ 36%] framework/wazuh/core/tests/test_pyDaemonModule.py ..... [ 36%] framework/wazuh/core/tests/test_results.py ........................................ [ 38%] framework/wazuh/core/tests/test_rootcheck.py ............. [ 38%] framework/wazuh/core/tests/test_rule.py ....................... [ 40%] framework/wazuh/core/tests/test_sca.py ............................. [ 41%] framework/wazuh/core/tests/test_security.py ............. [ 41%] framework/wazuh/core/tests/test_stats.py ................. [ 42%] framework/wazuh/core/tests/test_syscheck.py ....... [ 42%] framework/wazuh/core/tests/test_syscollector.py ... [ 43%] framework/wazuh/core/tests/test_task.py ........ [ 43%] framework/wazuh/core/tests/test_utils.py .......................................................................................................................................................... [ 50%] ........................................................................................................................................... [ 56%] framework/wazuh/core/tests/test_wazuh_queue.py ....................... [ 57%] framework/wazuh/core/tests/test_wazuh_socket.py .................... [ 58%] framework/wazuh/core/tests/test_wdb.py ............................... [ 60%] framework/wazuh/core/tests/test_wlogging.py ............ [ 60%] framework/wazuh/rbac/tests/test_auth_context.py .. [ 60%] framework/wazuh/rbac/tests/test_decorators.py ............................................................................................................. [ 65%] framework/wazuh/rbac/tests/test_default_configuration.py ....................................................... [ 68%] framework/wazuh/rbac/tests/test_orm.py .............................................................. [ 70%] framework/wazuh/rbac/tests/test_preprocessor.py ........... [ 71%] framework/wazuh/tests/test_active_response.py ............ [ 71%] framework/wazuh/tests/test_agent.py .......................................................................................................................... [ 77%] framework/wazuh/tests/test_cdb_list.py ..................................................... [ 79%] framework/wazuh/tests/test_ciscat.py ................................. [ 81%] framework/wazuh/tests/test_cluster.py .......... [ 81%] framework/wazuh/tests/test_decoder.py .......................................................... [ 84%] framework/wazuh/tests/test_event.py .... [ 84%] framework/wazuh/tests/test_group.py ....... [ 84%] framework/wazuh/tests/test_logtest.py ...... [ 85%] framework/wazuh/tests/test_manager.py .................................... [ 86%] framework/wazuh/tests/test_mitre.py ....... [ 87%] framework/wazuh/tests/test_rootcheck.py .................................................. [ 89%] framework/wazuh/tests/test_rule.py .......................................................................... [ 92%] framework/wazuh/tests/test_sca.py ........... [ 93%] framework/wazuh/tests/test_security.py ....................................................................... [ 96%] framework/wazuh/tests/test_stats.py ............... [ 97%] framework/wazuh/tests/test_syscheck.py ......................... [ 98%] framework/wazuh/tests/test_syscollector.py ............ [ 98%] framework/wazuh/tests/test_task.py ............................ [100%] ==================================================================================== 2215 passed in 265.33s (0:04:25) ===================================================================================== ```

AITs update and execution:


(integrationtest-env-3.10) fdalmau@wazuhFW:~/.../api/test/integration(enhancement/23741-virustotal-api-option)$ pytest test_cluster_endpoints.tavern.yaml
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-0.13.1
rootdir: /home/fdalmau/git/wazuh/api/test/integration
configfile: pytest.ini
plugins: aiohttp-1.0.4, anyio-4.1.0, trio-0.7.0, metadata-2.0.2, tavern-1.23.5, asyncio-0.18.1, html-2.1.1
asyncio: mode=auto
collected 49 items

test_cluster_endpoints.tavern.yaml ................................................. [100%]

============================================================================== 49 passed, 50 warnings in 1066.51s (0:17:46) =============================================================================== (integrationtest-env-3.10) fdalmau@wazuhFW:~/.../api/test/integration(enhancement/23741-virustotal-api-option)$ pytest test_manager_endpoints.tavern.yaml =========================================================================================== test session starts =========================================================================================== platform linux -- Python 3.10.14, pytest-7.3.1, pluggy-0.13.1 rootdir: /home/fdalmau/git/wazuh/api/test/integration configfile: pytest.ini plugins: aiohttp-1.0.4, anyio-4.1.0, trio-0.7.0, metadata-2.0.2, tavern-1.23.5, asyncio-0.18.1, html-2.1.1 asyncio: mode=auto collected 33 items

test_manager_endpoints.tavern.yaml ................................. [100%]

=============================================================================== 33 passed, 34 warnings in 978.94s (0:16:18) ===============================================================================


- Documentation update in the `enhancement/23741-virustotal-public-key-api-option` wazuh-documentation branch.
- The `Integration API tests` are being extended adding the new feature tests. A mock server to simulate the Virus Total service is being developed.

fdalmaup commented 5 months ago

Issue Update

A docker image has been built to expose a simple https server with a self-signed certificate to respond to the possible requests made by the server. When running the container, the VM executing the API Integration Tests found in tests/integration/test_api/test_config needed to be modified with the following commands:

# Modify /etc/hosts to redirect requests to www.virustotal.com to localhost
sed -i '/^$/i\I127.0.0.1 www.virustotal.com' /etc/hosts

# Redirect HTTPS traffic from www.virustotal.com to localhost:8080
sudo iptables -t nat -A OUTPUT -p tcp --dport 443 -d www.virustotal.com -j REDIRECT --to-port 8080

The manual tests of the HTTPS mock server worked as expected:

root@vagrant:/# curl -s -k https://virustotal.com/api/v3/users/expected_api_key/overall_quotas --header 'x-apikey: expected_api_key'
{"data": {"api_requests_hourly": {"user": {"allowed": 240, "used": 0}}}}

root@vagrant:/# curl -s -k https://virustotal.com/api/v3/users/error_api_key/overall_quotas --header 'x-apikey: error_api_key'
{"error": {"code": "WrongCredentialsError", "message": "Wrong API key"}}

root@vagrant:/# docker run -p 8080:8080 http_server
Listening on port 8080...
172.17.0.1 - - [31/May/2024 09:42:13] "GET /api/v3/users/expected_api_key/overall_quotas HTTP/1.1" 200 -
172.17.0.1 - - [31/May/2024 09:42:16] "GET /api/v3/users/error_api_key/overall_quotas HTTP/1.1" 401 -

The problem is when the Framework tries to get the information from the mock server. Since a self-signed certificate is used, and the requests.get() method is not used with the verify=False argument, the request fails and the PUT /manager/configuration returns:

{"data": {"affected_items": [], "total_affected_items": 0, "total_failed_items": 1, "failed_items": [{"error": {"code": 1131, "message": "Virus Total API request error: HTTPSConnectionPool(host='www.virustotal.com', port=443): Max retries exceeded with url: /api/v3/users/expected_api_key/overall_quotas (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')))", "remediation": "The use of Virus Total Public API keys is disabled but could not be checked. To solve this, check your connection to the Virus Total API or disable the public key protection in the API settings: https://documentation.wazuh.com/4.8/user-manual/api/configuration.html"}, "id": ["manager"]}]}, "message": "Could not update configuration", "error": 1}

Therefore, one of the test's steps will be to modify the line that sends the request to the VirusTotal API to avoid certificate verification, being the result:

# curl -k -X PUT "https://localhost:55000/manager/configuration" -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/octet-stream" -d '<ossec_config>
        <integration>
          <name>virustotal</name>
          <api_key>expected_api_key</api_key>
          <group>syscheck</group>
          <alert_format>json</alert_format>
        </integration>
      </ossec_config>'
{"data": {"affected_items": [], "total_affected_items": 0, "total_failed_items": 1, "failed_items": [{"error": {"code": 1130, "message": "Public Virus Total API Key detected: integrations > virustotal", "remediation": "To solve this, either use a premium VirusTotal API key or disable the public key protection in the API settings: https://documentation.wazuh.com/4.8/user-manual/api/configuration.html"}, "id": ["manager"]}]}, "message": "Could not update configuration", "error": 1}

wazuh / wazuh

Add API option to set whether users can use the `virustotal` public api key in the `ossec.conf` #23741

Description

Checks

Issue Update

Issue Update

Issue Update