search

wazuh / wazuh-qa

Wazuh - Quality Assurance
GNU General Public License v2.0
64 stars 30 forks source link

Benchmarking tools and techniques investigation #5502

Open Rebits opened 2 weeks ago

Rebits commented 2 weeks ago

Description

The objective of this issue is to investigate and identify the most effective tools, techniques, and methodologies for benchmarking testing. These tests will serve as a baseline for comparison, enabling the tracking of performance improvements or regressions. By doing so, benchmarking will help maintain and enhance the efficiency of the product. The proposed tools and techniques should meet the proposed functional and non-functional requirements. This research will enable us to determine the necessary functionalities to develop a performance test for the product, ensuring that it is comprehensive and effective.

Funcional Requirements

Capabilities

KPTM analysis and data collection

Proposed tools should be capable of monitoring the following KPTM (Key Performance Test Metrics):

Implementation restrictions

Plan

  1. Agree on expected KPTM values
    • Identify the KPTM to be measured
    • Determine target values for these KPTM to ensure the system meets performance standards
  2. Research and Analysis:
    • Research current tools
    • Research tools to collect, analyze, and generate testing loads
    • Research tools for reporting and analysis.
    • Identify the most suitable tools for collecting and analyzing KPTM values.
    • Analyze the tools and their capabilities to determine the best fit for testing needs.

Related issues

Parent issue

rafabailon commented 1 week ago

Tools for Different Purposes

Tools Summary

The following is a summary table of the tools analyzed with the main characteristics of each one

Tool Test Language Multiplatform Requirements Purpose
Artillery YAML Yes Node.js, NPM Load and performance testing of web applications and APIs
Playwright JavaScript / TypeScript Yes Node.js, NPM Test automation in web browsers
OpenSearch Benchmark JSON Yes Python 3.6+ (Pip), Elasticsearch, OpenSearch Search and Analysis Cluster Performance Evaluation
Locust Python Yes Python 3.6+ (Pip) Load and stress testing in web applications
Fluentd - Yes Ruby, Gem (Installation Packages Available) Collect, transform, and send logs and events
Tsung XML Yes Erlang, GNU Make (Installation Packages Available) Load and performance testing of web applications and services
Cypress JavaScript / TypeScript Yes Node.js, NPM End-to-End Tests

The information on each tool explains the advantages and disadvantages of each one.

Tool Advantages Disadvantages
Artillery Modern & Easy-to-Use. Cloud-Scale Testing. Test Any Stack. Fast Adoption. Extensible. Scalable & Cost-Efficient. Open Source. Learning Curve. Limited UI Testing. Protocol Limitations.
Playwright Cross-Browser Support. Cross-Platform Compatibility. Mobile Emulation. Language Flexibility. Headless and GUI Modes. Advanced Automation Capabilities. Learning Curve. Limited UI Testing. Protocol Constraints.
OpenSearch Benchmark Performance Metrics. Decision Support. Resource Optimization. Complexity. Maintenance Overhead.
Locust User-Friendly Task Definition. Distributed Load Generation. Real-Time Web-Based Monitoring. Simulating User Behavior. Performance Metrics and Reporting. Python Dependency. Limited Protocol Support. Less Suitable for High-Concurrency Workloads.
Fluentd Pluggable Architecture. Real-Time Data Processing. Cross-Platform Support. Better Memory Usage. Decentralized Ecosystem. Transport and Buffering. Parsing Complexity.
Tsung Erlang-Based. Protocol Support. Stability. Distributed Load Generation. Automated Statistics. Learning Curve. Complexity. Limited Protocols.
Cypress Speed. User-Friendly Interface. Reliability. Flexibility. Stability. Active Community. Browser-Based. Limited Cross-Browser Support. No Native Mobile App Testing. Single Browser Session. No Direct Multiple Windows/Tabs Support

Tools Information

Artillery (Performance Testing)

Information
Artillery is an open source load testing tool. It allows you to simulate multiple concurrent users, making it an option for performance testing. Artillery supports HTTP, WebSocket and Server-Sent Events protocols. It is free and open source. You can use it to evaluate the performance and scalability of web services, APIs and other networked systems. Artillery test scripts are usually written as YAML, but they can also be written in JavaScript. Artillery scripts have two parts: `config` and `scenarios`. `config` is what defines how our load test will run. `scenarios` is where we define what the virtual users created by Artillery will do. - Interest URLs - [Website](https://www.artillery.io/) - [Documentation](https://www.artillery.io/docs) - [Installation](https://www.artillery.io/docs/get-started/get-artillery) - Requeriments - SO: Windows, MacOS, and Linux - Required Software: LTS Release of [Node.js](https://nodejs.org/en/download) and NPM.
Advantages and disadvantages
- Advantages: - Modern & Easy-to-Use: Artillery is a powerful and user-friendly performance testing toolkit. It prioritizes developer productivity and follows a "batteries-included" philosophy. - Cloud-Scale Testing: You can run distributed multi-region load tests using AWS Lambda or AWS Fargate without managing infrastructure. - Test Any Stack: Artillery supports testing HTTP APIs, WebSocket, Socket.io services, and complex web apps with real browsers. - Fast Adoption: Designed to be easy to start with, it offers extensions, plugins, and integrations with monitoring and CI/CD tools. - Extensible: You can extend Artillery and build custom integrations using Node.js. - Scalable & Cost-Efficient: Run large-scale load tests from your own AWS account using AWS Lambda or serverless AWS Fargate clusters. - Open Source: Licensed under MPL-2.0, making it easy for platform and SQA teams to build on top of Artillery. - Disadvantages: - Learning Curve: While Artillery aims for ease of use, some users may find a learning curve when exploring its features. - Limited UI Testing: Although Artillery can run Playwright-based scripts for UI testing, it's primarily designed for backend systems. - Protocol Limitations: While it supports HTTP, WebSocket, and Socket.io, additional protocols require custom plugins.
Installation
```console root@ubuntu2204:/home/vagrant# apt update root@ubuntu2204:/home/vagrant# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash root@ubuntu2204:/home/vagrant# nvm install 20 root@ubuntu2204:/home/vagrant# node -v root@ubuntu2204:/home/vagrant# npm -v root@ubuntu2204:/home/vagrant# npm install -g artillery@latest ``` It is also possible to install the VS Code extension for Artillery, making it easier to create tests. [Artillery VS Code Extension](https://www.artillery.io/docs/resources/vs-code-extension)
Artillery Test
- Create `asciiart-load-test.yml` ```console root@ubuntu2204:/home/vagrant# touch asciiart-load-test.yml ``` - Add Test Code ```yaml config: target: http://asciiart.artillery.io:8080 phases: - duration: 60 arrivalRate: 1 rampTo: 5 name: Warm up phase - duration: 60 arrivalRate: 5 rampTo: 10 name: Ramp up load - duration: 30 arrivalRate: 10 rampTo: 30 name: Spike phase plugins: ensure: {} apdex: {} metrics-by-endpoint: {} apdex: threshold: 100 ensure: thresholds: - http.response_time.p99: 100 - http.response_time.p95: 75 scenarios: - flow: - loop: - get: url: '/dino' - get: url: '/pony' - get: url: '/armadillo' count: 100 ``` - Run the Load Test ```console root@ubuntu2204:/home/vagrant# artillery run asciiart-load-test.yml Test run id: tkrrk_eap897yrfekgtcdemgax3jbqjxzwn_4qth Phase started: Warm up phase (index: 0, duration: 60s) 07:35:49(+0000) -------------------------------------- Metrics for period to: 07:36:00(+0000) (width: 9.895s) -------------------------------------- apdex.frustrated: .............................................................. 1 apdex.satisfied: ............................................................... 1017 apdex.tolerated: ............................................................... 16 http.codes.200: ................................................................ 1034 http.downloaded_bytes: ......................................................... 598768 http.request_rate: ............................................................. 106/sec http.requests: ................................................................. 1047 http.response_time: min: ......................................................................... 46 max: ......................................................................... 133 mean: ........................................................................ 53.7 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 80.6 http.responses: ................................................................ 1034 plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 339 plugins.metrics-by-endpoint./dino.codes.200: ................................... 350 plugins.metrics-by-endpoint./pony.codes.200: ................................... 345 plugins.metrics-by-endpoint.response_time./armadillo: min: ......................................................................... 46 max: ......................................................................... 133 mean: ........................................................................ 53.9 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 85.6 plugins.metrics-by-endpoint.response_time./dino: min: ......................................................................... 48 max: ......................................................................... 132 mean: ........................................................................ 53.6 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 76 plugins.metrics-by-endpoint.response_time./pony: min: ......................................................................... 47 max: ......................................................................... 130 mean: ........................................................................ 53.5 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 76 vusers.created: ................................................................ 13 vusers.created_by_name.0: ...................................................... 13 -------------------------------------- Metrics for period to: 07:36:10(+0000) (width: 9.961s) -------------------------------------- apdex.frustrated: .............................................................. 0 apdex.satisfied: ............................................................... 4153 apdex.tolerated: ............................................................... 20 http.codes.200: ................................................................ 4173 http.downloaded_bytes: ......................................................... 2413754 http.request_rate: ............................................................. 421/sec http.requests: ................................................................. 4189 http.response_time: min: ......................................................................... 47 max: ......................................................................... 99 mean: ........................................................................ 53.1 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 77.5 http.responses: ................................................................ 4173 plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 1388 plugins.metrics-by-endpoint./dino.codes.200: ................................... 1394 plugins.metrics-by-endpoint./pony.codes.200: ................................... 1391 plugins.metrics-by-endpoint.response_time./armadillo: min: ......................................................................... 47 max: ......................................................................... 99 mean: ........................................................................ 52.9 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 77.5 plugins.metrics-by-endpoint.response_time./dino: min: ......................................................................... 47 max: ......................................................................... 93 mean: ........................................................................ 53.2 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 77.5 plugins.metrics-by-endpoint.response_time./pony: min: ......................................................................... 47 max: ......................................................................... 97 mean: ........................................................................ 53.2 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 77.5 vusers.completed: .............................................................. 4 vusers.created: ................................................................ 20 vusers.created_by_name.0: ...................................................... 20 vusers.failed: ................................................................. 0 vusers.session_length: min: ......................................................................... 16380.2 max: ......................................................................... 17135.7 mean: ........................................................................ 16639 median: ...................................................................... 16486.1 p95: ......................................................................... 16486.1 p99: ......................................................................... 16486.1 -------------------------------------- Metrics for period to: 07:36:20(+0000) (width: 9.994s) -------------------------------------- apdex.frustrated: .............................................................. 0 apdex.satisfied: ............................................................... 6390 apdex.tolerated: ............................................................... 28 http.codes.200: ................................................................ 6418 http.downloaded_bytes: ......................................................... 3711633 http.request_rate: ............................................................. 643/sec http.requests: ................................................................. 6427 http.response_time: min: ......................................................................... 47 max: ......................................................................... 97 mean: ........................................................................ 53.5 median: ...................................................................... 51.9 p95: ......................................................................... 62.2 p99: ......................................................................... 85.6 http.responses: ................................................................ 6418 plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 2136 plugins.metrics-by-endpoint./dino.codes.200: ................................... 2145 plugins.metrics-by-endpoint./pony.codes.200: ................................... 2137 plugins.metrics-by-endpoint.response_time./armadillo: min: ......................................................................... 47 max: ......................................................................... 97 mean: ........................................................................ 53.6 median: ...................................................................... 51.9 p95: ......................................................................... 62.2 p99: ......................................................................... 85.6 plugins.metrics-by-endpoint.response_time./dino: min: ......................................................................... 47 max: ......................................................................... 96 mean: ........................................................................ 53.5 median: ...................................................................... 51.9 p95: ......................................................................... 62.2 p99: ......................................................................... 87.4 plugins.metrics-by-endpoint.response_time./pony: min: ......................................................................... 47 max: ......................................................................... 96 mean: ........................................................................ 53.4 median: ...................................................................... 51.9 p95: ......................................................................... 63.4 p99: ......................................................................... 85.6 vusers.completed: .............................................................. 18 vusers.created: ................................................................ 28 vusers.created_by_name.0: ...................................................... 28 vusers.failed: ................................................................. 0 vusers.session_length: min: ......................................................................... 15695.4 max: ......................................................................... 16830.3 mean: ........................................................................ 16144.8 median: ...................................................................... 16159.7 p95: ......................................................................... 16819.2 p99: ......................................................................... 16819.2 -------------------------------------- Metrics for period to: 07:36:30(+0000) (width: 9.999s) -------------------------------------- apdex.frustrated: .............................................................. 0 apdex.satisfied: ............................................................... 8548 apdex.tolerated: ............................................................... 30 http.codes.200: ................................................................ 8578 http.downloaded_bytes: ......................................................... 4962440 http.request_rate: ............................................................. 859/sec http.requests: ................................................................. 8592 http.response_time: min: ......................................................................... 46 max: ......................................................................... 95 mean: ........................................................................ 52.9 median: ...................................................................... 51.9 p95: ......................................................................... 58.6 p99: ......................................................................... 74.4 http.responses: ................................................................ 8578 plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 2852 plugins.metrics-by-endpoint./dino.codes.200: ................................... 2864 plugins.metrics-by-endpoint./pony.codes.200: ................................... 2862 plugins.metrics-by-endpoint.response_time./armadillo: min: ......................................................................... 47 max: ......................................................................... 92 mean: ........................................................................ 52.9 median: ...................................................................... 51.9 p95: ......................................................................... 58.6 p99: ......................................................................... 73 plugins.metrics-by-endpoint.response_time./dino: min: ......................................................................... 47 max: ......................................................................... 95 mean: ........................................................................ 53 median: ...................................................................... 51.9 p95: ......................................................................... 59.7 p99: ......................................................................... 74.4 plugins.metrics-by-endpoint.response_time./pony: min: ......................................................................... 46 max: ......................................................................... 95 mean: ........................................................................ 52.9 median: ...................................................................... 51.9 p95: ......................................................................... 59.7 p99: ......................................................................... 73 vusers.completed: .............................................................. 20 vusers.created: ................................................................ 34 vusers.created_by_name.0: ...................................................... 34 vusers.failed: ................................................................. 0 vusers.session_length: min: ......................................................................... 15811 max: ......................................................................... 16706.3 mean: ........................................................................ 16274.5 median: ...................................................................... 16159.7 p95: ......................................................................... 16819.2 p99: ......................................................................... 16819.2 -------------------------------------- Metrics for period to: 07:36:40(+0000) (width: 9.997s) -------------------------------------- apdex.frustrated: .............................................................. 0 apdex.satisfied: ............................................................... 10436 apdex.tolerated: ............................................................... 38 http.codes.200: ................................................................ 10474 http.downloaded_bytes: ......................................................... 6057553 http.request_rate: ............................................................. 1049/sec http.requests: ................................................................. 10485 http.response_time: min: ......................................................................... 46 max: ......................................................................... 96 mean: ........................................................................ 54.4 median: ...................................................................... 53 p95: ......................................................................... 67.4 p99: ......................................................................... 82.3 http.responses: ................................................................ 10474 plugins.metrics-by-endpoint./armadillo.codes.200: .............................. 3490 plugins.metrics-by-endpoint./dino.codes.200: ................................... 3493 plugins.metrics-by-endpoint./pony.codes.200: ................................... 3491 plugins.metrics-by-endpoint.response_time./armadillo: min: ......................................................................... 46 max: ......................................................................... 96 mean: ........................................................................ 54.5 median: ...................................................................... 53 p95: ......................................................................... 67.4 p99: ......................................................................... 82.3 plugins.metrics-by-endpoint.response_time./dino: min: ......................................................................... 46 max: ......................................................................... 96 mean: ........................................................................ 54.4 median: ...................................................................... 51.9 p95: ......................................................................... 67.4 p99: ......................................................................... 82.3 plugins.metrics-by-endpoint.response_time./pony: min: ......................................................................... 47 max: ......................................................................... 95 mean: ........................................................................ 54.4 median: ...................................................................... 53 p95: ......................................................................... 67.4 p99: ......................................................................... 82.3 vusers.completed: .............................................................. 30 vusers.created: ................................................................ 40 vusers.created_by_name.0: ...................................................... 40 vusers.failed: ................................................................. 0 vusers.session_length: min: ......................................................................... 15758.7 max: ......................................................................... 16982.9 mean: ........................................................................ 16260.6 median: ...................................................................... 16159.7 p95: ......................................................................... 16819.2 p99: ......................................................................... 16819.2 Phase completed: Warm up phase (index: 0, duration: 60s) 07:36:49(+0000) ``` [Test Script Walkthrough](https://www.artillery.io/docs/get-started/first-test#test-script-walkthrough)

Playwright (Test Automation Framework)

Information
Playwright is a test automation framework. Playwright allows you to perform end-to-end testing in web browsers and web scraping. It allows you to write code that interacts with web pages, performs actions like clicking, scrolling, and filling out forms, and captures screenshots or videos of the browser's state. Artillery and Playwright can be used together to create more realistic test scenarios. It is possible to create a script in Playwright and integrate it into Artillery. Since Artillery allows using JavaScript code, it is possible to create a JS function that calls the script in Playwright to simulate user interactions. - Interest URLs - [Website](https://playwright.dev/) - [Documentation](https://playwright.dev/docs/intro) - [Installation (npm, yarn, pnpm)](https://playwright.dev/docs/intro#installing-playwright) - [Installation (VS Code)](https://playwright.dev/docs/getting-started-vscode) - [Integrate Artilley and Playwright](https://www.workwithloop.com/blog/scaling-gracefully-using-artillery-with-playwright-for-realistic-load-scenarios) - Requeriments - Required Software: VS Code, npm, yarn, pnpm
Advantages and disadvantages
- Advantages: - Cross-Browser Support: Playwright works seamlessly with multiple browsers, including Chromium (Chrome, Edge), Firefox, and WebKit (Safari). This compatibility ensures consistent testing across different environments. - Cross-Platform Compatibility: You can use Playwright to test applications across various platforms, including mobile (Android), web, and desktop (MacOS, Linux, Windows). - Mobile Emulation: Playwright can emulate mobile devices, replicating geolocation, screen size, and other device-specific characteristics. - Language Flexibility: Initially built for Node.js, Playwright now offers bindings for JavaScript, TypeScript, Python, Java, and C#/.NET, making it accessible to a broader range of developers and testers. - Headless and GUI Modes: It can run browsers in headless mode (for faster execution in test environments) and GUI mode (for development and debugging). - Advanced Automation Capabilities: Playwright handles both traditional multi-page applications and complex single-page applications. It can interact with iframes and pierce through Shadow DOM, essential for testing modern web applications. - Disadvantages: - Learning Curve: While Playwright is powerful, some users may find a learning curve when exploring its features. - Limited UI Testing: Although it supports UI testing, Playwright is primarily designed for backend systems. - Protocol Constraints: Additional protocols beyond HTTP, WebSocket, and Socket.io require custom plugins.
Installation
```console root@ubuntu2204:/home/vagrant# apt update root@ubuntu2204:/home/vagrant# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash root@ubuntu2204:/home/vagrant# nvm install 20 root@ubuntu2204:/home/vagrant# node -v root@ubuntu2204:/home/vagrant# npm -v root@ubuntu2204:/home/vagrant# npm init playwright@latest root@ubuntu2204:/home/vagrant# npx playwright install-deps ``` It is also possible to install Playwright in VS Code, making it easier to create tests. [Playwright in VS Code](https://playwright.dev/docs/getting-started-vscode#introduction)
Playwright Test
- Run the Load Test ```console root@ubuntu2204:/home/vagrant# npx playwright test Running 6 tests using 1 worker 6 passed (8.3s) To open last HTML report run: npx playwright show-report ```
Integrate Playwright with Artillery
- Create `hello-world.yml` for Artillery ```console root@ubuntu2204:/home/vagrant# touch hello-world.yml ``` - Add Test Code to Artillery ```yaml config: target: https://www.artillery.io engines: playwright: {} processor: './flows.js' scenarios: - engine: playwright testFunction: 'helloFlow' ``` - Create `flows.js` for Playwright ```console root@ubuntu2204:/home/vagrant# touch flows.js ``` - Add Test Code to Playwright ```yaml module.exports = { helloFlow }; async function helloFlow(page) { await page.goto('https://www.artillery.io/'); await page.click('text=Cloud'); } ``` - Run the Load Test ```console root@ubuntu2204:/home/vagrant# artillery run hello-world.yml Test run id: t9yad_6437mptpftpgeqr6cya4egfyk5k3t_rwqw Phase started: unnamed (index: 0, duration: 1s) 09:16:07(+0000) Phase completed: unnamed (index: 0, duration: 1s) 09:16:08(+0000) -------------------------------------- Metrics for period to: 09:16:10(+0000) (width: 1.259s) -------------------------------------- browser.http_requests: ......................................................... 34 browser.page.codes.200: ........................................................ 35 browser.page.codes.206: ........................................................ 1 vusers.created: ................................................................ 1 vusers.created_by_name.0: ...................................................... 1 Warning: multiple batches of metrics for period 1719306960000 2024-06-25T09:16:00.000Z -------------------------------------- Metrics for period to: 09:16:20(+0000) (width: 1.465s) -------------------------------------- browser.http_requests: ......................................................... 28 browser.page.FCP.https://www.artillery.io/: min: ......................................................................... 1229.2 max: ......................................................................... 1229.2 mean: ........................................................................ 1229.2 median: ...................................................................... 1224.4 p95: ......................................................................... 1224.4 p99: ......................................................................... 1224.4 browser.page.FID.https://www.artillery.io/: min: ......................................................................... 14.2 max: ......................................................................... 14.2 mean: ........................................................................ 14.2 median: ...................................................................... 14.2 p95: ......................................................................... 14.2 p99: ......................................................................... 14.2 browser.page.LCP.https://www.artillery.io/: min: ......................................................................... 1229.2 max: ......................................................................... 1229.2 mean: ........................................................................ 1229.2 median: ...................................................................... 1224.4 p95: ......................................................................... 1224.4 p99: ......................................................................... 1224.4 browser.page.TTFB.https://www.artillery.io/: min: ......................................................................... 614.1 max: ......................................................................... 614.1 mean: ........................................................................ 614.1 median: ...................................................................... 608 p95: ......................................................................... 608 p99: ......................................................................... 608 browser.page.codes.200: ........................................................ 23 browser.page.codes.206: ........................................................ 1 browser.page.codes.302: ........................................................ 1 browser.page.codes.307: ........................................................ 1 browser.page.codes.308: ........................................................ 1 browser.page.codes.404: ........................................................ 1 vusers.completed: .............................................................. 1 vusers.failed: ................................................................. 0 vusers.session_length: min: ......................................................................... 2874.4 max: ......................................................................... 2874.4 mean: ........................................................................ 2874.4 median: ...................................................................... 2893.5 p95: ......................................................................... 2893.5 p99: ......................................................................... 2893.5 All VUs finished. Total time: 4 seconds -------------------------------- Summary report @ 09:16:13(+0000) -------------------------------- browser.http_requests: ......................................................... 62 browser.page.FCP.https://www.artillery.io/: min: ......................................................................... 1229.2 max: ......................................................................... 1229.2 mean: ........................................................................ 1229.2 median: ...................................................................... 1224.4 p95: ......................................................................... 1224.4 p99: ......................................................................... 1224.4 browser.page.FID.https://www.artillery.io/: min: ......................................................................... 14.2 max: ......................................................................... 14.2 mean: ........................................................................ 14.2 median: ...................................................................... 14.2 p95: ......................................................................... 14.2 p99: ......................................................................... 14.2 browser.page.LCP.https://www.artillery.io/: min: ......................................................................... 1229.2 max: ......................................................................... 1229.2 mean: ........................................................................ 1229.2 median: ...................................................................... 1224.4 p95: ......................................................................... 1224.4 p99: ......................................................................... 1224.4 browser.page.TTFB.https://www.artillery.io/: min: ......................................................................... 614.1 max: ......................................................................... 614.1 mean: ........................................................................ 614.1 median: ...................................................................... 608 p95: ......................................................................... 608 p99: ......................................................................... 608 browser.page.codes.200: ........................................................ 58 browser.page.codes.206: ........................................................ 2 browser.page.codes.302: ........................................................ 1 browser.page.codes.307: ........................................................ 1 browser.page.codes.308: ........................................................ 1 browser.page.codes.404: ........................................................ 1 vusers.completed: .............................................................. 1 vusers.created: ................................................................ 1 vusers.created_by_name.0: ...................................................... 1 vusers.failed: ................................................................. 0 vusers.session_length: min: ......................................................................... 2874.4 max: ......................................................................... 2874.4 mean: ........................................................................ 2874.4 median: ...................................................................... 2893.5 p95: ......................................................................... 2893.5 p99: ......................................................................... 2893.5 ``` Note: [Integrate Playwright with Artillery](https://www.artillery.io/docs/reference/engines/playwright#usage)

Opensearch Benchmark (Performance Test for OpenSearch Clusters)

Information
OpenSearch Benchmark is a macrobenchmark utility provided by the OpenSearch Project. It allows you to collect performance metrics from an OpenSearch cluster for a variety of purposes. It is possible to track the overall performance of the cluster, report on when to update the cluster, and evaluate how changes in workflow may affect the cluster. - Interest URLs - [Website](https://opensearch.org/docs/latest/benchmark/) - [Documentation](https://opensearch.org/docs/latest/benchmark/user-guide/index/) - [Installation](https://opensearch.org/docs/latest/benchmark/user-guide/installing-benchmark/) - Requeriments - Linux, MacOS or Docker - Python >= 3.8
Advantages and disadvantages
- Advantages: - Performance Metrics: OpenSearch Benchmark helps you gather performance metrics from an OpenSearch cluster. You can track the overall performance of your cluster, which is useful for monitoring and optimization. - Decision Support: It informs decisions about upgrading your cluster to a new version. By benchmarking performance, you can evaluate the benefits of upgrading and make informed choices. - Resource Optimization: The tool allows you to optimize cluster resource usage, potentially reducing operating costs. - Disadvantages: - Complexity: Benchmarking can be complex, especially when dealing with large clusters or intricate workflows. Proper configuration and interpretation of results are essential. - Maintenance Overhead: Regular benchmarking requires ongoing effort and resources.
Installation
```console root@ubuntu2204:/home/vagrant# apt install python3-pip root@ubuntu2204:/home/vagrant# pip install opensearch-benchmark ```
Installation (Docker)
```console root@ubuntu2204:/home/vagrant# docker pull opensearchproject/opensearch-benchmark:latest root@ubuntu2204:/home/vagrant# docker run opensearchproject/opensearch-benchmark -h ```
Opensearch Benchmark Test (With Wazuh)
- Check workloads ```console root@ubuntu2204:/home/vagrant# opensearch-benchmark list workloads ``` - Run Tests ```console root@ubuntu2204:/home/vagrant# opensearch-benchmark execute-test --pipeline=benchmark-only --workload=nested --target-host=https://localhost:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false ____ _____ __ ____ __ __ / __ \____ ___ ____ / ___/___ ____ ___________/ /_ / __ )___ ____ _____/ /_ ____ ___ ____ ______/ /__ / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \ / __ / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/ / /_/ / /_/ / __/ / / /__/ / __/ /_/ / / / /__/ / / / / /_/ / __/ / / / /__/ / / / / / / / / /_/ / / / ,< \____/ .___/\___/_/ /_/____/\___/\__,_/_/ \___/_/ /_/ /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/ /_/|_| /_/ [INFO] [Test Execution ID]: 73bdc966-abc6-4c84-a4d5-1be749c8062f [INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds. [INFO] Downloading workload data: documents.json.bz2 (663.3 MB total size) [100.0%] [INFO] Decompressing workload data from [/root/.benchmark/benchmarks/data/nested/documents.json.bz2] to [/root/.benchmark/benchmarks/data/nested/documents.json] (resulting size: [3.39] GB) ... [OK] [INFO] Preparing file offset table for [/root/.benchmark/benchmarks/data/nested/documents.json] ... [OK] [INFO] Executing test with workload [nested], test_procedure [nested-search-test_procedure] and provision_config_instance ['external'] with version [7.10.2]. [WARNING] merges_total_time is 102 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] indexing_total_time is 536 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] refresh_total_time is 954 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] flush_total_time is 138 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. Running delete-index [100% done] Running create-index [100% done] Running check-cluster-health [100% done] Running index-append [100% done] Running refresh-after-index [100% done] Running force-merge [100% done] Running refresh-after-force-merge [100% done] Running wait-until-merges-finish [100% done] Running randomized-nested-queries [100% done] Running randomized-term-queries [100% done] Running randomized-sorted-term-queries [100% done] Running match-all [100% done] Running nested-date-histo [100% done] Running randomized-nested-queries-with-inner-hits_default [100% done] Running randomized-nested-queries-with-inner-hits_default_big_size [100% done] ------------------------------------------------------ _______ __ _____ / ____(_)___ ____ _/ / / ___/_________ ________ / /_ / / __ \/ __ `/ / \__ \/ ___/ __ \/ ___/ _ \ / __/ / / / / / /_/ / / ___/ / /__/ /_/ / / / __/ /_/ /_/_/ /_/\__,_/_/ /____/\___/\____/_/ \___/ ------------------------------------------------------ | Metric | Task | Value | Unit | |---------------------------------------------------------------:|-----------------------------------------------------------:|------------:|-------:| | Cumulative indexing time of primary shards | | 8.58217 | min | | Min cumulative indexing time across primary shards | | 0 | min | | Median cumulative indexing time across primary shards | | 0.000258333 | min | | Max cumulative indexing time across primary shards | | 8.57323 | min | | Cumulative indexing throttle time of primary shards | | 0 | min | | Min cumulative indexing throttle time across primary shards | | 0 | min | | Median cumulative indexing throttle time across primary shards | | 0 | min | | Max cumulative indexing throttle time across primary shards | | 0 | min | | Cumulative merge time of primary shards | | 4.00155 | min | | Cumulative merge count of primary shards | | 6 | | | Min cumulative merge time across primary shards | | 0 | min | | Median cumulative merge time across primary shards | | 0 | min | | Max cumulative merge time across primary shards | | 3.99985 | min | | Cumulative merge throttle time of primary shards | | 1.66413 | min | | Min cumulative merge throttle time across primary shards | | 0 | min | | Median cumulative merge throttle time across primary shards | | 0 | min | | Max cumulative merge throttle time across primary shards | | 1.66413 | min | | Cumulative refresh time of primary shards | | 0.297567 | min | | Cumulative refresh count of primary shards | | 141 | | | Min cumulative refresh time across primary shards | | 0 | min | | Median cumulative refresh time across primary shards | | 0.00189167 | min | | Max cumulative refresh time across primary shards | | 0.281667 | min | | Cumulative flush time of primary shards | | 0.666633 | min | | Cumulative flush count of primary shards | | 23 | | | Min cumulative flush time across primary shards | | 0 | min | | Median cumulative flush time across primary shards | | 0.000275 | min | | Max cumulative flush time across primary shards | | 0.664333 | min | | Total Young Gen GC time | | 7.809 | s | | Total Young Gen GC count | | 651 | | | Total Old Gen GC time | | 0 | s | | Total Old Gen GC count | | 0 | | | Store size | | 3.21809 | GB | | Translog size | | 5.12227e-07 | GB | | Heap used for segments | | 0 | MB | | Heap used for doc values | | 0 | MB | | Heap used for terms | | 0 | MB | | Heap used for norms | | 0 | MB | | Heap used for points | | 0 | MB | | Heap used for stored fields | | 0 | MB | | Segment count | | 50 | | | Min Throughput | index-append | 39833.5 | docs/s | | Mean Throughput | index-append | 41103.6 | docs/s | | Median Throughput | index-append | 41187.7 | docs/s | | Max Throughput | index-append | 42569.8 | docs/s | | 50th percentile latency | index-append | 422.194 | ms | | 90th percentile latency | index-append | 667.805 | ms | | 99th percentile latency | index-append | 2576.67 | ms | | 99.9th percentile latency | index-append | 6141.64 | ms | | 100th percentile latency | index-append | 6759.82 | ms | | 50th percentile service time | index-append | 422.194 | ms | | 90th percentile service time | index-append | 667.805 | ms | | 99th percentile service time | index-append | 2576.67 | ms | | 99.9th percentile service time | index-append | 6141.64 | ms | | 100th percentile service time | index-append | 6759.82 | ms | | error rate | index-append | 0 | % | | Min Throughput | wait-until-merges-finish | 0.06 | ops/s | | Mean Throughput | wait-until-merges-finish | 0.06 | ops/s | | Median Throughput | wait-until-merges-finish | 0.06 | ops/s | | Max Throughput | wait-until-merges-finish | 0.06 | ops/s | | 100th percentile latency | wait-until-merges-finish | 16252.9 | ms | | 100th percentile service time | wait-until-merges-finish | 16252.9 | ms | | error rate | wait-until-merges-finish | 0 | % | | Min Throughput | randomized-nested-queries | 19.9 | ops/s | | Mean Throughput | randomized-nested-queries | 19.94 | ops/s | | Median Throughput | randomized-nested-queries | 19.95 | ops/s | | Max Throughput | randomized-nested-queries | 19.97 | ops/s | | 50th percentile latency | randomized-nested-queries | 28.4881 | ms | | 90th percentile latency | randomized-nested-queries | 43.5573 | ms | | 99th percentile latency | randomized-nested-queries | 71.1674 | ms | | 99.9th percentile latency | randomized-nested-queries | 120.874 | ms | | 100th percentile latency | randomized-nested-queries | 129.945 | ms | | 50th percentile service time | randomized-nested-queries | 26.4684 | ms | | 90th percentile service time | randomized-nested-queries | 41.1465 | ms | | 99th percentile service time | randomized-nested-queries | 68.0754 | ms | | 99.9th percentile service time | randomized-nested-queries | 106.919 | ms | | 100th percentile service time | randomized-nested-queries | 125.263 | ms | | error rate | randomized-nested-queries | 0 | % | | Min Throughput | randomized-term-queries | 24.99 | ops/s | | Mean Throughput | randomized-term-queries | 25 | ops/s | | Median Throughput | randomized-term-queries | 25 | ops/s | | Max Throughput | randomized-term-queries | 25 | ops/s | | 50th percentile latency | randomized-term-queries | 7.88739 | ms | | 90th percentile latency | randomized-term-queries | 11.0647 | ms | | 99th percentile latency | randomized-term-queries | 15.2192 | ms | | 100th percentile latency | randomized-term-queries | 22.8419 | ms | | 50th percentile service time | randomized-term-queries | 6.12509 | ms | | 90th percentile service time | randomized-term-queries | 8.878 | ms | | 99th percentile service time | randomized-term-queries | 12.6071 | ms | | 100th percentile service time | randomized-term-queries | 20.7344 | ms | | error rate | randomized-term-queries | 0 | % | | Min Throughput | randomized-sorted-term-queries | 15.99 | ops/s | | Mean Throughput | randomized-sorted-term-queries | 15.99 | ops/s | | Median Throughput | randomized-sorted-term-queries | 15.99 | ops/s | | Max Throughput | randomized-sorted-term-queries | 15.99 | ops/s | | 50th percentile latency | randomized-sorted-term-queries | 15.7295 | ms | | 90th percentile latency | randomized-sorted-term-queries | 21.3126 | ms | | 99th percentile latency | randomized-sorted-term-queries | 37.9242 | ms | | 100th percentile latency | randomized-sorted-term-queries | 46.5997 | ms | | 50th percentile service time | randomized-sorted-term-queries | 13.7363 | ms | | 90th percentile service time | randomized-sorted-term-queries | 19.2022 | ms | | 99th percentile service time | randomized-sorted-term-queries | 34.7268 | ms | | 100th percentile service time | randomized-sorted-term-queries | 44.1017 | ms | | error rate | randomized-sorted-term-queries | 0 | % | | Min Throughput | match-all | 5 | ops/s | | Mean Throughput | match-all | 5 | ops/s | | Median Throughput | match-all | 5 | ops/s | | Max Throughput | match-all | 5 | ops/s | | 50th percentile latency | match-all | 6.94698 | ms | | 90th percentile latency | match-all | 10.5454 | ms | | 99th percentile latency | match-all | 14.2265 | ms | | 100th percentile latency | match-all | 20.7373 | ms | | 50th percentile service time | match-all | 4.8253 | ms | | 90th percentile service time | match-all | 8.11081 | ms | | 99th percentile service time | match-all | 10.3598 | ms | | 100th percentile service time | match-all | 16.6433 | ms | | error rate | match-all | 0 | % | | Min Throughput | nested-date-histo | 1 | ops/s | | Mean Throughput | nested-date-histo | 1 | ops/s | | Median Throughput | nested-date-histo | 1 | ops/s | | Max Throughput | nested-date-histo | 1 | ops/s | | 50th percentile latency | nested-date-histo | 634.563 | ms | | 90th percentile latency | nested-date-histo | 684.109 | ms | | 99th percentile latency | nested-date-histo | 738.059 | ms | | 100th percentile latency | nested-date-histo | 777.098 | ms | | 50th percentile service time | nested-date-histo | 631.674 | ms | | 90th percentile service time | nested-date-histo | 680.681 | ms | | 99th percentile service time | nested-date-histo | 736.705 | ms | | 100th percentile service time | nested-date-histo | 775.063 | ms | | error rate | nested-date-histo | 0 | % | | Min Throughput | randomized-nested-queries-with-inner-hits_default | 17.83 | ops/s | | Mean Throughput | randomized-nested-queries-with-inner-hits_default | 17.99 | ops/s | | Median Throughput | randomized-nested-queries-with-inner-hits_default | 17.99 | ops/s | | Max Throughput | randomized-nested-queries-with-inner-hits_default | 18 | ops/s | | 50th percentile latency | randomized-nested-queries-with-inner-hits_default | 33.6864 | ms | | 90th percentile latency | randomized-nested-queries-with-inner-hits_default | 52.4591 | ms | | 99th percentile latency | randomized-nested-queries-with-inner-hits_default | 81.8842 | ms | | 99.9th percentile latency | randomized-nested-queries-with-inner-hits_default | 568.065 | ms | | 100th percentile latency | randomized-nested-queries-with-inner-hits_default | 617.288 | ms | | 50th percentile service time | randomized-nested-queries-with-inner-hits_default | 30.9282 | ms | | 90th percentile service time | randomized-nested-queries-with-inner-hits_default | 48.6987 | ms | | 99th percentile service time | randomized-nested-queries-with-inner-hits_default | 68.033 | ms | | 99.9th percentile service time | randomized-nested-queries-with-inner-hits_default | 100.44 | ms | | 100th percentile service time | randomized-nested-queries-with-inner-hits_default | 616.964 | ms | | error rate | randomized-nested-queries-with-inner-hits_default | 0 | % | | Min Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s | | Mean Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s | | Median Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s | | Max Throughput | randomized-nested-queries-with-inner-hits_default_big_size | 16 | ops/s | | 50th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 34.9921 | ms | | 90th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 51.1011 | ms | | 99th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 69.6266 | ms | | 99.9th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 87.5509 | ms | | 100th percentile latency | randomized-nested-queries-with-inner-hits_default_big_size | 114.079 | ms | | 50th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 33.2397 | ms | | 90th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 49.1744 | ms | | 99th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 66.9583 | ms | | 99.9th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 84.0898 | ms | | 100th percentile service time | randomized-nested-queries-with-inner-hits_default_big_size | 111.146 | ms | | error rate | randomized-nested-queries-with-inner-hits_default_big_size | 0 | % | ---------------------------------- [INFO] SUCCESS (took 2042 seconds) ---------------------------------- ``` > Note: [Workloads List](https://opensearch.org/docs/latest/benchmark/user-guide/installing-benchmark/#choosing-appropriate-hardware)

Locust (Load Testing Framework)

Information
Locust is a Python load testing framework. It allows you to define user behavior using Python code and simulate thousands of users performing actions concurrently. You can install Locust from PyPI. - Interest URLs - [Website](https://locust.io/) - [Documentation](https://docs.locust.io/en/stable/) - [Installation](https://docs.locust.io/en/stable/installation.html) - Requeriments - Python >= 3.9
Advantages and disadvantages
- Advantages: - User-Friendly Task Definition: Locust allows users to define tasks using Python code, making it easy to emulate genuine user behavior. You can create tasks like making HTTP queries, decoding responses, or performing custom actions. - Distributed Load Generation: Locust supports distributed load generation, allowing you to spread the load across multiple machines. This scalability is useful for testing complex systems and applications. - Real-Time Web-Based Monitoring: Locust provides a web-based interface that displays live metrics such as response times, requests per second, and user counts. This monitoring feature helps identify performance issues. - Simulating User Behavior: Users can establish user behavior patterns through scenarios. You can specify the number of users, their actions, and frequency, enabling realistic load testing. - Performance Metrics and Reporting: Locust tracks various performance indicators. You can link these metrics with other monitoring systems or export them for analysis. - Disadvantages: - Python Dependency: Since Locust is Python-based, familiarity with Python is necessary for creating test scripts. - Limited Protocol Support: While it excels in HTTP-based load testing, it may not be suitable for protocols beyond HTTP. - Less Suitable for High-Concurrency Workloads: Although other tools may handle more requests per second, Locust's low overhead per user makes it better suited for highly concurrent workloads.
Installation
```console root@ubuntu2204:/home/vagrant# apt install python3-pip root@ubuntu2204:/home/vagrant# pip install locust ```
Locust Test
- Create Locust Test ```console root@ubuntu2204:/home/vagrant# touch locustfile.py ``` - Add Locust Test Code ```python from locust import HttpUser, task class HelloWorldUser(HttpUser): @task def hello_world(self): self.client.get("/hello") self.client.get("/world") ``` - Run Test ```console root@ubuntu2204:/home/vagrant# locust [2024-06-25 10:47:39,471] ubuntu2204/INFO/locust.main: Starting web interface at http://0.0.0.0:8089 [2024-06-25 10:47:39,477] ubuntu2204/INFO/locust.main: Starting Locust 2.29.0 ``` > Note: [Locust Web Interface](https://docs.locust.io/en/stable/quickstart.html#locust-s-web-interface)

Fluentd (Data Collector for Unified Logging Layer)

Information
Fluentd is an open source data collector that allows you to unify the collection and consumption of data from various sources and destinations. Fluentd provides a unified layer of record between data sources and backend systems. This allows you to decouple data sources from target systems. One of Fluentd's most useful features is its ability to filter and enrich records as they are collected. Users can create custom filtering rules to remove unwanted records, add additional fields to records, and restructure data to make it more useful for further analysis. In addition, it can also be used to monitor system status and alert users if a problem occurs. Users can set up alerts based on certain criteria, such as the number of errors in a given time period, and receive real-time notifications if a problem occurs. - Interest URLs - [Website](https://www.fluentd.org/) - [Documentation](https://docs.fluentd.org/) - [Installation](https://docs.fluentd.org/installation) - Requeriments - Linux, MacOS or Docker
Advantages and disadvantages
- Advantages: - Pluggable Architecture: Fluentd's strength lies in its pluggable architecture. It seamlessly integrates with various data sources and outputs through a vast library of over 500 community-contributed plugins. - Real-Time Data Processing: Fluentd excels in real-time data processing, making it ideal for handling substantial data volumes efficiently. Whether capturing logs from servers or managing IoT device data streams, Fluentd performs well in high-throughput scenarios. - Cross-Platform Support: Fluentd runs on both Windows and Linux, making it a versatile choice for different environments. - Better Memory Usage: Written in CRuby, Fluentd consumes fewer resources compared to Logstash. It scales well and is efficient for small to medium-sized deployments. - Disadvantages: - Decentralized Ecosystem: Fluentd's decentralized plugin ecosystem means that it hosts fewer official plugins (around 10) compared to Logstash. However, it compensates with community-contributed plugins. - Transport and Buffering: Fluentd provides an in-built buffering system that can be configured based on needs. In contrast, Logstash relies on external queues like Redis or Kafka for consistency. Choose Fluentd when you want more straightforward configuration or Logstash when reliability is critical. - Parsing Complexity: While Fluentd excels in parsing both structured and unstructured logs, Logstash relies on plugins for log parsing.
Installation
```console root@ubuntu2204:/home/vagrant# curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-fluent-package5-lts.sh | sh root@ubuntu2204:/home/vagrant# sudo systemctl start fluentd.service root@ubuntu2204:/home/vagrant# sudo systemctl status fluentd.service root@ubuntu2204:/home/vagrant# curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test root@ubuntu2204:/home/vagrant# tail -n 1 /var/log/fluent/fluentd.log ``` > Note: [Installing Fluent Package](https://docs.fluentd.org/installation/install-by-deb#installing-fluent-package)
Post-Installation
- Configuration ```console root@ubuntu2204:/home/vagrant# cat /etc/fluent/fluentd.conf ``` - Logging ```console root@ubuntu2204:/home/vagrant# cat /var/log/fluent/fluentd.log ``` > Note: [Post-Installation Guide](https://docs.fluentd.org/installation/post-installation-guide)
Integrating Wazuh with Fluentd
Following the official guide, it is possible to integrate Wazuh and Fluentd (Wazuh v4.8.0) > Note: [Forward Alerts with Fluentd](https://wazuh.com/blog/forward-alerts-with-fluentd/)

Tsung (Distributed Load Testing Tool)

Information
Tsung is an open source distributed load testing tool. It can be used to stress servers that handle multiple protocols. Tsung is not limited to a single protocol and can simulate thousands of concurrent virtual users. It can also be distributed across multiple client machines to increase the testing load. - Interest URLs - [Website](http://tsung.erlang-projects.org/) - [Documentation](http://tsung.erlang-projects.org/user_manual/) - [Installation](http://tsung.erlang-projects.org/user_manual/installation.html) - Requeriments - SO: Linux - Software: Erlang/OTP R16B03, pgsql module, mysql module, mochiweb libs, gnuplot and perl5, python and matplotlib - Disadvantages - Complex installation - Uses Jabber/XMPP protocol
Advantages and disadvantages
- Advantages: - Erlang-Based. - Protocol Support: Tsung can stress test various protocols, including HTTP, WebDAV, LDAP, MySQL, PostgreSQL, SOAP, and XMPP. - Stability. - Distributed Load Testing: Tsung is designed for distributed load testing, making it suitable for large-scale applications. - Automated Statistics. - Disadvantages: - Learning Curve: As an Erlang-based tool, Tsung may have a learning curve for users unfamiliar with the language. - Complexity: Setting up and configuring Tsung for specific scenarios can be intricate. - Limited Protocol: While it supports several protocols, it may not cover all possible use cases or niche protocols.
Installation
To install without using the packages, the following dependencies must be satisfied: - Erlang/OTP R16B03 and up - pgsql module - mysql module - mochiweb libs - gnuplot and perl5 - Bash It is also necessary to compile the code (you have to download it beforehand): ```code root@ubuntu2204:/home/vagrant# ./configure root@ubuntu2204:/home/vagrant# make root@ubuntu2204:/home/vagrant# make install ``` Another option is to download the packages: [Tsung (Dist)](http://tsung.erlang-projects.org/dist/)
Tsung Tests
Tsung uses XML files to run the tests. The learning curve is steep and does not provide example code to test the tool.

Cypress (Testing Frameworks for Javascript) (E2E Tests)

Information
Cypress is a next-generation front-end testing tool designed for modern web applications. Here are some key points: 1. Testing Framework: - Cypress allows you to create tests for your web apps easily. - You can debug tests visually and run them automatically in your CI builds. - It uses Mocha and Chai for test organization and assertions. 2. How It Works: - Cypress runs directly in the browser, providing real-time feedback. - Debugging failed tests is straightforward using familiar in-browser developer tools. - It eliminates flaky tests by interacting with your app consistently. 3. CI Integration: - Integrate Cypress with your existing CI provider for early failure detection. - Use Docker images or bring your own setup. 4. Cypress Cloud: - Optimize test runs with parallelization, load balancing, and spec prioritization. - Visually review and debug CI failures using Test Replay. - Monitor test suite health with detailed analytics. In summary, Cypress simplifies end-to-end testing and enhances your testing workflow. - Interest URLs - [Website](https://www.cypress.io/) - [Documentation](https://docs.cypress.io/guides/overview/why-cypress) - [Installation](https://www.cypress.io/) - Requeriments - SO: Windows, MacOS, and Linux - Required Software: LTS Release of Node.js and NPM.
Advantages and disadvantages
- Advantages: - Speed: Cypress executes tests quickly, allowing you to see results promptly. - User-Friendly Interface: Its simple and intuitive interface makes it accessible even for inexperienced developers. - Reliability: Tests written in Cypress are less likely to fail compared to other automation tools. - Flexibility: Cypress can handle various tasks, including end-to-end testing, unit testing, and integration testing. - Stability: Many companies have successfully used Cypress in production for years. - Active Community: Cypress has a vibrant community with numerous plugins and integrations available. - Browser-Based: It runs directly in the browser, eliminating the need for additional installations. - Disadvantages: - Limited Cross-Browser Support: Cypress may not support all browsers equally well. - No Native Mobile App Testing: Unlike some tools, Cypress lacks direct support for native mobile app testing. - Single Browser Session: It restricts access to multiple browser sessions during test execution. - No Direct Multiple Windows/Tabs Support: Cypress doesn't directly handle multiple windows or tabs.
Installation
```console root@ubuntu2204:/home/vagrant# apt update root@ubuntu2204:/home/vagrant# curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash root@ubuntu2204:/home/vagrant# nvm install 20 root@ubuntu2204:/home/vagrant# node -v root@ubuntu2204:/home/vagrant# npm -v root@ubuntu2204:/home/vagrant# npm install cypress --save-dev ```

Conclusions

Of all the tools analyzed, the most notable are: Artillery, Playwright, Fluentd and Cypress. Artillery and Cypress can also be integrated to work together. Cypress can be used in place of Playwright but not integrated with Artillery. Cypress has in its favor that it was previously used to test the Wazuh Dashboard. Likewise, for Fluentd there is documentation to integrate it and use it with Wazuh. Cypress, from the tests carried out, seems more versatile and powerful than Playwright. It all depends on the specific use you want to give it.

Of the rest of the tools, I would highlight OpenSearch Benchmark for Indexer Cluster tests (it is a very specific Benchmark, for OpenSearch). Locust could also be useful if you want to measure loading times in requests. Although it is true that this could be done with other tools.

MARCOSD4 commented 1 week ago

Metrics analysis tools

Summary table

Tool Configuration Languaje Multiplatform Requirements Generate report Automated analysis
Prometheus and Grafana YML Yes None Yes Yes
Netdata None (custom) Yes gcc, make, curl No Yes
Nagios - - - - -
Zabbix - - - - -
Prometheus and Grafana
Prometheus is a Open Source systems monitoring and alerting toolkit. Prometheus collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. Prometheus consists of multiple components, like the main server which scrapes and stores time series data, client libraries for instrumenting application code, an alert manager to handle alerts, etc. Prometheus scrapes metrics from instrumented jobs, either directly or via an intermediary push gateway for short-lived jobs. It stores all scraped samples locally and runs rules over this data to either aggregate and record new time series from existing data or generate alerts. Grafana can be used to visualize the collected data. Grafana enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they are stored. It supports querying Prometheus, so both platforms can be used to monitor and visualize the metrics of our systems. - Interest URLs: - [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) Website - [Prometheus](https://prometheus.io/docs/introduction/overview/) and [Grafana](https://grafana.com/docs/grafana/latest/) documentation - [Prometheus integration with grafana](https://grafana.com/docs/grafana/latest/datasources/prometheus/?pg=oss-prom&plcmt=deploy-box-1) and [video examples](https://prometheus.io/docs/tutorials/visualizing_metrics_using_grafana/) - Advantages: - The installation of Prometheus and Grafana is very quick and simple. - YML configuration file. - The UI is user-friendly. - Prometheus and Grafana integration is simple. - Setting up discrepancy alerts using YML files It is necessary to install and configure [Node Exporter](https://prometheus.io/docs/guides/node-exporter/#installing-and-running-the-node-exporter). The integration is very simple, it is done through the Grafana dashboard (UI). In it, Grafana is configured to generate a dashboard through the information that Prometheus sends. There are many dashboards and many different options and configurations to obtain the metrics. It has been tested with a very complete Node Exporter dashboard in which we can see a multitude of graphs about the system metrics.
Examples
![image](https://github.com/wazuh/wazuh-qa/assets/101177239/2ca77cbd-ae7e-4023-9142-e4564b80668a) ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/d7a5e388-095a-4407-af2a-1837d13d5e21) ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/170db849-ff7c-4f16-8242-6a6aedf1e291)
I think this is a good tool to get metrics such as CPU, memory, file descriptors, disk operations, hardware usage of all Wazuh components: Agent, Manager, Indexer, Dashboard. It is flexible, as we can choose what kind of data we want to plot, and it offers a very complete user interface with many configurable options. Another positive aspect of these tools is that it is possible to configure an [alert module](https://prometheus.io/docs/alerting/latest/alertmanager/) in which, if the values obtained by Prometheus differ from a threshold value, an alert will be generated with the corresponding information. This is done by constructing rules in a YML file. The configuration is really simple, you only need to create a YML file with the rule and add its path to the Prometheus configuration file. Therefore, this allows to analyse the data automatically and alert when a discrepancy with the "accepted" values is found.
Examples
I have performed a simple test where if the CPU use exceeds 2%, an alert is generated. The YML file with the rule that triggers the alert is: ```console groups: - name: example_alert_rules rules: - alert: HighCPUUsage expr: avg(rate(node_cpu_seconds_total{mode="system"}[1m])) > 0.02 labels: severity: critical annotations: summary: "High CPU Usage" description: "The CPU usage is above 2%." ``` And then specify the path to the file in the configuration file: ```console rule_files: - "rule.yml" ``` The result is: ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/908cd0cd-1524-4b70-a2ee-838e15665843) ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/35dbe75e-8376-4a76-97a4-0cb1ea89b13b)
In conclusion, I think that these tools largely fulfill the capabilities requested in this issue.
Netdata
Netdata is an open-source, real-time performance and health monitoring tool for systems and applications. It provides comprehensive insights into the performance of your entire IT infrastructure, offering detailed metrics and visualizations. There are 3 components in the Netdata structure: Netdata Agents, to monitor the physical or virtual nodes of your infrastructure, including all applications and containers running on them; Netdata Parents, to create observability centralization pointswithin your infrastructure, to offload Netdata Agents functions from your production systems, to provide high-availability of your data, increased data retention and isolation of your nodes; and Netdata Cloud combining all your infrastructure, all your Netdata Agents and Parents, into one uniform, distributed, scalable, monitoring database, offering custom dashboards, troubleshooting tools, etc. - Interest URLs: - [Netdata website](https://www.netdata.cloud/) - [Netdata documentation](https://learn.netdata.cloud/docs/deployment-guides/) - [Netdata video demo](https://www.youtube.com/watch?v=PCyNLdo6lzI) - Requirements: - Linux and MacOS. - Advantages: - Installation is too simple. - The UI is user-friendly. - Disadvantages: - Rule configuration is more complex. The installation is completed by logging into Netdata and executing a single command. After this, we can access the Netdata dashboard and get graphs of different system metrics:
Examples ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/b407e0a5-41dd-4d38-a957-48d0e5f823f6) ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/963bdfc3-9d3b-4fd0-9834-5bb29a762c79)
In this tool, there are a number of preconfigured alerts with default values to indicate when the metrics obtained exceed these values. It is also possible to create new customised rules. The configuration of the alerts is quite simple and is done in .conf files with the specifications in the [documentation](https://learn.netdata.cloud/docs/alerts-&-notifications/alert-configuration-reference).
Examples
The file containing the rule is: ```console template: 10min_cpu_usage on: system.cpu class: Utilization type: System component: CPU host labels: _os=linux lookup: average -10m unaligned of user,system,softirq,irq,guest units: % every: 1m warn: $this > (($status >= $WARNING) ? (1) : (2)) crit: $this > (($status == $CRITICAL) ? (3) : (4)) delay: down 15m multiplier 1.5 max 1h summary: System CPU utilization info: Average CPU utilization over the last 10 minutes (excluding iowait, nice and steal) to: sysadmin ``` ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/360122cf-046f-4714-8c63-173763a19405) ![image](https://github.com/wazuh/wazuh-qa/assets/101177239/1f31c5e4-24fe-48cf-8cd8-fd43425c6fc2)
Like Prometheus and Grafana, Netdata is a tool that can fulfill the capabilities requested in this issue, but I think the configuration of the alerts is more complex than in the previous tools.
Nagios
Nagios is an open-source monitoring system designed to monitor systems, networks, and infrastructure. It provides monitoring capabilities to ensure the availability and performance of IT systems and services. There are different types of Nagios solutions, but we should focus on Nagios Core and Nagios XI. Nagios XI is capable of monitoring hundreds of different types of applications, services, protocols, and computer hardware through the use of built-in capabilities and third-party extensions and addons. Nagios Core is an Open Source system and network monitoring application. It watches hosts and services that you specify, alerting you when things go bad and when they get better. - Interest URLs: - [Nagios website](https://www.nagios.org/) - [Nagios documentation](https://www.nagios.org/documentation/) - [Video demo](https://www.nagios.com/products/nagios-xi/) - Requirements: - SO: RHEL, CentOS, Oracle Linux, Debian, Ubuntu. - PHP version 5.4 - 8.2. - Disadvantages: - There are quite a few problems when installing Nagios by following the Nagios documentation. Tested on RHEL and Ubuntu machines, all without success. - The installation has only worked for me on Centos 7. - The documentation is less intuitive. - The configuration to monitor the system is more complex. In order to obtain system metrics, it is necessary to install a number of plugins and make changes to the Nagios configuration, which is more tedious than the previous tools. Therefore, we will first move forward with further research on the above tools, and if necessary, further research on Nagios will be carried out.
Zabbix
Zabbix is software that monitors numerous parameters of a network and the health and integrity of servers, virtual machines, applications, services, databases, websites, the cloud, and more. Zabbix uses a flexible notification mechanism that allows users to configure email-based alerts for virtually any event. Zabbix offers reporting and data visualization features based on the stored data. It is composed of different components, like the Zabbix server, DataBase storage, a Web interface and a Zabbix agent that are deployed on monitoring targets to actively monitor local resources and applications and report the gathered data to the Zabbix server. - Interest URLs: - [Zabbix website](https://www.zabbix.com/) - [Zabbix documentation](https://www.zabbix.com/documentation/current/en/manual) - [Demo video](https://www.youtube.com/watch?v=7inJAmqyc0g) - Requirements: - SQL database, PHP. - Linux, MacOS, etc. - Disadvantages: - The installation and configuration of Zabbix is considerably more complex than the previous tools. - Documentation and UI are less intuitive. - Its configuration takes more time. Due to the disadvantages provided by this tool and given that there are other tools that are simpler and offer better features, I do not think that Zabbix is an option to consider. Therefore, we will first move forward with further research on the above tools, and if necessary, further research on Zabbix will be carried out.
rafabailon commented 2 days ago

Metrics Analysis Tools

Tool Advantages Disadvantages
Prometheus Easy Installation and Deployment. Metrics Collection. Data Visualization. Scalability. Configuration and Setup. Scalability. Customer Service.
Grafana Customizable Dashboards. User-Friendly Visualization. Customization Effort. Limited Visualization Styles. External Data Storage.
Netdata Real-Time Monitoring. Lightweight. Automatic Dashboard Creation. Limited Historical Data. Complex Configuration.
Nagios Alerting and Incident Management. Compatible Platforms. Documentation and Support. Installation Complexity. UI/UX Design. Scalability.
Zabbix Metrics Collection. Scalability. Incident Management. Installation and Setup. UI/UX Design. Data Visualization.

Advantages and Disadvantages (Explained)

Prometheus
- Advantages: - Easy Installation and Deployment: Prometheus is straightforward to install and deploy. - Metrics Collection: It excels at collecting time series data from servers. - Data Visualization: Although it lacks built-in visualization, it pairs well with Grafana for creating customizable dashboards. - Scalability: Prometheus scales well. - Disadvantages: - Configuration and Setup: Can be development-intensive. - Scalability: Becomes challenging at large scale. - Customer Service: Known for slow response times.
Grafana
- Advantages: - Customizable Dashboards: Grafana allows you to create and customize dashboards for visualizing metrics. - User-Friendly Visualization: Provides a variety of visualization options. - Disadvantages: - Customization Effort: Customizing Grafana dashboards can be time-consuming. - Limited Visualization Styles: Some limitations in visual representation. - External Data Storage: Requires external data storage.
Netdata
- Advantages: - Real-Time Monitoring: Provides real-time insights into system performance. - Lightweight: Minimal resource overhead. - Automatic Dashboard Creation: Automatically generates dashboards for various services. - Disadvantages: - Limited Historical Data: Focuses on real-time data, lacks extensive historical data storage. - Complex Configuration: Setting up custom monitoring can be intricate.
Nagios
- Advantages: - Alerting and Incident Management: Strong alerting capabilities. - Compatible Platforms: Works on various Unix variants. - Documentation and Support: Well-documented and supported. - Disadvantages: - Installation Complexity: Requires additional setup (Apache server, Nagios Plugins). - UI/UX Design: Could be improved. - Scalability: May face challenges at scale.
Zabbix
- Advantages: - Metrics Collection: Robust data collection. - Scalability: Scales well. - Incident Management: Effective alerting and incident handling. - Disadvantages: - Installation and Setup: Installation guide lacks details on database and web server setup. - UI/UX Design: Could be enhanced. - Data Visualization: Requires integration with other tools like Grafana.
rafabailon commented 2 days ago

Update

Research must be expanded to the tools that are already being used. Specifically, we must review how hardware resources are monitored (tests and footprints). Also, alerts and specific Wazuh measurements are generated that are not possible to cover with an external tool.

Rebits commented 23 hours ago

Tools for saturation of wazuh modules (internal) @santipadilla :large_blue_circle:

Rebits commented 23 hours ago

Tools for running processes and system utilization monitoring (external and internal tools) @santipadilla 🔵