Closed cborla closed 1 year ago
Some tasks remain pending for qa to carry out its tests:
Also, could you check our template for new Issue?
In this template we define a list of items that are necessary and facilitate our work
Note: This issue is blocked by https://github.com/wazuh/wazuh/issues/13077
It is closed because it is being worked on https://github.com/wazuh/wazuh-qa/issues/2947
Tester | PR commit |
---|---|
@CamiRomero | 032cc7 |
OS | OS version | Deployment | Image/AMI | Notes |
---|---|---|---|---|
Centos | 8 | AWS | ami-029496e60f56b4b13 |
wazuh-manager |
wazuh-agent |
---|---|
4.4.0 | - |
It proposed two improvements:
It was discussed with the development team and we decided to improve these issues in another PR.
π‘ Everything seems to be working properly. Several suggestions have been proposed in https://github.com/wazuh/wazuh/issues/14665 https://github.com/wazuh/wazuh/issues/14666 and will be reviewed in future PRs.
The issue is reopened because we still need to test the settings using the configuration uploaded through the API.
Tester | PR commit |
---|---|
@juliamagan | 24d2fd4 |
OS | OS version | Deployment | Image/AMI | Notes |
---|---|---|---|---|
CentOS | CentOS 8 | Vagrant | qactl/centos_8 |
wazuh-manager |
---|
4.4.0 |
Everything seems to work as expected. However, there are some suggestions:
<ossec_config>
blocks. ossec.conf
, it will fail, because there are two <ossec_config>
blocks. (cc @vicferpoy @davidjiglesias).wazuh-manager
start dropping events because the queue is full. This is quite important to let the user know that they are losing event analysis (cc @TomasTurina @vikman90).<maximum>
is an error that does not start the wazuh-manager
. Reported in this issue (cc @TomasTurina @vikman90).This will be discussed with the development team.
If we try to update the configuration with the default ossec.conf, it will fail, because there are two
blocks.
This is not related to this development, but it is something we must address. Thus, it should not block this development but a new issue should be opened to investigate it.
No message is displayed when entering a wrong value in the API configuration, taking the default value.
This was actually a bug introduced in this development. A fix has been added in https://github.com/wazuh/wazuh/pull/13608/commits/2f789dc794e6409ea29cb0807ca10bf7a99b9583.
Everything seems to work correctly after the added fix.
π’ | Solved |
π‘ | Pending resolution in this development |
π΅ | Proposed to be fixed in future versions or developments |
β« | Discarded |
After discussions with the development team, the following has been decided:
(1) If we try to update the configuration with the default ossec.conf
, it will fail, because there are two <ossec_config>
blocks π΅
This appears to have been introduced in previous developments within 4.4.0
(does not occur in earlier versions). The following issue wazuh#15076 https://github.com/wazuh/wazuh/issues/15082 has been opened and will be looked into to see if it is fixed in 4.4.0
.
(2) No message is displayed when entering a wrong value in the API configuration, taking the default value π’
This has been fixed in this development itself. After this fix, it has been tested here and it is now working correctly
(3) It is suggested to add a log message for when available credits are consumed. This would be quite useful to let users know that they have to upgrade to a higher plan (cc @TomasTurina @vikman90). β«
It has been discussed with the development team and it has been concluded that it can be confusing for the user and that it is not very useful, so it is accepted to discard this idea.
(4) It is suggested to add a log message for when the wazuh-manager
start dropping events because the queue is full. This is quite important to let the user know that they are losing event analysis (cc @TomasTurina @vikman90). π‘
The development team is in agreement with this idea, and will endeavor to implement it in this very development.
(5) It is suggested to add WARNING logs when parameters are missing in the
The development team is in agreement with this idea, and will endeavor to implement it in this very development.
(6) Check if the normal behavior after exceeding the value of <maximum>
is an error that does not start the wazuh-manager
. Reported in this issue (cc @TomasTurina @vikman90). β«
It has been discussed with the development team and it has been concluded that this is the designed and expected behavior. We accept the discarding of this idea.
We look forward to the implementation of the changes suggested in (3) and (4).
To solve https://github.com/wazuh/wazuh/issues/14665, this warning was added when the maximum
setting is not defined in the eps limits block:
"EPS limit disabled. The maximum value is missing in the configuration block."
To solve:
(4) It is suggested to add a log message for when the wazuh-manager start dropping events because the queue is full. This is quite important to let the user know that they are losing event analysis
This warning was added when wazuh-analysisd
starts dropping because eps limit:
"Events are being dropped due to there are no analysis credits."
Beside, when it stops dropping events, this information message was added:
"Events dropping has stopped due to there are available analysis credits now."
These logs can flood the ossec.log
file if wazuh-analysisd
continues to go in and out of drop state for a period of time. To control this, only the first time in the current hour will display these warning and information messages, and subsequent times will display a debug 2 message until the current system hour changes.
When <maximum>
tag is missing, 9 WARNING messages are showed instead of 1.
π’ | Solved |
π΅ | Proposed to be fixed in future versions or developments |
β« | Discarded |
After talking with the development team, the testing has been approved taking into account the following considerations proposed in the QA review:
(1) If we try to update the configuration with the default ossec.conf
, it will fail, because there are two <ossec_config>
blocks π΅
This appears to have been introduced in previous developments within 4.4.0
(does not occur in earlier versions). The following issue wazuh#15076 https://github.com/wazuh/wazuh/issues/15082 has been opened and will be looked into to see if it is fixed in 4.4.0
.
(2) No message is displayed when entering a wrong value in the API configuration, taking the default value π’
This has been fixed in this development itself. After this fix, it has been tested here and it is now working correctly
(3) It is suggested to add a log message for when available credits are consumed. This would be quite useful to let users know that they have to upgrade to a higher plan. β«
It has been discussed with the development team and it has been concluded that it can be confusing for the user and that it is not very useful, so it is accepted to discard this idea.
(4) It is suggested to add a log message for when the wazuh-manager
start dropping events because the queue is full. This is quite important to let the user know that they are losing event analysis. π’
Log messages are now displayed for the indicated cases. Added in https://github.com/wazuh/wazuh/pull/13608/commits/87aaf72bbacbde295d700b8e0f3cf928770b54cc
(5) It is suggested to add WARNING logs when parameters are missing in the
Log messages are now displayed for the indicated cases. Added in https://github.com/wazuh/wazuh/pull/13608/commits/87aaf72bbacbde295d700b8e0f3cf928770b54cc, although it has been reported that the log appears multiple times. After discussing this with the development team, it seems that this has been happening before. The following issue has been opened to report it wazuh#15188.
(6) Check if the normal behavior after exceeding the value of <maximum>
is an error that does not start the wazuh-manager
. Reported in this issue. β«
It has been discussed with the development team and it has been concluded that this is the designed and expected behavior. We accept the discarding of this idea.
In addition, all analysisd
tests have been released, along with the new ones introduced to test the EPS limitation and the results are both π’ for local and Jenkins.
Description
In order to validate the changes of the branch https://github.com/wazuh/wazuh/tree/dev-cloud-limits, some manual testing is required.
As part of https://github.com/wazuh/wazuh/issues/12512, a new mechanism has been created to limit the amount of EPS than a manager can process.
This mechanism is implemented within the
wazuh-analysisd
daemon and works by using a circular buffer that tracks the total number of events over a defined period of time.Whenever the circular buffer fills up, the events are held within the related queues until some space is freed. This works like a moving average, this is to support event spikes.
In this first iteration, main configuration will be located in global section from
ossec.conf
file. Link to documentation.Configuration
ossec.conf
Events per second (EPS) limitation is disabled by default.
Logs
EPS functionality disabled:
EPS functionality enabled:
Feature validation
State files could be useful to check if EPS limits are working properly.
Test cases
For the following tests, always use the same type of events (for example,
dbsync
) so that they are directed to the same queue. Also, keep in mind that this mechanism works like a moving average (freeing up some space every second), so the results may not always be exactly the same because they depend on the second the events reach the manager, so it should be better to compare between ranges of expected values instead of exact values.[x] Tests cases fresh install
[x] Check that
wazuh-analysisd
stops processing events when the limit is reached. For example, if the eps is 50 and the timeframe is 30, it should stop processing events when reaching 1500 in the last 30 seconds (timeframe) until the moving average frees up some space.[x] Check that
wazuh-analysisd
starts queuing events when the limit is reached and the corresponding queue is not full. For example, if the eps is 50 and the timeframe is 30, it should start queuing events after reaching 1500 in the last 30 seconds (timeframe) and the corresponding queue is not full until the moving average frees up some space.[x] Check that
wazuh-analysisd
starts dropping events when the limit is reached and the corresponding queue is full. For example, if the eps is 50 and the time frame is 30, it should start discarding events when reaching 1500 in the last 30 seconds (timeframe) and the corresponding queue is full until the moving average frees up some space.[x] Check that
wazuh-analysisd
processes queued events first instead of new events when the moving average frees up some space. For example, if the eps is 50 and the time frame is 30, it should start processing queued events after freeing up a slot of the 1500 events from the last 30 seconds (timeframe) instead of new events (which are sent to the end of the queue).[x] Check that
wazuh-analysisd
works as olders versions if the eps is 0.[x] Test the new configuration block.
[x] Tests cases upgrade from 4.3.6 to 4.4.0
[x] Check that
wazuh-analysisd
stops processing events when the limit is reached. For example, if the eps is 50 and the timeframe is 30, it should stop processing events when reaching 1500 in the last 30 seconds (timeframe) until the moving average frees up some space.[x] Check that
wazuh-analysisd
starts queuing events when the limit is reached and the corresponding queue is not full. For example, if the eps is 50 and the timeframe is 30, it should start queuing events after reaching 1500 in the last 30 seconds (timeframe) and the corresponding queue is not full until the moving average frees up some space.[x] Check that
wazuh-analysisd
starts dropping events when the limit is reached and the corresponding queue is full. For example, if the eps is 50 and the time frame is 30, it should start discarding events when reaching 1500 in the last 30 seconds (timeframe) and the corresponding queue is full until the moving average frees up some space.[x] Check that
wazuh-analysisd
processes queued events first instead of new events when the moving average frees up some space. For example, if the eps is 50 and the time frame is 30, it should start processing queued events after freeing up a slot of the 1500 events from the last 30 seconds (timeframe) instead of new events (which are sent to the end of the queue).[x] Check that
wazuh-analysisd
works as olders versions if the eps is 0.[x] Test the new configuration block.
Note: This issue is blocked by https://github.com/wazuh/wazuh/issues/13077