roronoasins
commented
1 year ago Review data
| Tester | PR commit |
|---|---|
| @roronoasins | https://github.com/wazuh/wazuh/issues/11334 |
Testing environment
| OS | OS version | Deployment | Image/AMI |
|---|---|---|---|
| Ubuntu | 20.04 | Vagrant | generic/ubuntu2004 |
Tested packages
| OS | Package |
|---|---|
| Ubuntu | Engine's vagrantfile |
The dev team provided a vagrantfile with the engine's mvp
Conclusion
Each problem mentioned in the description has details about each tested case.
Here it is intended to add relevant information about the tests carried out.
Improvements proposed:
I'll start listing the ones related to events and APIs:
-
Error logs that are not that descriptive for a user: 🟡
Referencing the `store` files
**Log Error:** ``` Error: [Catalog] Could not get content [rule] from store, [FileDriver] File [/var/ossec/engine/store/rule] does not exist ``` **These logs appear for these cases:** - List an empty `item-type` - Get a non-existing version for a certain item - Create an item where it already exists - Update a uncorrect id or version for a given `item-type` - Invoke `delete` without an `item-type` at least. The log refers to `name` when it can confuse an user because it is the error log when there is something happening, for example, with the name of a decoder(defined in its yaml file). And here the user may try to delete all the possible combinations by just running the delete option. The `--help` advice is good, but the `name` reference may confuse the user - Delete non-valid item ids and versions - Validate a decoder with no name section - Load an already existing `item-type`Referencing the store and files
**Log Error:** ``` 14:44:31.406714 cmdGraph.cpp:77 ERR[4089 ] Exception while building environment: [exception: [Environment] Error retreiving environment [my-env] from store: [FileDriver] File [/var/ossec/engine/store/my-env] does not exist ``` **These logs appear for these cases:** - Generate a graph file using a non-valid environment - Set with a env name that does not exist - Running test with a non-existent kvdb path - Use a path where no assets are located - Running test with a non-existent environmentLogs that are very short and could be more descriptive, like adding the proper way to call them
**Log Error:** ``` root@engine:/home/vagrant/engine/wazuh/src/engine# /var/ossec/engine/wazuh-engine env set environment is required Run with --help for more information. ``` **This affects to the following cases:** - Set without any name - Set with a not supported environment format - Set the environment that is already active - Invoke delete without an env name - Delete the current env - Delete the current environment when it has not been set yet. or ``` root@engine:/home/vagrant/engine/wazuh/src/engine# /var/ossec/engine/wazuh-engine env get No active environments found root@engine:/home/vagrant/engine/wazuh/src/engine# ``` **This affects to the following cases:** - Get current env when there is no env setUncompleted logging
**Log Error:** ``` Error: Invalid collection type [newtype] ``` or ``` Could not convert: --protocol_queue = 128 Run with --help for more information. ``` **These logs appear for these cases:** - Create a not supported collection type - Use a not expected queue value - Delete an empty `item-type` - Delete non-valid item-typesJSON wrong format
**Log Error:** ``` 10:44:00.649499 cmdKvdb.cpp:77 ERR[3870 ] Error parsing JSON: exception: [Json(jsonString)] Unable to build json document because: Invalid encoding in string. at 33 ``` > We could add some logging to the stack trace so the last logging message is more user-friendly. **These logs appear for these cases:** - Creating a database using a non-well-formatted json input fileLogs that has not the right format
**Log Error:** ``` root@engine:/home/vagrant/engine/wazuh/src/engine# /var/ossec/engine/wazuh-engine env set environment/wazuh/1 Environment created and startedroot@engine:/home/vagrant/engine/wazuh/src/engine# ``` **These logs appear for these cases:** - Set with a not supported environment format - Set an existent environment - Set the environment that is already active - Delete the current env - Delete the current environment when it has not been set yet.Unsupported type
**Log Error:** ``` --input_type: yaml not in {json} ``` This log can be improved so the log do not seem to be a conditional fragment from a source code. **These logs appear for these cases:** - Create a db using a non valid input file format -
Rename the kvdb's
-ioption: 🟡 The format used for--input-typeis-t, so we could use the same format. -
If the
--kvdb-pathoption it could warn the users about this, so they can know that any action that needs the kvdb won't work 🟡Affected cases: Dir that is not kvdb path.
-
The
--kvdb-pathis\sensitive in the end of the string. It should accept both. 🟡 Affected cases:testsubcommand does not accept kvdb paths without an ending\testsubcommand accepts kvdb paths with an ending\
-
When the log does not trigger any rule, it appears empty 🟡 This occurs, for example, in
Run test with different environments: one with a loaded rule and other without itcase. Maybe it could not appear or say that no rule has been triggered. -
Setting a custom protocol queue option, which also appears here. 🟡
-
Check that validation is correct: 🟡 Is there a way to have a trace of the checks that it performs to know if the validate went good? If not, could we add it?
-
The
catalog listcommand must have default rules and filter. 🟡 -
Helper of catalog command: 🟡
A clarification about item-type values expected could be usefull, so the users can have detailed info without run the tool and getting errors.
We could also add a reference to the engine's wiki based on the command. For example, if we use the help option with the whole engine, a reference to the index or very first page of wiki. If we use the help option with a command, a reference to the command's entry.
I will continue with the behavior of the Ruleset:
- When changing some decoders, rules or something you have to update and reload the components. This can be automated with a custom script but I don't know if another approach to update and reload the items could be better/faster. 🟡
It needs to be fixed:
I'll start listing the ones related to events and APIs.
damarisg
Description
Since the team is reworking the engine, we need to cover this new engine rework. This issue will add the new tests to the qa framework and if it was necessary, the manual testing.
First of all, the engine module should be added to the framework so we could start adding modules, their tests and their cases.
Proposed test cases
Considerations
After the research and when the first module approach is created, we will add a module for each type(input, resources, output) and a test for each scenario(events, api, etc.) and its specific Tcases.