The macOS package logs error messages after a system restart if the MANAGER_IP value is not defined

QU3B1M commented 4 months ago

Target version	Related issue	Related PR/dev branch
4.8.0-beta2	https://github.com/wazuh/wazuh/issues/22072	https://github.com/wazuh/wazuh/tree/v4.8.0-beta2

Description

The Test install for macos-12 ARM64 seems to be starting the system before the MANAGER_IP is configured. It could be an error on the tests or on the package side, the agent could be auto-starting after the installation, as reported on the issue https://github.com/wazuh/wazuh/issues/21636.

The log is new and does not comes from the package generation as we can validate by checking the timestamp:

2024/02/28 07:18:47 wazuh-agentd: ERROR: (4112): Invalid server address found: 'MANAGER_IP'

the rest of the logs are from similar datetime

2024/02/28 07:18:54 wazuh-execd: INFO: Started (pid: 576).
2024/02/28 07:18:55 wazuh-agentd: INFO: (1410): Reading authentication keys file.

Failing builds:

Current behavior

The agent is being started before having the MANAGER_IP configured, the test install_agent_test.py::test_check_log_errors fails

12:20:31  install_agent_test.py:91: in test_check_log_errors
12:20:31      assert expected_error == True, assert_description['error']['exception_not_detected'] + " in line {}".format(line)
12:20:31  E   AssertionError: An error was found in the log in line 2024/02/28 07:18:47 wazuh-agentd: ERROR: (4112): Invalid server address found: 'MANAGER_IP'
12:20:31  E     
12:20:31  E   assert False == True

Expected results

The agent should be started after the MANAGER_IP is configured

MarcelKemp commented 4 months ago

Hi @QU3B1M,

After testing locally with an intel64 macOS environment (as I don't have ARM, but it should work similarly), installing the package using the instructions in the documentation, I had no problems and the IP replacement was successful:

https://documentation.wazuh.com/current/installation-guide/wazuh-agent/wazuh-agent-package-macos.html

So, the problem seems to come from the installation test procedure itself. It is possible that the steps indicated in Jenkins are not the correct ones, or are causing the error by trying to start Wazuh before replacing the MANAGER_IP variable.

According to the log messages, it seems that it first tries to start, and in case of failure, it applies the replacement. To solve this problem, you could specify the environment variable before installation, so that when installing the package, the replacement is applied:

echo "WAZUH_MANAGER='10.0.0.2'" > /tmp/wazuh_envs

Please check if this is the issue. If the problem persists and comes from our side, let us know so we can continue.

nbertoldo commented 4 months ago

Hi team, I have tested it on a macOS-12 ARM64 instance and it works as expected.

Check system information

sh-3.2# uname -a
Darwin ip-172-31-40-120.ec2.internal 21.6.0 Darwin Kernel Version 21.6.0: Sun Dec 17 22:56:47 PST 2023; root:xnu-8020.240.18.706.2~1/RELEASE_ARM64_T8101 arm64

sh-3.2# sw_vers
ProductName:    macOS
ProductVersion: 12.7.3
BuildVersion:   21H1015

Download 4.8.0 package

sh-3.2# curl -sO https://packages-dev.wazuh.com/pre-release/macos/wazuh-agent-4.8.0-1.arm64.pkg

Deployment

sh-3.2# echo "WAZUH_MANAGER='192.168.100.150'" > /tmp/wazuh_envs && installer -pkg wazuh-agent-4.8.0-1.arm64.pkg -target /
installer: Package name is Wazuh Agent
installer: Installing at base path /
installer: The install was successful.

Check
configuration

sh-3.2# grep "<address>" /Library/Ossec/etc/ossec.conf 
      <address>192.168.100.150</address>

Check for errors

sh-3.2# cat /Library/Ossec/logs/ossec.log | grep -iE "err|warn"
sh-3.2#

Start Wazuh agent

sh-3.2# /Library/Ossec/bin/wazuh-control start
Starting Wazuh v4.8.0...
Started wazuh-execd...
Started wazuh-agentd...
Started wazuh-syscheckd...
Started wazuh-logcollector...
Started wazuh-modulesd...
Completed.

Check for errors after start-up

sh-3.2# cat /Library/Ossec/logs/ossec.log | grep -iE "err|warn"
sh-3.2#

nbertoldo commented 4 months ago

I have repeated the test 10 times with no errors.

Update test Nº	Result	Observations
1	🟢
2	🟢
3	🟢
4	🟢
5	🟢
6	🟢
7	🟢
8	🟢
9	🟢
10	🟢

QU3B1M commented 4 months ago

Hi team! I did some research on the test's installation process and found some interesting points.

The agent is not being installed with the "one-line installation" nor the WAZUH_MANAGER var is being configured

install_agent.yaml playbook
```yaml --- - hosts: all become: yes vars: build_dir: "{{ hostvars[inventory_hostname].build_dir }}" tasks: - name: Make new dir to store test file: path: "{{ build_dir }}" state: directory - name: Download the wazuh package get_url: url: "{{ package_url }}" dest: "{{ build_dir }}/{{ package_name }}" mode: 0755 validate_certs: no register: download_agent_package retries: 6 delay: 10 until: download_agent_package is success - name: Install wazuh agent become: yes command: "installer -pkg {{ package_name }} -target /" args: chdir: "{{ build_dir }}" ```
The registration is done in another playbook where it just replaces the MANAGER_IP with the actual IP

register_agent.yaml playbook
```yaml --- - hosts: all gather_facts: no become: yes tasks: - name: Set manager IP on MacOS agent lineinfile: path: "/Library/Ossec/etc/ossec.conf" regexp: "
MANAGER_IP
" line: "
{{ manager_ip }}
" - name: Set manager port on MacOS agent lineinfile: path: "/Library/Ossec/etc/ossec.conf" regexp: '(\d+)<\/port>' line: "{{ manager_comm_port }}" - name: Execute authd registering command: "/Library/Ossec/bin/agent-auth -m {{ manager_ip }} -A {{ agent_name }} " when: - manager_authd_port is undefined - agent_name is defined - version is version_compare("3.13.2", "<=") - name: Execute authd registering with a custom authd port command: "/Library/Ossec/bin/agent-auth -m {{ manager_ip }} -A {{ agent_name }} -p {{ manager_authd_port }}" when: - manager_authd_port is defined - agent_name is defined - version is version_compare("3.13.2", "<=") - name: Execute authd registering without agent name command: "/Library/Ossec/bin/agent-auth -m {{ manager_ip }}" when: - agent_name is undefined - manager_authd_port is undefined - version is version_compare("3.13.2", "<=") - name: Restart Wazuh services command: "/Library/Ossec/bin/wazuh-control restart" ```

In the jenkins execution, the difference on time between the ERROR and the next log message is 7 seconds, and between the installation and registration steps execution the difference is ~25 seconds Check the execution of the Install task ( 15:18:26) and the Register's one (15:18:50)

install_agent.yaml

[2024-02-28T15:18:26.308Z] TASK [Install wazuh agent] *****************************************************
[2024-02-28T15:18:26.308Z] changed: [vagrant_host]

register_agent.yaml

[2024-02-28T15:18:50.893Z] TASK [Set manager port on MacOS agent] *****************************************
[2024-02-28T15:18:50.894Z] changed: [vagrant_host]
[2024-02-28T15:19:01.740Z] 
[2024-02-28T15:19:01.741Z] TASK [Restart Wazuh services] **************************************************
[2024-02-28T15:19:01.741Z] changed: [vagrant_host]

And the times on the ossec.log log messages. (ERROR at 07:18:47 and INFO at 07:18:54).

ossec.log

[2024-02-28T15:20:31.202Z] 2024/02/28 07:18:47 wazuh-agentd: ERROR: (4112): Invalid server address found: '
[2024-02-28T15:20:31.202Z] 2024/02/28 07:18:47 wazuh-agentd: ERROR: (1215): No client configured. 
[2024-02-28T15:20:31.202Z] 2024/02/28 07:18:54 wazuh-execd: INFO: Started (pid: 576).

Still not clear where the error comes from.

nbertoldo commented 4 months ago

Hi team, I repeated the test by following the steps in comment.

Install Wazuh agent

sh-3.2# installer -pkg wazuh-agent-4.8.0-1.arm64.pkg -target /
installer: Package name is Wazuh Agent
installer: Installing at base path /
installer: The install was successful.

Check for errors

sh-3.2# cat /Library/Ossec/logs/ossec.log | grep -iE "err|warn"
sh-3.2#

Set manager IP address

sh-3.2# nano /Library/Ossec/etc/ossec.conf
sh-3.2# grep "<address>" /Library/Ossec/etc/ossec.conf
  <address>192.168.100.150</address>
sh-3.2#

Restart agent

sh-3.2# /Library/Ossec/bin/wazuh-control restart
wazuh-modulesd not running...
wazuh-logcollector not running...
wazuh-syscheckd not running...
wazuh-agentd not running...
wazuh-execd not running...
Wazuh v4.8.0 Stopped
Starting Wazuh v4.8.0...
Started wazuh-execd...
Started wazuh-agentd...
Started wazuh-syscheckd...
Started wazuh-logcollector...
Started wazuh-modulesd...
Completed.

Check for errors after start-up

sh-3.2# cat /Library/Ossec/logs/ossec.log | grep -iE "err|warn"
sh-3.2#

nbertoldo commented 4 months ago

Update: I ran the test from scratch and the result was the same.

vikman90 commented 4 months ago

It seems appropriate to hand over the issue to @wazuh/devel-qa for further debugging of the test, as we've been unable to reproduce the error locally.

vikman90 commented 4 months ago

Closed in favor of https://github.com/wazuh/wazuh-jenkins/issues/6341.

rauldpm commented 4 months ago

It has been determined that the package writes those message after a system restart, see the https://github.com/wazuh/wazuh-jenkins/issues/6341#issuecomment-1983724437 conclusion for more info


sh-3.2# curl -sO https://packages-dev.wazuh.com/pre-release/macos/wazuh-agent-4.8.0-1.arm64.pkg
sh-3.2# installer -pkg wazuh-agent-4.8.0-1.arm64.pkg -target /
installer: Package name is Wazuh Agent
installer: Installing at base path /
installer: The install was successful.
sh-3.2# cat /Library/Ossec/logs/ossec.log 
sh-3.2# shutdown -r now
Shutdown NOW!

*** FINAL System shutdown message from vagrant@macos-12 ***                  
System going down IMMEDIATELY                                                  

System shutdown time has arrived
Connection to 10.10.0.250 closed by remote host.
Connection to 10.10.0.250 closed.
╭─user@pop-os ~/Documents/Wazuh/Repositories/wazuh-jenkins/wazuh-jenkins  ‹bug/6341-macos-start*› 
╰─➤  ssh vagrant@10.10.0.250 -p 17754                                                             255 ↵
vagrant@10.10.0.250's password: 
Last login: Thu Mar  7 16:06:34 2024
vagrant@macos-12 ~ % sudo su
sh-3.2# cat /Library/Ossec/logs/ossec.log 
2024/03/07 16:06:34 wazuh-agentd: ERROR: (4112): Invalid server address found: 'MANAGER_IP'
2024/03/07 16:06:35 wazuh-agentd: ERROR: (1215): No client configured. Exiting.

Moving this issue to the wazuh-packages repository
An issue to investigate why the VM is restarted will be opened to the @wazuh/devel-devops team

vikman90 commented 3 months ago

Issue reproduced

installer -pkg wazuh-agent-4.8.0-1.arm64.pkg -target /
cat /Library/Ossec/logs/ossec.log

(empty)
reboot
# (...)
cat /Library/Ossec/logs/ossec.log
2024/03/08 12:16:44 wazuh-agentd: ERROR: (4112): Invalid server address found: 'MANAGER_IP' 2024/03/08 12:16:44 wazuh-agentd: ERROR: (1215): No client configured. Exiting.

Problem Summary

Action

Install the package without defining WAZUH_MANAGER.
Reboot the machine.

Symptom

After system reboot, the error log is found in ossec.log, and the agent stops.

Explanation

Unlike the Linux agent, where the service needs to be explicitly enabled (systemctl enable wazuh-agent), the macOS agent is configured to start automatically on the next reboot. If no IP for the manager is specified at that moment, an error is printed without significant consequences. This behavior is inconsistent with the Linux agent and deviates from the installation guide, which specifies the need to specify the manager's IP during installation.

Options

a) Do nothing and adjust the test to follow the installation guide, considering the error is harmless. b) Install the service disabled (<false />) and require the user to explicitly enable it:

:x: launchctl enable -w /Library/LaunchDaemons/com.wazuh.agent.plist (does not work).
:white_checkmark: Modify the file /Library/LaunchDaemons/com.wazuh.agent.plist_.

Note: I have been unable to modify the file after accessing it; possibly due to SIP or an equivalent mechanism. Perhaps modifying the file after its creation is not a good idea.

Issue status

This issue will remain blocked until the management team makes a decision. It's not a stopper IMO.

davidjiglesias commented 3 months ago

We agreed together with @havidarou this is a Low or Medium impact bug.

vikman90 commented 3 months ago

The @wazuh/devel-qa's tests determined that the agent is restarting unexpectedly. This caused an error in the log, as the agent attempted to start before the manager's IP was configured. This is an expected error and has no impact.

Therefore, we will proceed to close this issue.

wazuh / wazuh-packages