On Windows 2019 AWS (used in Footprint tests), pending is barely observable in state file on reboot

[pro-akim]

Wazuh version	Component	Install type	Install method	Platform
4.9.0-rc1	Wazuh component	Agent	Packages	Windows 2019 AWS (ami-0bf33f4cb48993eb)

Analyzing the issue https://github.com/wazuh/wazuh-qa/issues/5705 It was observed that there is a different behavior in Windows 2019 AWS (ami-0bf33f4cb48993eb) to other Windows operating systems reviewed (Vagrant Windows 2019, Desktop 10) (AWS Windows 2012). This behavior has to do with the fact that in this AMI, when the agent is restarted, the state file transitions extremely quickly from pending to connected, altering the morphology of the stress test graphs starting from 4.9.0-Alpha1. This behavior was not present in 4.8.1 and it is understood that the only change made was

• https://github.com/wazuh/wazuh/pull/20425 I must emphasize that the disappearance of the state file after the agent stops should not interfere with the test result

This behavior could not be replicated in all operating systems but specifically in the one mentioned.

---

Details

Testing in AWS

Windows Server 2019 Datacenter 1809 (Build 17763.1999) c5a.2xlarge The c5a.2xlarge instance is in the compute optimized family with 8 vCPUs, 16.0 GiB

4,9,0 Build: https://ci.wazuh.info/job/Test_stress/5554/ B5554_agent_windows.tar.gz

The behaviour is different, almost no pending is shown in the screen (monitoring time 1s)

ossec.conf

```

172.31.11.40

1514 tcp aes 10 60 yes no 10 100 yes 60 ./shared/win_applications_rcl.txt ./shared/win_malware_rcl.txt no yes 60s yes yes 60 50 %WINDIR%\regedit.exe %WINDIR%\system.ini %WINDIR%\win.ini %WINDIR%\SysNative\at.exe %WINDIR%\SysNative\attrib.exe %WINDIR%\SysNative\cacls.exe %WINDIR%\SysNative\cmd.exe %WINDIR%\SysNative\drivers\etc %WINDIR%\SysNative\eventcreate.exe %WINDIR%\SysNative\ftp.exe %WINDIR%\SysNative\lsass.exe %WINDIR%\SysNative\net.exe %WINDIR%\SysNative\net1.exe %WINDIR%\SysNative\netsh.exe %WINDIR%\SysNative\reg.exe %WINDIR%\SysNative\regedt32.exe %WINDIR%\SysNative\regsvr32.exe %WINDIR%\SysNative\runas.exe %WINDIR%\SysNative\sc.exe %WINDIR%\SysNative\schtasks.exe %WINDIR%\SysNative\sethc.exe %WINDIR%\SysNative\subst.exe %WINDIR%\SysNative\wbem\WMIC.exe %WINDIR%\SysNative\WindowsPowerShell\v1.0\powershell.exe %WINDIR%\SysNative\winrm.vbs %WINDIR%\System32\at.exe %WINDIR%\System32\attrib.exe %WINDIR%\System32\cacls.exe %WINDIR%\System32\cmd.exe %WINDIR%\System32\drivers\etc %WINDIR%\System32\eventcreate.exe %WINDIR%\System32\ftp.exe %WINDIR%\System32\net.exe %WINDIR%\System32\net1.exe %WINDIR%\System32\netsh.exe %WINDIR%\System32\reg.exe %WINDIR%\System32\regedit.exe %WINDIR%\System32\regedt32.exe %WINDIR%\System32\regsvr32.exe %WINDIR%\System32\runas.exe %WINDIR%\System32\sc.exe %WINDIR%\System32\schtasks.exe %WINDIR%\System32\sethc.exe %WINDIR%\System32\subst.exe %WINDIR%\System32\wbem\WMIC.exe %WINDIR%\System32\WindowsPowerShell\v1.0\powershell.exe %WINDIR%\System32\winrm.vbs %PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup C:\tmp\syscheck_test\files C:\tmp\syscheck_test\directories .htm$|.jpg$|.png$|.chm$|.pnf$|.evtx$ %PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup\desktop.ini HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer HKEY_LOCAL_MACHINE\Software\Classes\Protocols HKEY_LOCAL_MACHINE\Software\Policies HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\URL HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components HKEY_LOCAL_MACHINE\Software\Classes\.log HKEY_LOCAL_MACHINE\Software\Classes\.db HKEY_LOCAL_MACHINE\Software\Classes\.local HKEY_LOCAL_MACHINE\Software\Classes\Accounts HKEY_LOCAL_MACHINE\Software\Classes\AllProtocols HKEY_LOCAL_MACHINE\Software\Classes\Applications HKEY_LOCAL_MACHINE\Software\Classes\chkfile HKEY_LOCAL_MACHINE\Software\Classes\DesktopBackground HKEY_LOCAL_MACHINE\Software\Classes\DiagnosticLog HKEY_LOCAL_MACHINE\Software\Classes\file HKEY_LOCAL_MACHINE\Software\Classes\http HKEY_LOCAL_MACHINE\Software\Classes\https HKEY_LOCAL_MACHINE\Software\Classes\Installer HKEY_LOCAL_MACHINE\Software\Classes\regedit HKEY_LOCAL_MACHINE\Software\Classes\regfile HKEY_LOCAL_MACHINE\Software\Classes\search HKEY_LOCAL_MACHINE\Software\Classes\sysfile HKEY_LOCAL_MACHINE\Software\Classes\txtfile HKEY_LOCAL_MACHINE\System\Setup HKEY_LOCAL_MACHINE\Software\Classes\batfile HKEY_LOCAL_MACHINE\Software\Classes\cmdfile HKEY_LOCAL_MACHINE\Software\Classes\comfile HKEY_LOCAL_MACHINE\Software\Classes\exefile HKEY_LOCAL_MACHINE\Software\Classes\piffile HKEY_LOCAL_MACHINE\Software\Classes\AllFilesystemObjects HKEY_LOCAL_MACHINE\Software\Classes\Directory HKEY_LOCAL_MACHINE\Software\Classes\Folder HKEY_LOCAL_MACHINE\Security HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurePipeServers\winreg HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx HKEY_CURRENT_USER\Console HKEY_CURRENT_USER\Environment HKEY_CURRENT_USER\Volatile Environment \Enum$ HKEY_LOCAL_MACHINE\Security\Policy\Secrets HKEY_LOCAL_MACHINE\Security\SAM\Domains\Account\Users HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\AppCs HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\DHCP HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSIn HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSOut HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\RPC-EPMap HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\Teredo HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Parameters\Cache HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ADOVMPPackage\Final 300 10 yes 5m 10m 10 no 1h yes yes yes yes 100 yes 1800 60s yes C:\Program Files\Java\jre1.8.0_221\bin\java.exe C:\cis-cat xccdf_org.cisecurity.benchmarks_profile_Level_1_-_Domain_Controller yes yes C:\Program Files\osquery\osqueryd C:\Program Files\osquery\log\osqueryd.results.log C:\Program Files\osquery\osquery.conf yes yes wpk_root.pem yes plain ```

4.8.1 Build: https://ci.wazuh.info/job/Test_stress/5559/ B5559_agent_windows.tar.gz

ossec.conf

```

172.31.4.231

1514 tcp aes 10 60 yes no 10 100 yes 60 ./shared/win_applications_rcl.txt ./shared/win_malware_rcl.txt no yes 60s yes yes 60 50 %WINDIR%\regedit.exe %WINDIR%\system.ini %WINDIR%\win.ini %WINDIR%\SysNative\at.exe %WINDIR%\SysNative\attrib.exe %WINDIR%\SysNative\cacls.exe %WINDIR%\SysNative\cmd.exe %WINDIR%\SysNative\drivers\etc %WINDIR%\SysNative\eventcreate.exe %WINDIR%\SysNative\ftp.exe %WINDIR%\SysNative\lsass.exe %WINDIR%\SysNative\net.exe %WINDIR%\SysNative\net1.exe %WINDIR%\SysNative\netsh.exe %WINDIR%\SysNative\reg.exe %WINDIR%\SysNative\regedt32.exe %WINDIR%\SysNative\regsvr32.exe %WINDIR%\SysNative\runas.exe %WINDIR%\SysNative\sc.exe %WINDIR%\SysNative\schtasks.exe %WINDIR%\SysNative\sethc.exe %WINDIR%\SysNative\subst.exe %WINDIR%\SysNative\wbem\WMIC.exe %WINDIR%\SysNative\WindowsPowerShell\v1.0\powershell.exe %WINDIR%\SysNative\winrm.vbs %WINDIR%\System32\at.exe %WINDIR%\System32\attrib.exe %WINDIR%\System32\cacls.exe %WINDIR%\System32\cmd.exe %WINDIR%\System32\drivers\etc %WINDIR%\System32\eventcreate.exe %WINDIR%\System32\ftp.exe %WINDIR%\System32\net.exe %WINDIR%\System32\net1.exe %WINDIR%\System32\netsh.exe %WINDIR%\System32\reg.exe %WINDIR%\System32\regedit.exe %WINDIR%\System32\regedt32.exe %WINDIR%\System32\regsvr32.exe %WINDIR%\System32\runas.exe %WINDIR%\System32\sc.exe %WINDIR%\System32\schtasks.exe %WINDIR%\System32\sethc.exe %WINDIR%\System32\subst.exe %WINDIR%\System32\wbem\WMIC.exe %WINDIR%\System32\WindowsPowerShell\v1.0\powershell.exe %WINDIR%\System32\winrm.vbs %PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup C:\tmp\syscheck_test\files C:\tmp\syscheck_test\directories .htm$|.jpg$|.png$|.chm$|.pnf$|.evtx$ %PROGRAMDATA%\Microsoft\Windows\Start Menu\Programs\Startup\desktop.ini HKEY_LOCAL_MACHINE\Software\Microsoft\Internet Explorer HKEY_LOCAL_MACHINE\Software\Classes\Protocols HKEY_LOCAL_MACHINE\Software\Policies HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\URL HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Policies HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Windows HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon HKEY_LOCAL_MACHINE\Software\Microsoft\Active Setup\Installed Components HKEY_LOCAL_MACHINE\Software\Classes\.log HKEY_LOCAL_MACHINE\Software\Classes\.db HKEY_LOCAL_MACHINE\Software\Classes\.local HKEY_LOCAL_MACHINE\Software\Classes\Accounts HKEY_LOCAL_MACHINE\Software\Classes\AllProtocols HKEY_LOCAL_MACHINE\Software\Classes\Applications HKEY_LOCAL_MACHINE\Software\Classes\chkfile HKEY_LOCAL_MACHINE\Software\Classes\DesktopBackground HKEY_LOCAL_MACHINE\Software\Classes\DiagnosticLog HKEY_LOCAL_MACHINE\Software\Classes\file HKEY_LOCAL_MACHINE\Software\Classes\http HKEY_LOCAL_MACHINE\Software\Classes\https HKEY_LOCAL_MACHINE\Software\Classes\Installer HKEY_LOCAL_MACHINE\Software\Classes\regedit HKEY_LOCAL_MACHINE\Software\Classes\regfile HKEY_LOCAL_MACHINE\Software\Classes\search HKEY_LOCAL_MACHINE\Software\Classes\sysfile HKEY_LOCAL_MACHINE\Software\Classes\txtfile HKEY_LOCAL_MACHINE\System\Setup HKEY_LOCAL_MACHINE\Software\Classes\batfile HKEY_LOCAL_MACHINE\Software\Classes\cmdfile HKEY_LOCAL_MACHINE\Software\Classes\comfile HKEY_LOCAL_MACHINE\Software\Classes\exefile HKEY_LOCAL_MACHINE\Software\Classes\piffile HKEY_LOCAL_MACHINE\Software\Classes\AllFilesystemObjects HKEY_LOCAL_MACHINE\Software\Classes\Directory HKEY_LOCAL_MACHINE\Software\Classes\Folder HKEY_LOCAL_MACHINE\Security HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\KnownDLLs HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\SecurePipeServers\winreg HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx HKEY_CURRENT_USER\Console HKEY_CURRENT_USER\Environment HKEY_CURRENT_USER\Volatile Environment \Enum$ HKEY_LOCAL_MACHINE\Security\Policy\Secrets HKEY_LOCAL_MACHINE\Security\SAM\Domains\Account\Users HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\AppCs HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\DHCP HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSIn HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\IPTLSOut HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\RPC-EPMap HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\MpsSvc\Parameters\PortKeywords\Teredo HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\Parameters\Cache HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnceEx HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\ADOVMPPackage\Final 300 10 yes 5m 10m 10 no 1h yes yes yes yes 100 yes 1800 60s yes C:\Program Files\Java\jre1.8.0_221\bin\java.exe C:\cis-cat xccdf_org.cisecurity.benchmarks_profile_Level_1_-_Domain_Controller yes yes C:\Program Files\osquery\osqueryd C:\Program Files\osquery\log\osqueryd.results.log C:\Program Files\osquery\osquery.conf yes yes wpk_root.pem yes plain ```

---

The result of this analysis should allow us to define whether this behavior is expected or abnormal and, through this, allow us to define the new graphics as a new standard or make changes to the stress tests.

[vikman90]

Based on a local test of version 4.8.1 conducted in a meeting with @juliamagan, we have observed the following:

• The state and log are consistent, with a maximum delay of 5 seconds due to the configuration settings (agent.state_interval).
• When renewing the agent's key using agent-auth.exe, there is an additional delay corresponding to the wazuh-remoted configuration (remoted.key_update_interval).

It is highly likely that the discrepancies we are observing are due to configuration misalignments. However, considering the following points:

• In version 4.9.0, the only change made was the removal of the state file upon agent shutdown.
• The behavior observed in version 4.9.0 is correct and even better than in version 4.8.1.

For these reasons, we do not believe this issue should be a blocker for the release of version 4.9.0.

[cborla]

Hello @pro-akim Is it possible to repeat the test using version 4.9.0 on an AMI where the unexpected behaviour is reproduced, and then another test on an AMI where it behaves correctly, always with 4.9.0? without destroying the environment.

[pro-akim]

Yes @cborla, the test was always performed with 4.9.0. What is possible is to run the test without destroying the EC2s and then enter them to perform tests (this is how I found the difference against 4.8.1). I have not been able to reproduce the same behavior in Vagrant using the same machines

[vikman90]

Hi guys,

We think that having the ossec.log and _local_internaloptions.conf files should be enough to explain the behavior of file wazuh-agent.state.

Therefore, we don't need access to the environment, but just those files for version 4.9.0:

• ossec.log
• local_internal_options.conf
• wazuh-agent.state (plot)

In the environment where we noted different behavior:

• Windows 2019 AWS
• Vagrant Windows 2019, Desktop 10 or AWS Windows 2012

[cborla]

Footprint metrics information

Main release stage issue #	#
Main footprint metrics issue #	#25092
Version	4.9.0
Release stage #	Beta 1
Tag	https://github.com/wazuh/wazuh/tree/v4.9.0-beta1

Stress test documentation

Packages used

• Repository: packages-dev.wazuh.com

• Package path: pre-release

• Package revision: 1

• Jenkins build: https://ci.wazuh.info/job/Test_stress/5584/

---

Manager

+

Plots

                  

• Logs and configuration

  [ossec_Test_stress_B5584_manager_2024-09-04.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_manager_centos/logs/ossec_Test_stress_B5584_manager_2024-09-04.zip)
• CSV

  [monitor-manager-Test_stress_B5584_manager-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_manager_centos/data/monitor-manager-Test_stress_B5584_manager-pre-release.csv) [Test_stress_B5584_manager_analysisd_events.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_manager_centos/data/Test_stress_B5584_manager_analysisd_events.csv) [Test_stress_B5584_manager_analysisd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_manager_centos/data/Test_stress_B5584_manager_analysisd_state.csv) [Test_stress_B5584_manager_remoted_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_manager_centos/data/Test_stress_B5584_manager_remoted_state.csv)

Centos agent

+

Plots

• Logs and configuration
• CSV

Ubuntu agent

+

Plots

• Logs and configuration
• CSV

Windows agent

+

Plots

              

• Logs and configuration

  [ossec_Test_stress_B5584_windows_2024-09-04.zip](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_agent_windows/logs/ossec_Test_stress_B5584_windows_2024-09-04.zip)
• CSV

  [monitor-winagent-Test_stress_B5584_windows-pre-release.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_agent_windows/data/monitor-winagent-Test_stress_B5584_windows-pre-release.csv) [Test_stress_B5584_windows_agentd_state.csv](https://ci.wazuh.com/data/Test_stress/pre-release/4.9.0/B5584-20m/B5584_agent_windows/data/Test_stress_B5584_windows_agentd_state.csv)

macOS agent

+

Plots

• Logs and configuration
• CSV

Solaris agent

+

Plots

• Logs and configuration
• CSV

[cborla]

Analysis

The stress test was run again for 20 minutes, the result is attached in the previous comment.

Test changes:

• Windows agent debug 2 is enabled. windows.debug=2.
• Test sample rate is every 1 second. FREQUENCY_SCAN = 1

Result

• Graph

• Monitor log As can be seen in the monitor.log file, the reading of the wazuh-agent.state file is approximately every 1 second.

  2024-09-04 19:58:31,406 Getting statistics data from C:\Program Files (x86)\ossec-agent\wazuh-agent.state
  2024-09-04 19:58:31,406 Writing agentd.state info to Test_stress_B5584_windows_agentd_state.csv.
  2024-09-04 19:58:31,923 Writing binary info to monitor-winagent-Test_stress_B5584_windows-pre-release.csv.
  2024-09-04 19:58:32,938 Collecting resources usage of wazuh-agent.exe.
  2024-09-04 19:58:32,938 Getting statistics data from C:\Program Files (x86)\ossec-agent\wazuh-agent.state
  2024-09-04 19:58:32,938 Writing agentd.state info to Test_stress_B5584_windows_agentd_state.csv.
  2024-09-04 19:58:33,454 Writing binary info to monitor-winagent-Test_stress_B5584_windows-pre-release.csv.
  2024-09-04 19:58:34,455 Collecting resources usage of wazuh-agent.exe.
  2024-09-04 19:58:34,455 Getting statistics data from C:\Program Files (x86)\ossec-agent\wazuh-agent.state

• Agent ossec.log When agent debugging is enabled, when updating the wazuh-agent.state file, the message state.c:78 at write_state() is printed: DEBUG: Updating state file. The log shows that the Updating state file message is printed the same second the agent connects to the manager, the order in which they are printed in this case does not determine which came first. Also, it can be observed that the thread in charge of creating/updating the file is created in the same second that the agent is informing that it is connected.

• Pendings There are only 2 peaks in the graph, in the log sequence, which are the only cases where the Updating state file message is above the connection message and in the same second.

  
  2024/09/04 20:02:04 wazuh-agent[6100] state.c:78 at write_state(): DEBUG: Updating state file.
  2024/09/04 20:02:04 wazuh-agent[6100] start_agent.c:365 at agent_handshake_to_server(): INFO: (4102): Connected to the server ([172.31.2.9]:1514/tcp).

2024/09/04 20:03:25 wazuh-agent[5656] state.c:78 at write_state(): DEBUG: Updating state file. 2024/09/04 20:03:25 wazuh-agent[5656] start_agent.c:365 at agent_handshake_to_server(): INFO: (4102): Connected to the server ([172.31.2.9]:1514/tcp).

- CSV file pending state.
![image](https://github.com/user-attachments/assets/73271289-db1f-4e56-bbc7-8efe9d0fa806)

- ossec.log filtered only the messages Updating state file and manager connection message.
[ossec_filtered.zip](https://github.com/user-attachments/files/16882614/ossec_filtered.zip)

### Conclusion
- Even if the sampling rate of the test is set to 1 second, it is not guaranteed to read the pending state of the agent.
- When the agent is started, the thread that creates and updates the wazuh-aget.state file is launched and in the AWS environment, it usually happens after the agent has established the connection with the manager.
- Counting the number of pendings of a csv file does not guarantee that it represents the number of times that the agent disconnected, in this case the sampling frequency is higher, that's why you see more, but it doesn't mean that a delay in the connection can't occur.