Closed c-bordon closed 1 year ago
[root@redhat9 vagrant]# service wazuh-manager stop
Redirecting to /bin/systemctl stop wazuh-manager.service
[root@redhat9 vagrant]# service wazuh-dashboard start
wazuh-dashboard started
[root@redhat9 vagrant]# log [19:03:01.643] [info][plugins-service] Plugin "visTypeXy" is disabled.
[root@redhat7 vagrant]# service wazuh-dashboard start
Starting wazuh-dashboard (via systemctl): [ OK ]
init.d
, for the Red Hat system if
statement that is incorporated into the Wazuh dashboard base package, a reference is made to the /etc/rc.d/init.d/functions
script, this script exists in Red Hat 7 and 8 but not in Red Hat 9.[root@redhat9 vagrant]# ls -l /etc/rc.d/init.d/functions
ls: cannot access '/etc/rc.d/init.d/functions': No such file or directory
[root@redhat8 vagrant]# ls -l /etc/rc.d/init.d/functions
-rw-r--r--. 1 root root 18434 Feb 15 2021 /etc/rc.d/init.d/functions
[root@redhat7 vagrant]# ls -l /etc/rc.d/init.d/functions
-rw-r--r--. 1 root root 18281 May 22 2020 /etc/rc.d/init.d/functions
/etc/init.d/opensearch-dashboards
file, it is possible to start the opensearch-dashboards
service without the logs being displayed by the terminal.[root@redhat9 vagrant]# chmod +x /etc/init.d/opensearch-dashboards
[root@redhat9 vagrant]# /etc/init.d/opensearch-dashboards status
opensearch-dashboards is not running
[root@redhat9 vagrant]# service opensearch-dashboards status
opensearch-dashboards is not running
[root@redhat9 vagrant]# service opensearch-dashboards start
opensearch-dashboards started
[root@redhat9 vagrant]#
service
command indicates that it has been done via systemctl
[root@redhat7 vagrant]# chmod +x /etc/init.d/opensearch-dashboards
[root@redhat7 vagrant]# service opensearch-dashboards status
opensearch-dashboards is not running
[root@redhat7 vagrant]# service opensearch-dashboards start
Starting opensearch-dashboards (via systemctl): [ OK ]
[root@redhat7 vagrant]#
[root@redhat7 vagrant]# grep "via" /etc/rc.d/init.d/functions
s=$"Starting $prog (via systemctl): "
s=$"Stopping $prog (via systemctl): "
s=$"Reloading $prog configuration (via systemctl): "
s=$"Restarting $prog (via systemctl):
The reported error also occurs on CentOS 9 Stream.
The error could be reproduced by commenting the line https://github.com/wazuh/wazuh-packages/blob/da21a806dc8667626cfc52a8b76a8e570dd84eea/stack/dashboard/base/files/etc/services/wazuh-dashboard#L10
/var/log/opensearch-dashboards/opensearch-dashboards.std[err|out]
chroot --userspec "$user":"$group" "$chroot" sh -c "
cd \"$chdir\"
exec \"$program\"
" >> /var/log/opensearch-dashboards/opensearch-dashboards.stdout 2>> /var/log/opensearch-dashboards/opensearch-dashboards.stderr &
The Wazuh dashboard service, on the other hand, does not redirect any log:
args=-c\\\ /etc/wazuh-dashboard/opensearch_dashboards.yml
...
chroot --userspec "$user":"$group" "$chroot" sh -c "cd \"$chdir\";exec \"$program\" $args" &
By adding the redirection in the Wazuh dashboard service, the reported error has stopped showing and can be consulted in the indicated log file.
The reported error has been reproduced in a Wazuh dashboard package using the OVA v4.3.9 since no log redirection has been done in any version of the Wazuh dashboard package.
[root@wazuh-server wazuh-user]# cat /usr/share/wazuh-dashboard/VERSION
4.3.9
[root@wazuh-server wazuh-user]# service wazuh-dashboard start
wazuh-dashboard started
[root@wazuh-server wazuh-user]# log [17:15:25.416] [info][plugins-service] Plugin "visTypeXy" is disabled.
[root@wazuh-server wazuh-user]# log [17:15:25.566] [info][plugins-system] Setting up [42] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,reportsDashboards,indexManagementDashboards,dashboard,visualizations,visTypeTable,visTypeVega,visTypeTimeline,timeline,visTypeMarkdown,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,bfetch,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,wazuh,savedObjectsManagement]
[root@wazuh-server wazuh-user]# log [17:15:25.786] [info][savedobjects-service] Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations...
log [17:15:25.821] [info][savedobjects-service] Starting saved objects migrations
log [17:15:25.869] [info][plugins-system] Starting [42] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,opensearchUiShared,legacyExport,embeddable,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,reportsDashboards,indexManagementDashboards,dashboard,visualizations,visTypeTable,visTypeVega,visTypeTimeline,timeline,visTypeMarkdown,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,bfetch,charts,visTypeVislib,visTypeTimeseries,visTypeTagcloud,visTypeMetric,discover,wazuh,savedObjectsManagement]
log [17:15:26.088] [info][listening] Server running at https://0.0.0.0:443
log [17:15:26.175] [info][server][OpenSearchDashboards][http] http server running at https://0.0.0.0:443
Different behavior was found in Red Hat 9 and Red Hat 7 for the same OpenSearch Dashboard package, in Red Hat 9, when starting the opensearch-dashboards
service, logs were written to /var/log/opensearch-dashboards
, however, in Red Hat 7, this behavior has not been reproduced despite using the same service
code (where log redirection is specified)
Regarding the message displayed by Wazuh server when starting the service using the service
command, the definition has been found in the /sbin/service
file
[root@redhat9 vagrant]# grep "Redirecting" /sbin/service
echo $"Redirecting to /bin/systemctl ${ACTION} ${SERVICE_MANGLED}${OPTIONS:+ }${OPTIONS}" >&2
In this file, the following conditional is accessed when the wazuh-manager
service is started:
elif [[ $ACTION =~ ^(start|stop|restart|try-restart|reload|reload-or-restart|try-reload-or-restart|force-reload|status|condrestart)$ ]]; then
SERVICE_MANGLED=$(/usr/bin/systemd-escape --mangle "${SERVICE}")
echo $"Redirecting to /bin/systemctl ${ACTION} ${SERVICE_MANGLED}${OPTIONS:+ }${OPTIONS}" >&2
exec /bin/systemctl "${ACTION}" "${SERVICE_MANGLED}" ${OPTIONS}
Instead, for the opensearch-dashboards
and wazuh-dashboard
service, the first conditional is accessed:
if [ -f "${SERVICEDIR}/${SERVICE}" ]; then
# LSB daemons that dies abnormally in systemd looks alive in systemd's eyes due to RemainAfterExit=yes
# lets reap them before next start
if [ "${ACTION}" = 'start' ] && \
[ "$(systemctl show -p ActiveState "${SERVICE}".service --value)" = 'active' ] && \
[ "$(systemctl show -p SubState "${SERVICE}".service --value)" = 'exited' ]; then
/bin/systemctl stop "${SERVICE}".service
fi
# Workaround to be able to "stop" network.service when it's in inactive state using service instead of systemctl
# Useful for manual testing of network
if [ "${SERVICE}" = 'network' ] && [ "${ACTION}" = 'stop' ] && \
[ "$(systemctl show -p ActiveState network.service --value)" = 'inactive' ] && \
[ "$(systemctl show -p SourcePath network.service --value)" = '/etc/rc.d/init.d/network' ]; then
export SYSTEMCTL_SKIP_REDIRECT=1
fi
env -i PATH="$PATH" TERM="$TERM" SYSTEMCTL_IGNORE_DEPENDENCIES="${SYSTEMCTL_IGNORE_DEPENDENCIES}" SYSTEMCTL_SKIP_REDIRECT="${SYSTEMCTL_SKIP_REDIRECT}" "${SERVICEDIR}/${SERVICE}" "${ACTION}" ${OPTIONS}
This is because the "${SERVICEDIR}/${SERVICE}"
conditional gets the following values:
/etc/init.d/opensearch-dashboards
/etc/init.d/wazuh-manager
/etc/init.d/wazuh-dashboard
[root@redhat9 vagrant]# ls -l /etc/init.d/wazuh-manager
ls: cannot access '/etc/init.d/wazuh-manager': No such file or directory
[root@redhat9 vagrant]# ls -l /etc/init.d/opensearch-dashboards
-rwxr-xr-x. 1 root root 4179 Oct 18 17:47 /etc/init.d/opensearch-dashboards
[root@redhat9 vagrant]# ls -l /etc/init.d/wazuh-dashboard
-rwxr-x---. 1 wazuh-dashboard wazuh-dashboard 3682 Oct 17 17:26 /etc/init.d/wazuh-dashboard
After performing some tests, if the /etc/init.d/wazuh-dashboard
service is renamed, when starting the service using the service
command, a redirection to systemctl
is made (third if), fixing the reported error.
Thus, the following solutions are proposed:
systemctl
every time the service
command is used, as is done with the wazuh-manager
service (implies removing the service from init.d)These options should be discussed with the team as they have a major impact on the future of the package.
All services fail to start because certificates have not been configured. No unwanted log is displayed on the terminal.
The Amazon Linux 2 system is not affected by the bug reported in CentOS and Red Hat systems.
The proposed fix on removing the service from init.d
works, as it natively redirects to systemctl
.
A different problem has been found using the service
command, when starting the service a message is received that it has started but when checking the status it is reported that it is stopped.
Same behavior occurs on Fedora when using the service
command.
In Fedora 36 the same reported behavior is observed as in Red Hat 9 and CentOS 9 Stream systems, when starting the service using the service
command the log is shown by terminal.
When renaming the service, the proposed fix corrects the problem by redirecting it to systemctl
.
The reported bug has not been reproduced on Debian systems:
init.d
service from RPM systems so that the native redirection to systemctl
can be carried out.After removing the service file from the RPM SPEC file, the generated package does not reproduce the problem and the use of the service
command redirects to systemctl
.
Package construction using the Packages_builder
pipeline ends with SUCCESS
status: https://devel.ci.wazuh.info/view/Packages/job/Packages_builder/8845/
The package generated with the modifications made in the SPEC file, performs all the actions through systemctl
when used with the service
command, the reported behavior is not observed.
An error was found in the Test_install_stack
pipeline, which does not get the package since it looks for the path: warehouse/test/4.4/rpm/var/wazuh-indexer-4.4.0-wp.1880.x86_64.rpm
when it should be warehouse/test/4.4/rpm/wazuh-indexer-4.4.0-wp.1880.x86_64.rpm
This is because the Wazuh server and Wazuh agent packages are generated in the installation folder (/var
or /opt
), while the Wazuh indexer and Wazuh dashboard use /usr/share
, so they are not created in that folder.
case CSystem.CENTOS:
case CSystem.CENTOS_5:
if(target == TARGET.INDEXER || target == TARGET.DASHBOARD || debug == "yes"){
path = "rpm"
}else{
path = "rpm/${shortInstallation}"
}
break
break
case CSystem.DEBIAN:
if(target == TARGET.INDEXER || target == TARGET.DASHBOARD || debug == "yes"){
path = "deb"
}else{
path = "deb/${shortInstallation}"
}
break
if
statement for the Wazuh dashboard and Wazuh indexer in the following code:
if(target_system == 'rpm' || target_system == 'rpm5' || target_system == 'deb'){
result += install_path + '/'
}
19:30:13 Removing queryWorkbenchDashboards...
19:30:13 Plugin removal complete
19:30:13 Removing anomalyDetectionDashboards...
19:30:13 Plugin removal complete
19:30:13 Removing observabilityDashboards...
19:30:13 Plugin removal complete
19:30:13 Building target platforms: x86_64
19:30:13 Building for target x86_64
19:30:13 Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.pr84NL
19:30:13 + umask 022
19:30:13 + cd /build/rpmbuild/BUILD
19:30:13 + cp /tmp/wazuh-dashboard-base-4.4.0-wp.1880-linux-x64.tar.xz ./
19:30:13 + groupadd wazuh-dashboard
19:30:13 + useradd -g wazuh-dashboard wazuh-dashboard
19:30:13 + exit 0
19:30:13 Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.Wmtv0K
19:30:13 + umask 022
19:30:13 + cd /build/rpmbuild/BUILD
19:30:13 + tar -xf wazuh-dashboard-base-4.4.0-wp.1880-linux-x64.tar.xz
19:30:13 xz: (stdin): Compressed data is corrupt
19:30:13 tar: Child returned status 1
19:30:13 tar: Error is not recoverable: exiting now
19:30:13 error: Bad exit status from /var/tmp/rpm-tmp.Wmtv0K (%build)
19:30:13 RPM build errors:
19:30:13 Bad exit status from /var/tmp/rpm-tmp.Wmtv0K (%build)
System | Build | Result | Artifacts |
---|---|---|---|
CentOS 7 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/527/ | :green_circle: | archive_CentOS7.zip |
CentOS 8 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/528/ | :green_circle: | archive_CentOS8.zip |
Red Hat 7 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/529/ | :green_circle: | archive_RedHat7.zip |
Red Hat 8 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/530/ | :green_circle: | archive_RedHat8.zip |
Red Hat 9 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/531/ | :green_circle: | archive_RedHat9.zip |
Amazon Linux 2 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/532/ | :green_circle: | archive_AmazonLinux2.zip |
init.d
.init.d
does not present any problem (the wazuh-indexer
service fails due to certificates), however, in Red Hat 9 there is a more complex error with init.d
.
command
itself.systemctl
is made and it fails by the daemon
command.The installation of OpenSearch 2.3.0 produces the same result, so the error reported is derived from the OpenSearch code
After applying the fix proposed for Wazuh dashboard (remove the service file from init.d
), redirection to systemctl
is performed. No error found.
/etc/rc.d/init.d/functions
and /etc/init.d/functions
(they have the same content) so that they are installed on the system if these do not exist.service
command, performing the native redirection to systemctl
.systemctl
and the corresponding binary in systems that do not have systemctl
or service
System | Build | Result | Artifacts |
---|---|---|---|
CentOS 7 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/546/console | :green_circle: | archive_CentOS7.zip |
CentOS 8 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/547/console | :green_circle: | archive_CentOS8.zip |
Red Hat 7 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/548/console | :green_circle: | archive_RedHat7.zip |
Red Hat 8 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/549/console | :green_circle: | archive_RedHat8.zip |
Red Hat 9 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/550/console | :green_circle: | archive_RedHat9.zip |
Amazon Linux 2 | https://devel.ci.wazuh.info/view/Tests/job/Test_install_stack/551/console | :green_circle: | archive_AmazonLinux2.zip |
init.d
service.NAME
and VERSION
of the /etc/os-release
file is checked, so if NAME
contains the value Red Hat Enterprise Linux
and VERSION
contains the value 9
or higher, the service in init.d
is removed. This is the behavior that is performed in the Wazuh server SPEC file, so it is expected that the Wazuh indexer and Wazuh dashboard do not remove the init.d
service on CentOS 9 Stream
and Fedora 36
systems, however, Wazuh server does not install the file in the init.d
directory and the cause of this has not been found in the Wazuh server SPEC file.CentOS 9 Stream
and Fedora 36
are not systems on the recommended list, so they cannot be taken into account explicitly. If these systems were to be part of the list of recommended systems at some point, it would suffice to add in the conditional what is necessary for it to match according to the /etc/os-release
file.After an investigation of the problem in CentOS 9 Stream
and Fedora 36
, the following has been discovered:
Wazuh dashboard
and Wazuh indexer
is the same as the Wazuh server
init.d
services are created in /etc/rc.d/init.d
init.d
services are created in /etc/rc.d/init.d
init.d
services are created in /etc/rc.d/init.d
init.d
services are not created in /etc/rc.d/init.d
This behavior of the Wazuh dashboard
and Wazuh indexer
differs from OpenSearch
and OpenSearch dashboards
, which in CentOS 9 Stream
and Red Hat 9
install the init.d
service in /etc/init.d
, this is because these systems have left to actively use init.d
in favor of systemd
and that directory is no longer a link to /etc/rc.d/init.d
, thus presenting the errors reported in this issue.
That said since such systems are not on the recommended list, no further action is required for such behavior in the CentOS 9 Stream
and Fedora 36
systems with the Wazuh server
install process.
The changes requested in the pull requests have been worked on.
Description
Unexpected behavior was detected in the RPM package when manipulating the service with
init.d
By default,
Wazuh dashboard
logs are thrown tostr.out
, what happens when you start the service with init.d is that the logs are constantly thrown to the console:An issue in
Red Hat 9
withinit.d
andWazuh indexer
:Tasks
Wazuh dashboard
Wazuh indexer
Validation
Wazuh dashboard
Wazuh indexer