Working on https://github.com/wazuh/wazuh-packages/issues/2879, I noticed that the installation in the RPM system was significantly faster than the installation in the APT system. This behavior could be normal as they are different OSs.
However, I noticed that the Installation assistant in the APT systems is slowed by the APT package manager, specifically while checking the installed packages.
Currently:
In APT systems, we use apt list --installed | grep <package_name> to check if a package is installed.
In RPM systems, we used yum list installed | grep <package_name> to check if a package is installed until 4.8.2. In this PR, the command was changed to rpm -q <package_name> because the original command was causing trouble with the YUM lock. Besides, the new command was faster than the original command.
Description
The aim of this issue is to try to improve the performance of the APT packages check. I've done some research about the consumed time of these commands:
We can see that to check if a package is installed using the apt list option, the CPU consumes about 0.5 seconds (considering user+sys times).
root@ip-172-31-71-48:/home/ubuntu# time apt list --installed | grep -E ^"curl"
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
- Using another option, for example `dpkg` filtering with the `ii` string (check if it's actually installed), the time consumed is significantly lower, taking 0.32 seconds:
```console
root@ip-172-31-71-48:/home/ubuntu# time dpkg -l curl | grep -E '^ii\s'
ii curl 7.81.0-1ubuntu1.16 amd64 command line tool for transferring data with URL syntax
real 0m0.030s
user 0m0.014s
sys 0m0.018s
To reaffirm this hypothesis, I created a simple script to chronometer the time consumed checking a package list. I used the following script:
#!/bin/bash
# List of packages to check
packages=(systemd grep tar coreutils sed gawk curl lsof openssl)
dpkg_total_time=0
apt_total_time=0
# Measure total time for dpkg
dpkg_start_time=$(date +%s%3N)
for package in "${packages[@]}"; do
dpkg -l $package | grep -E '^ii\s' > /dev/null
done
dpkg_end_time=$(date +%s%3N)
dpkg_total_time=$((dpkg_end_time - dpkg_start_time))
# Measure total time for apt
apt_start_time=$(date +%s%3N)
for package in "${packages[@]}"; do
apt list --installed 2>/dev/null | grep -q -E ^"${package}" > /dev/null
done
apt_end_time=$(date +%s%3N)
apt_total_time=$((apt_end_time - apt_start_time))
# Output the results
echo "Total time for dpkg: $dpkg_total_time ms"
echo "Total time for apt: $apt_total_time ms"
The results were the following:
root@ip-172-31-71-48:/home/ubuntu# bash measure.sh
Total time for dpkg: 135 ms
Total time for apt: 5240 ms
[!NOTE]
With the new option, the package list check is 38.81 times faster, in other words, 97,42% faster.
Context
Working on https://github.com/wazuh/wazuh-packages/issues/2879, I noticed that the installation in the RPM system was significantly faster than the installation in the APT system. This behavior could be normal as they are different OSs.
However, I noticed that the Installation assistant in the APT systems is slowed by the APT package manager, specifically while checking the installed packages.
Currently:
apt list --installed | grep <package_name>to check if a package is installed.yum list installed | grep <package_name>to check if a package is installed until 4.8.2. In this PR, the command was changed torpm -q <package_name>because the original command was causing trouble with the YUM lock. Besides, the new command was faster than the original command.Description
The aim of this issue is to try to improve the performance of the APT packages check. I've done some research about the consumed time of these commands:
apt listoption, the CPU consumes about 0.5 seconds (considering user+sys times).WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
curl/jammy-updates,jammy-security,now 7.81.0-1ubuntu1.16 amd64 [installed]
real 0m0.506s user 0m0.497s sys 0m0.011s
To reaffirm this hypothesis, I created a simple script to chronometer the time consumed checking a package list. I used the following script:
The results were the following:
Changes
The involved changes are the following:
See all changes
- https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/common_functions/common.sh#L94 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/common_functions/common.sh#L106 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/common_functions/common.sh#L117 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/common_functions/common.sh#L128 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/install_functions/checks.sh#L473-L478 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/install_functions/installCommon.sh#L124 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/install_functions/installCommon.sh#L604 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/install_functions/installCommon.sh#L632 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/install_functions/installCommon.sh#L661 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/install_functions/installCommon.sh#L690 - https://github.com/wazuh/wazuh-packages/blob/6f5758ec72e9d778ad7d6169037107ca3de84c86/unattended_installer/install_functions/wazuh-offline-installation.sh#L21Tasks
dpkgproposed option fits the requirements.aptoccurrences with thedpkg.