Closed verdx closed 1 year ago
The proposed solutions intend to modify those curls that fetch files and not modify those that are used with pipes. These last ones have enough checks and parameters to prevent errors.
About the ones that fetch files, the solution can be the parameters of the curl
commands to insert the error into the log file.
Two notes here about this:
curl
commands will not be modified to show the errors in the console prompt.curl
performs the action silently, without generating the error.To solve this, new parameters to the curl
commands that fetch files will be added:
-f|--fail
: (HTTP) Fail fast with no output at all on server errors. This is useful to enable scripts and users to better deal with failed attempts. In normal cases when an HTTP server fails to deliver a document, it returns an HTML document stating so (which often also describes why and more). This flag will prevent curl from outputting that and return error 22.-S|--show-error
: When used with [-s, --silent]
, it makes curl show an error message if it fails.This info is extracted from the official documentation of the curl
command.
It could be great to add the --fail-with-body
option, but this was added into the 7.76.0 curl
version, and older supported systems of Wazuh could be affected by this parameter.
Some testing has been performed to check the behavior of the proposed solution. These tests take into account the two modes of executing the script: with and without the -v|--verbose
option that displays the debug messages in the console prompt or not.
Note: the https://raw.githubusercontent.com/wazuh/wazuh/4.6/extensions/elasticsearch/7.x/wazuh-template.json URL does not exists at this moment.
:x: Error case:
davidcr01:~/Wazuh/$ curl -sfSo ./wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.6/extensions/elasticsearch/7.x/wazuh-template.json >> logfile.txt 2>&1
davidcr01:~/Wazuh/$ cat logfile.txt
curl: (22) The requested URL returned error: 404
davidcr01:~/Wazuh$ cat wazuh-template.json
cat: wazuh-template.json: No existe el archivo o el directorio
davidcr01:~/Wazuh/$ curl -sfSo ./wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.6/extensions/elasticsearch/7.x/wazuh-template.json 2>&1 | tee -a logfile.txt
curl: (22) The requested URL returned error: 404
davidcr01:~/Wazuh/$ cat logfile.txt
curl: (22) The requested URL returned error: 404
davidcr01:~/Wazuh$ cat wazuh-template.json
cat: wazuh-template.json: No existe el archivo o el directorio
:heavy_check_mark: Success case:
davidcr01:~/Wazuh$ curl -sfSo ./wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.3/extensions/elasticsearch/7.x/wazuh-template.json >> logfile.txt 2>&1
davidcr01:~/Wazuh$ cat logfile.txt
davidcr01:~/Wazuh$ curl -sfSo ./wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.3/extensions/elasticsearch/7.x/wazuh-template.json 2>&1 | tee -a logfile.txt
davidcr01:~/Wazuh$ cat logfile.txt
:bulb: Notice that the error is displayed only when is generated.
This issue goes to On Hold due to: https://github.com/wazuh/wazuh-ansible/issues/977
Some curls that use pipes can not be modified with the redirection to the log file. This also applies for the curl commands inside conditionals. This is because the curl is inside a conditional, and that conditional works with the error code of the execution of the command. If the command performs a failed curl, and a redirection, the result of the execution will be success. Here is an example: the curl fails but the error code is 0 when the tee command is executed as well
root@ubuntu22:/home/vagrant# eval 'curl -sSo wazuh-offline/wazuh-packages/wazuh-indexer_4.8.0-1_amd64.deb https://packages-dev.wazuh.com/staging/apt/pool/main/w/wazuh-indexer/wazuh-indexer_4.8.0-1_amd64.deb --max-time 300 --retry 5 --retry-delay 5 --fail 2>&1'
curl: (22) The requested URL returned error: 404
root@ubuntu22:/home/vagrant# echo $?
22
root@ubuntu22:/home/vagrant# eval 'curl -sSo wazuh-offline/wazuh-packages/wazuh-indexer_4.8.0-1_amd64.deb https://packages-dev.wazuh.com/staging/apt/pool/main/w/wazuh-indexer/wazuh-indexer_4.8.0-1_amd64.deb --max-time 300 --retry 5 --retry-delay 5 --fail 2>&1 | tee -a /var/log/wazuh-install.log'
curl: (22) The requested URL returned error: 404
root@ubuntu22:/home/vagrant# echo $?
0
Because of this, some curls that use pipelines can not be modified. This is not a problem, as these curls are checked, and if an error is generated, the script will inform about the error and exit.
The line that downloads the packages has been modified:
package_name="${package}_${package_type}_package"
eval "package_base_url=${package}_${package_type}_base_url"
if output=$(common_curl -sSo "${dest_path}/${!package_name}" "${!package_base_url}/${!package_name}" 2>&1); then
common_logger "The ${package} package was downloaded."
else
common_logger -e "The ${package} package could not be downloaded. Exiting."
eval "echo \${output} ${debug}"
exit 1
fi
This change prints the error output using the debug
variable:
-v
option:
root@ubuntu22:/home/vagrant# bash wazuh-install.sh -dw deb -v
18/08/2023 10:02:54 INFO: Starting Wazuh installation assistant. Wazuh version: 4.8.0
18/08/2023 10:02:54 INFO: Verbose logging redirected to /var/log/wazuh-install.log
Hit:1 https://mirrors.edge.kernel.org/ubuntu jammy InRelease
Hit:2 https://mirrors.edge.kernel.org/ubuntu jammy-updates InRelease
Hit:3 https://mirrors.edge.kernel.org/ubuntu jammy-backports InRelease
Hit:4 https://mirrors.edge.kernel.org/ubuntu jammy-security InRelease
Reading package lists...
Hit:1 https://mirrors.edge.kernel.org/ubuntu jammy InRelease
Hit:2 https://mirrors.edge.kernel.org/ubuntu jammy-updates InRelease
Hit:3 https://mirrors.edge.kernel.org/ubuntu jammy-backports InRelease
Hit:4 https://mirrors.edge.kernel.org/ubuntu jammy-security InRelease
Reading package lists...
18/08/2023 10:03:01 INFO: --- Download Packages ---
18/08/2023 10:03:01 INFO: Starting Wazuh packages download.
18/08/2023 10:03:01 INFO: Downloading Wazuh deb packages for x86_64.
18/08/2023 10:03:03 ERROR: The manager package could not be downloaded. Exiting.
curl: (22) The requested URL returned error: 404
-v
option (no error is displayed in the console):
root@ubuntu22:/home/vagrant# bash wazuh-install.sh -dw deb
18/08/2023 10:03:47 INFO: Starting Wazuh installation assistant. Wazuh version: 4.8.0
18/08/2023 10:03:47 INFO: Verbose logging redirected to /var/log/wazuh-install.log
18/08/2023 10:03:53 INFO: --- Download Packages ---
18/08/2023 10:03:53 INFO: Starting Wazuh packages download.
18/08/2023 10:03:53 INFO: Downloading Wazuh deb packages for x86_64.
18/08/2023 10:03:56 ERROR: The manager package could not be downloaded. Exiting.
The same solution has been applied to download the configuration files and assets.
I have noticed that many commands with the debug
variable are not completely correct. This debug variable is added at the end of complex commands (with pipes) and the debug variable only makes the last command output to be redirected to the log file. Related: #2363.
Two curl commands have been modified: https://github.com/wazuh/wazuh-packages/blob/e9f5519221d235b44eec8e20321f17022af02a86/unattended_installer/install_functions/filebeat.sh#L11 https://github.com/wazuh/wazuh-packages/blob/e9f5519221d235b44eec8e20321f17022af02a86/unattended_installer/install_functions/filebeat.sh#L19
:x: Error case
root@ubuntu22:/home/vagrant# curl -sSo /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/master/extensions/elasticsearch/7.x/wazuh-template.jso --max-time 300 --retry 5 --retry-delay 5 --fail 2>&1 | tee -a /var/log/wazuh-install.log
curl: (22) The requested URL returned error: 404
root@ubuntu22:/home/vagrant# tail -n 1 /var/log/wazuh-install.log
curl: (22) The requested URL returned error: 404
:heavy_check_mark: Success case:
root@ubuntu22:/home/vagrant# curl -sSo /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/master/extensions/elasticsearch/7.x/wazuh-template.json --max-time 300 --retry 5 --retry-delay 5 --fail 2>&1 | tee -a /var/log/wazuh-install.log
:x: Error case:
root@ubuntu22:/home/vagrant# curl -sS https://packages-dev.wazuh.com/staging/filebeat/wazu-filebeat-0.2.tar.gz --max-time 300 --retry 5 --retry-delay 5 --fail | tar -xvz -C /usr/share/filebeat/module 2>&1 | tee -a /var/log/wazuh-install.log
curl: (22) The requested URL returned error: 404
gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
:heavy_check_mark: Success case:
root@ubuntu22:/home/vagrant# curl -sS https://packages-dev.wazuh.com/staging/filebeat/wazuh-filebeat-0.1.tar.gz --max-time 300 --retry 5 --retry-delay 5 --fail | tar -xvz -C /usr/share/filebeat/module 2>&1 | tee -a /var/log/wazuh-install.log
wazuh/
wazuh/module.yml
wazuh/archives/
wazuh/archives/config/
wazuh/archives/config/archives.yml
wazuh/archives/ingest/
wazuh/archives/ingest/pipeline.json
wazuh/archives/manifest.yml
wazuh/alerts/
wazuh/alerts/config/
wazuh/alerts/config/alerts.yml
wazuh/alerts/ingest/
wazuh/alerts/ingest/pipeline.json
wazuh/alerts/manifest.yml
wazuh/_meta/
wazuh/_meta/config.yml
wazuh/_meta/fields.yml
wazuh/_meta/docs.asciidoc
Two curl commands have been modified (same command, different systems) https://github.com/wazuh/wazuh-packages/blob/e9f5519221d235b44eec8e20321f17022af02a86/unattended_installer/install_functions/installCommon.sh#L358 https://github.com/wazuh/wazuh-packages/blob/e9f5519221d235b44eec8e20321f17022af02a86/unattended_installer/install_functions/installCommon.sh#L367
:heavy_check_mark: Success case
root@ubuntu22:/home/vagrant# curl -sSo /tmp/wazuh-install-files/chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb --max-time 100 --retry 5 --retry-delay 5 --fail 2>&1 | tee -a /var/log/wazuh-install.log
:x: Error case
root@ubuntu22:/home/vagrant# curl -sSo /tmp/wazuh-install-files/chrome.deb https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb --max-time 100 --retry 5 --retry-delay 5 --fail 2>&1 | tee -a /var/log/wazuh-install.log
curl: (23) Failure writing output to destination
root@ubuntu22:/home/vagrant# tail -n 1 /var/log/wazuh-install.log
curl: (23) Failure writing output to destination
root@ubuntu22:/home/vagrant#
wazuh-install.sh
The linux command
curl
, used all through the installation assistant, doesn't output progress, errors or any message. When called to use with a pipe that's the intended functioning, but when called with arguments-so
to download some file, those messages would be useful. The calls to it normally even have the"${debug}"
option, which redirects the nonexistent output to the log file.Example of usage with the two options of variable
debug
:Usage with argument
-v
: