wazuh / wazuh-docker

Wazuh - Docker containers
https://wazuh.com
Other
706 stars 400 forks source link

4.5.4 - Dashboard and Indexer containers keeps rebooting #1076

Open Pestage opened 1 year ago

Pestage commented 1 year ago

Hello,

I discovered recently this project and wanted to test it but unfortunately the dashboard and the indexer containers are rebooting in loop.

Here are the logs for the dashboard :

2023-10-24T01:07:57.752329704Z An OpenSearch Dashboards keystore already exists. Overwrite? [y/N] Created OpenSearch Dashboards keystore in /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore
2023-10-24T01:07:59.164926576Z grep: /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml: Permission denied
2023-10-24T01:07:59.165989719Z /wazuh_app_config.sh: line 53: /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml: Permission denied
2023-10-24T01:07:59.485812200Z internal/fs/utils.js:332
2023-10-24T01:07:59.485990451Z     throw err;
2023-10-24T01:07:59.486024895Z     ^
2023-10-24T01:07:59.486050019Z 
2023-10-24T01:07:59.486075522Z Error: EACCES: permission denied, open '/usr/share/wazuh-dashboard/config/opensearch_dashboards.yml'
2023-10-24T01:07:59.486113930Z     at Object.openSync (fs.js:498:3)
2023-10-24T01:07:59.486144565Z     at Object.readFileSync (fs.js:394:35)
2023-10-24T01:07:59.486176387Z     at readYaml (/usr/share/wazuh-dashboard/node_modules/@osd/apm-config-loader/target/utils/read_config.js:37:52)
2023-10-24T01:07:59.486213555Z     at Object.exports.getConfigFromFiles (/usr/share/wazuh-dashboard/node_modules/@osd/apm-config-loader/target/utils/read_config.js:62:22)
2023-10-24T01:07:59.486250789Z     at exports.loadConfiguration (/usr/share/wazuh-dashboard/node_modules/@osd/apm-config-loader/target/config_loader.js:43:38)
2023-10-24T01:07:59.486284344Z     at module.exports (/usr/share/wazuh-dashboard/src/apm.js:58:15)
2023-10-24T01:07:59.486316443Z     at Object.<anonymous> (/usr/share/wazuh-dashboard/src/cli/dist.js:32:18)
2023-10-24T01:07:59.486355195Z     at Module._compile (internal/modules/cjs/loader.js:1085:14)
2023-10-24T01:07:59.486388762Z     at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
2023-10-24T01:07:59.486423607Z     at Module.load (internal/modules/cjs/loader.js:950:32) {
2023-10-24T01:07:59.486453665Z   errno: -13,
2023-10-24T01:07:59.486479502Z   syscall: 'open',
2023-10-24T01:07:59.486505558Z   code: 'EACCES',
2023-10-24T01:07:59.486536593Z   path: '/usr/share/wazuh-dashboard/config/opensearch_dashboards.yml'
2023-10-24T01:07:59.486568123Z }

And the logs for the indexer :

2023-10-24T01:07:46.654014811Z grep: /usr/share/wazuh-indexer/opensearch.yml: Permission denied
2023-10-24T01:07:46.661009616Z grep: /usr/share/wazuh-indexer/opensearch.yml: Permission denied
2023-10-24T01:07:49.841746364Z Exception in thread "main" SettingsException[Failed to load settings from /usr/share/wazuh-indexer/opensearch.yml]; nested: AccessDeniedException[/usr/share/wazuh-indexer/opensearch.yml];
2023-10-24T01:07:49.850778534Z  at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:98)
2023-10-24T01:07:49.850956040Z  at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:118)
2023-10-24T01:07:49.851002101Z  at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:109)
2023-10-24T01:07:49.851039433Z  at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
2023-10-24T01:07:49.851075737Z  at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2023-10-24T01:07:49.851208806Z  at org.opensearch.cli.MultiCommand.execute(MultiCommand.java:104)
2023-10-24T01:07:49.851249128Z  at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2023-10-24T01:07:49.851281970Z  at org.opensearch.cli.Command.main(Command.java:101)
2023-10-24T01:07:49.851314398Z  at org.opensearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:56)
2023-10-24T01:07:49.851349448Z Caused by: java.nio.file.AccessDeniedException: /usr/share/wazuh-indexer/opensearch.yml
2023-10-24T01:07:49.851389723Z  at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
2023-10-24T01:07:49.851427301Z  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
2023-10-24T01:07:49.851460544Z  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
2023-10-24T01:07:49.851497734Z  at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
2023-10-24T01:07:49.851531927Z  at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
2023-10-24T01:07:49.851563877Z  at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
2023-10-24T01:07:49.851600805Z  at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
2023-10-24T01:07:49.851637116Z  at java.base/java.nio.file.Files.newInputStream(Files.java:160)
2023-10-24T01:07:49.851669389Z  at org.opensearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1111)
2023-10-24T01:07:49.851708511Z  at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:96)
2023-10-24T01:07:49.851742518Z  ... 8 more
2023-10-24T01:07:51.109626175Z WARNING: A terminally deprecated method in java.lang.System has been called
2023-10-24T01:07:51.110208329Z WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.6.0.jar)
2023-10-24T01:07:51.110309844Z WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
2023-10-24T01:07:51.110358585Z WARNING: System::setSecurityManager will be removed in a future release
2023-10-24T01:07:52.741238869Z Exception in thread "main" SettingsException[Failed to load settings from /usr/share/wazuh-indexer/opensearch.yml]; nested: AccessDeniedException[/usr/share/wazuh-indexer/opensearch.yml];
2023-10-24T01:07:52.741837035Z  at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:98)
2023-10-24T01:07:52.741951331Z  at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:118)
2023-10-24T01:07:52.742006459Z  at org.opensearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:109)
2023-10-24T01:07:52.742043200Z  at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
2023-10-24T01:07:52.742091833Z  at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
2023-10-24T01:07:52.742144681Z  at org.opensearch.cli.Command.main(Command.java:101)
2023-10-24T01:07:52.742180794Z  at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
2023-10-24T01:07:52.742224043Z  at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
2023-10-24T01:07:52.742257996Z Caused by: java.nio.file.AccessDeniedException: /usr/share/wazuh-indexer/opensearch.yml
2023-10-24T01:07:52.742312880Z  at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
2023-10-24T01:07:52.744432573Z  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
2023-10-24T01:07:52.744588344Z  at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
2023-10-24T01:07:52.744628574Z  at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
2023-10-24T01:07:52.744665415Z  at java.base/java.nio.file.Files.newByteChannel(Files.java:380)
2023-10-24T01:07:52.744702498Z  at java.base/java.nio.file.Files.newByteChannel(Files.java:432)
2023-10-24T01:07:52.744737530Z  at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:422)
2023-10-24T01:07:52.744777393Z  at java.base/java.nio.file.Files.newInputStream(Files.java:160)
2023-10-24T01:07:52.744811279Z  at org.opensearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1111)
2023-10-24T01:07:52.744864900Z  at org.opensearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:96)
2023-10-24T01:07:52.744922436Z  ... 7 more

All I did was to change the output ports 514:514 and 443:5601

Thank you in advance.

gecube commented 1 year ago

@Pestage Hi! What distribution do you use? I mean - what type of installation of wazuh you sticked to? Also if it's a Kubernetes cluster - what is the type and version of it?

Pestage commented 1 year ago

Hi, I used Docker to install Wazuh under a Synology Host (NAS) (There is no Kubernetes cluster)

I finally test on a VPS with the Ubuntu install and it's working fine but it's just a trial VPS. (except that Windows vulnerabilities seems to be not displayed but it's another story)

I would like to use Docker so it will be free and hosted on my NAS. Ports 514 and 443 are already used on my docker host so maybe it is the issue.

cerw commented 1 year ago

Same

Pestage commented 1 year ago

I tried with version 4.6.0 but same issue

jay-oswald commented 11 months ago

I just tried to run it with 4.7.0 and got the same error, running from the single-node folder

gecube commented 11 months ago

@jay-oswald Hi! What are the precise steps to reproduce the issue? Just docker-compose up -d on local machine?

jay-oswald commented 11 months ago

@jay-oswald Hi! What are the precise steps to reproduce the issue? Just docker-compose up -d on local machine?

Here's all the steps I followed, server is unraid using a Docker compose plugin.

Cloned the repo on my laptop. Checked out the v4.7.0 tag Scp the single node folder to the sever Ran the Docker compose script to generate certs Ran the main Docker compose script

The only modification I have made is changed the port for the dashboard, since 443 is used by my reverse proxy.

I checked the compose file and it's set the yaml and ranamed it in the volumes, and that file is at the correct path locally.

It could potentially be permissions issues? Part of my debugging I set every file (except the certs) to 644 and it didn't change anything

The auctal error I get is that it can't read the file, not a perms error

gecube commented 11 months ago
% git clone https://github.com/wazuh/wazuh-docker.git
% cd wazuh-docker
% git checkout v4.7.0
Note: switching to 'v4.7.0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at dcf4842 Merge pull request #1129 from wazuh/chenge_revision_number
% cd single-node
% docker-compose -f generate-indexer-certs.yml run --rm generator
[+] Creating 1/1
 ✔ Network single-node_default  Created                                                                                             0.2s 
[+] Running 5/5
 ✔ generator 4 layers [⣿⣿⣿⣿]      0B/0B      Pulled                                                                                13.1s 
   ✔ edaedc954fb5 Pull complete                                                                                                     8.7s 
   ✔ 573f4d11a520 Pull complete                                                                                                    10.1s 
   ✔ 8f200922197d Pull complete                                                                                                    10.1s 
   ✔ 55a86de68c5c Pull complete                                                                                                    10.1s 
The tool to create the certificates exists in the in Packages bucket
02/12/2023 09:25:06 INFO: Admin certificates created.
02/12/2023 09:25:06 INFO: Wazuh indexer certificates created.
02/12/2023 09:25:06 INFO: Wazuh server certificates created.
02/12/2023 09:25:06 INFO: Wazuh dashboard certificates created.
Moving created certificates to the destination directory
Changing certificate permissions
Setting UID indexer and dashboard
Setting UID for wazuh manager and worker
cp: cannot create regular file '/certificates/root-ca-manager.pem': Operation not permitted
cp: cannot create regular file '/certificates/root-ca-manager.key': Operation not permitted
chown: cannot access '/certificates/root-ca-manager.pem': No such file or directory
chown: cannot access '/certificates/root-ca-manager.key': No such file or directory
% docker compose up -d
[+] Running 43/3
 ✔ wazuh.dashboard 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                 226.1s 
 ✔ wazuh.manager 16 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                               59.6s 
 ✔ wazuh.indexer 13 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿]      0B/0B      Pulled                                                                 243.8s 
[+] Running 14/10
 ✔ Volume "single-node_wazuh-dashboard-config"   Created                                                                            0.1s 
 ✔ Volume "single-node_wazuh_agentless"          Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_api_configuration"  Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_var_multigroups"    Created                                                                            0.0s 
 ✔ Volume "single-node_filebeat_etc"             Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh-indexer-data"       Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_queue"              Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_etc"                Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh-dashboard-custom"   Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_active_response"    Created                                                                            0.0s 
 ✔ Volume "single-node_filebeat_var"             Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_logs"               Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_integrations"       Created                                                                            0.0s 
 ✔ Volume "single-node_wazuh_wodles"             Created                                                                            0.0s 
 ⠋ Container single-node-wazuh.indexer-1         Creating                                                                           0.0s 
 ⠋ Container single-node-wazuh.manager-1         Creating                                                                           0.0s 
Error response from daemon: No such image: wazuh/wazuh-manager:4.7.0

During this process I noticed three things:

  1. the errors when certificate generation, but the PEM files are present
  2. very long time to load images... Wazuh.indexeer is 1.5 GB large ! Screenshot 2023-12-02 at 10 29 12
  3. There is no image for wazuh/wazuh-manager:4.7.0

p.s. please discard the last observation - probably it was an issue with docker desktop running on macbook, as it collects images to make some free space.

alexfornuto commented 11 months ago

Same issue. Steps:

  1. Clone the repo
  2. cd wazuh-docker/single-node
  3. docker-compose -f generate-indexer-certs.yml run --rm generator
  4. Unlike @jay-oswald I imported the single-node docker-compose file into Synology's container manager software.

The error message is exactly the same as OP's.

alexfornuto commented 11 months ago

So I found something I wouldn't call a fix, but a workaround. It looks like the issue is that the uid/gid the docker container runs as is problematic for Synology devices. My workaround is to chmod o+r the files in wazuh_dashboard and wazuh_indexer.

This is not a good idea, it's insecure. This is my first install of Wazuh so I don't know how critical these files are. I've not yet found any official documentation from Wazuh on the user/groups in their docker images, or the validity of changing them in docker-compose.

P.S. If you're like me and tried to stand it up a bunch of times before it worked, you may have to edit wazuh_dashboard/wazuh.yml to remove a bunch of duplicate entries.

JokoBurger commented 1 month ago

I'm having a similar issue with accessing the ssl certificate (also running single node via docker on synology).

I had wazuh 4.8.0 up and running and upgraded without reading the upgrade guide. Now I started from scratch but the indexer is constantly rebooting with the following error: Likely root cause: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/wazuh-indexer/certs/indexer.pem" "read")

The mounts and permissions are correct, and the contianer can also access the .pem file (cat /etc/wazuh-indexer/certs/indexer.pem).

Anyone an idea what could be wrong here? I nearly tried every adjustment but nothing gets it back running.

The Dashboard says this in the logs but I guess that makes sense as the indexer is not correctly coming up: [ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200

Thanks in advance for any suggestions!

gecube commented 1 month ago

@JokoBurger Hi! Probably we need more debug.

The mounts and permissions are correct, and the contianer can also access the .pem file (cat /etc/wazuh-indexer/certs/indexer.pem).

How did you assure it? Maybe something change and permission outside container is not the same like inside. And also one needs to check the user / group under which the process is running.

The Dashboard says this in the logs but I guess that makes sense as the indexer is not correctly coming up: [ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200

yep, if indexer is down, the error will persist.

JokoBurger commented 1 month ago

@gecube thanks for your quick reply!

From outside I have checked the ownership (Synology/Host side) and it is user 1000 (same that is set when chown file from within the container) On the inside the indexer container can open the file as the cert is shown via the cat command.

I have adjusted certificate mount paths as there was an error after upgrading from 4.8.0 to 4.9.0 (expected under etc/... instead of use/...)

I have build the container stack from scratch with only keeping the ossec files and deleted the volumes too.

I can provide you also docker compose config, ossec files and logs if thats needed.

Thanks for your support!

JokoBurger commented 3 weeks ago

Hi @gecube,

just as a follow up I will provide you with the log files and my docker compose file wazuh_xdr-wazuh.dashboard-1_logs.txt wazuh_xdr-wazuh.indexer-1_logs.txt

Hope you can help.

Best, Jonas