search

wazuh / wazuh-puppet

Wazuh - Puppet module
https://wazuh.com
GNU General Public License v2.0
46 stars 134 forks source link

Add option for distributed deploy #632

Closed vcerenu closed 11 months ago

vcerenu commented 1 year ago

In the documentation of Wazuh Puppet deploy only have an example for deploy an AIO and it is necessary add another example about deployment with distributed servers.

vcerenu commented 1 year ago

An analysis was made of the changes that are required on the wazuh-puppet repository in order to deploy a distributed environment: Modify the use of the template for the config.yml. The config.yml file that is currently used is a fixed template that only contains this data:

# config.yml
nodes:
   indexer:
     -name: indexer
       ip: 127.0.0.1
   server:
     - name: server
       ip: 127.0.0.1
   dashboard:
     - name: dashboard
       ip: 127.0.0.1

It is necessary to modify the wazuh::certificates class so that when creating the certificates it creates a config.yml file according to the environment that is going to be deployed.

Modify the movement of the .pem and key.pem keys of each node. These files are created on the same server where Wazuh indexer is going to be deployed, so when moving these files it only moves them to the corresponding directories. This process is generated with hard coded data, since just as the config.yml file is fixed, so are the names of the keys and that is why it performs a move of the exact files, without taking into account variables and so on.

In itself, the dynamics of the current AIO and the distributed environment are very different, since each current class of each product is almost independent, so at the time of installation they do not require previous steps as if the distributed environment requires and it is extremely necessary that the certificates of each domain are generated in one location and then moved, or at least a unique root-ca is generated. Then each node creates its certificates with this root-ca in common, which is currently not contemplated and is done in a very simple way on a single server.

vcerenu commented 1 year ago

A new manifest was created where the wazuh::indexer class is declared, which mainly removes many hardcoded options and allows using variables for indexer deployment with more than one node, such as the list of nodes to install.

A new template of the config,yml file was created, to be able to assign variables with the names and IPs of all the nodes for which a certificate is going to be generated. The previous template was too basic and was hardcoded with the names and IPs that belonged to localhost, the new template contemplates different variables assignable by the deployment manifest:

nodes:
   indexer:
     - name: indexer-<%= @indexer_node1_name %>
       ip: <%= @node1-ip %>
     - name: indexer-<%= @indexer_node2_name %>
       ip: <%= @node2-ip %>
     - name: indexer-<%= @indexer_node3_name %>
       ip: <%= @node2-ip %>
   server:
     -name: manager-master
       ip: <%= @master-ip %>
       node_type: master
     -name: manager-worker
       ip: <%= @worker-ip %>
       node_type: worker
   dashboard:
     - name: dashboard
       ip: <%= @dashboard-ip %>

A new opensearch.yml template was created since the previous one had the names of all the node certificates hardcoded, so it is necessary to change these values.

The section where the securityadmin was executed, which only has to be run on the main node of the cluster, was removed from within the indexer manifest, and a new class was created to perform this action later.

We worked on a new deployment script, it is not finished but its objective is to be able to assign the necessary variables for the deployment beforehand, such as the names of the nodes and their IPs:

$node1-ip =
$node2-ip =
$node3-ip =
$master-ip =
$worker-ip =
$dashboard-ip =
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'

class['wazuh::certificates-dist']

node "node1" {
  if $facts['os']['family'] == 'Debian' {
    Class['wazuh::repo'] -> Class['apt::update'] -> Package['wazuh-indexer']
  } else {
    Class['wazuh::repo'] -> Package['wazuh-indexer']
  }
  class { 'wazuh::indexer-dist':
   indexer_node_name => "$indexer_node1_name",
   indexer_network_host => "$node1-ip",
  }
}
node "node2" {
  if $facts['os']['family'] == 'Debian' {
    Class['wazuh::repo'] -> Class['apt::update'] -> Package['wazuh-indexer']
  } else {
    Class['wazuh::repo'] -> Package['wazuh-indexer']
  }
  class { 'wazuh::indexer-dist':
   indexer_node_name => "$indexer_node2_name",
   indexer_network_host => "$node2-ip",
  }
}
node "node3" {
  if $facts['os']['family'] == 'Debian' {
    Class['wazuh::repo'] -> Class['apt::update'] -> Package['wazuh-indexer']
  } else {
    Class['wazuh::repo'] -> Package['wazuh-indexer']
  }
  class { 'wazuh::indexer-dist':
   indexer_node_name => "$indexer_node3_name",
   indexer_network_host => "$node3-ip",
  }
}

Tests must be done and verification of running securityadmin at the end of the installation and subsequent transfer of the previously created certificates.

vcerenu commented 1 year ago

I started by creating the Puppet nodes for the deployment of the distributed environment, 3 nodes will be used in which it is intended to install 1 Wazuh indexer cluster node in each one and then install Wazuh manager master in node 1, Wazuh manager worker in the node 2 and Wazuh dashboard on node 3. The tests of the deployment manifest were carried out, with several errors, firstly with the use of variables, and then with the declaration of the classes.

This error was generated because the names in the Puppet manifest must match the name of the class to create:

Error: Unacceptable name. The name 'wazuh::certificates-dist' is unacceptable as the name of a Host Class Definition (file: /etc/puppetlabs/code/environments/production/modules/wazuh/manifests/certificates_dist.pp, line: 3, column: 1)
Error: Unacceptable location. The name 'wazuh::certificates-dist' is unacceptable in file '/etc/puppetlabs/code/environments/production/modules/wazuh/manifests/certificates_dist.pp' (file: /etc/puppetlabs/code/environments/production/ modules/wazuh/manifests/certificates_dist.pp, line: 3, column: 1)
Error: Language validation logged 2 errors. Giving up

Variable names cannot be declared with "-", so I wanted to take the variable name with "--" as two different variables, so all the characters with problems were removed:

Error: Evaluation Error: Error while evaluating a Function Call, Failed to parse template wazuh/wazuh_config_dist_yml.erb:
   Filepath: /etc/puppetlabs/code/environments/production/modules/wazuh/templates/wazuh_config_dist_yml.erb
   Line: 4
   Detail: undefined local variable or method `ip' for #<Puppet::Parser::TemplateWrapper:0x0000563d80d19520>
  (file: /etc/puppetlabs/code/environments/production/modules/wazuh/manifests/certificates_dist.pp, line: 12, column: 16) on node ip-172-31-18-215.us-west-1. compute.internal

It was possible to execute the class that impacts the template of the config.yml file with the data of the servers that we are going to install and it was possible to execute the creation of the certificates on the Puppet master node:

root@ip-172-31-18-215:/tmp# puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for ip-172-31-18-215.us-west-1.compute.internal
Info: Applying configuration version '1675712457'
Notice: /Stage[main]/Wazuh::Certificates_dist/File[Configure Wazuh Certificates config.yml]/ensure: defined content as '{sha256}81459fdf30b8c9d5c9a29d1bddae46342ae450e3ddbea4d359f98be6bc889628' (corrective)
Notice: /Stage[main]/Wazuh::Certificates_dist/File[/tmp/wazuh-certs-tool.sh]/ensure: defined content as '{mtime}2023-01-20 14:24:28 UTC' (corrective)
Notice: /Stage[main]/Wazuh::Certificates_dist/Exec[Create Wazuh Certificates]/returns: executed successfully (corrective)
Notice: Applied catalog in 0.82 seconds
root@ip-172-31-18-215:/tmp# ls -ltr
total 64
drwx------ 3 root root  4096 Feb  6 13:21 systemd-private-afd44dc8a20b44658aca584c897f907d-systemd-timesyncd.service-VDoiXh
drwx------ 3 root root  4096 Feb  6 13:21 systemd-private-afd44dc8a20b44658aca584c897f907d-systemd-resolved.service-CcQjZh
drwx------ 3 root root  4096 Feb  6 13:21 systemd-private-afd44dc8a20b44658aca584c897f907d-systemd-logind.service-3FeUFf
drwx------ 3 root root  4096 Feb  6 13:21 systemd-private-afd44dc8a20b44658aca584c897f907d-ModemManager.service-5G9tkg
drwx------ 3 root root  4096 Feb  6 13:21 snap-private-tmp
-rwxr----- 1 root root 31528 Feb  6 19:40 wazuh-certs-tool.sh
drwxr-xr-x 2 root root  4096 Feb  6 19:40 wazuh-install-files
-rw------- 1 root root   381 Feb  6 19:40 config.yml
drwxr--r-- 2 root root  4096 Feb  6 19:40 wazuh-certificates
root@ip-172-31-18-215:/tmp# cat config.yml 
nodes:
  indexer:
    - name: indexer-node1
      ip: 172.31.2.205
    - name: indexer-node2
      ip: 172.31.12.65
    - name: indexer-node3
      ip: 172.31.6.144
  server:
    - name: manager-master
      ip: 172.31.2.205
      node_type: master
    - name: manager-worker
      ip: 172.31.12.65
      node_type: worker
  dashboard:
    - name: dashboard
      ip: 172.31.6.144

root@ip-172-31-18-215:/tmp# cd wazuh-certificates/
root@ip-172-31-18-215:/tmp/wazuh-certificates# ls -ltr
total 64
-rwxr--r-- 1 root root 1704 Feb  6 19:40 root-ca.key
-rwxr--r-- 1 root root 1204 Feb  6 19:40 root-ca.pem
-rwxr--r-- 1 root root 1708 Feb  6 19:40 admin-key.pem
-rwxr--r-- 1 root root 1119 Feb  6 19:40 admin.pem
-rwxr--r-- 1 root root 1708 Feb  6 19:40 indexer-node1-key.pem
-rwxr--r-- 1 root root 1245 Feb  6 19:40 indexer-node1.pem
-rwxr--r-- 1 root root 1708 Feb  6 19:40 indexer-node2-key.pem
-rwxr--r-- 1 root root 1245 Feb  6 19:40 indexer-node2.pem
-rwxr--r-- 1 root root 1704 Feb  6 19:40 indexer-node3-key.pem
-rwxr--r-- 1 root root 1245 Feb  6 19:40 indexer-node3.pem
-rwxr--r-- 1 root root 1704 Feb  6 19:40 manager-master-key.pem
-rwxr--r-- 1 root root 1245 Feb  6 19:40 manager-master.pem
-rwxr--r-- 1 root root 1704 Feb  6 19:40 manager-worker-key.pem
-rwxr--r-- 1 root root 1245 Feb  6 19:40 manager-worker.pem
-rwxr--r-- 1 root root 1704 Feb  6 19:40 dashboard-key.pem
-rwxr--r-- 1 root root 1237 Feb  6 13:21 dashboard.pem
root@ip-172-31-18-215:/tmp/wazuh-certificates#

The deployment of the wazuh::repo_use class was successful, configuring the Wazuh repository on each server, a task previously included in the indexer.pp manifest, which maintained an execution relationship that generated errors, because puppet does not allow executing 2 times the same class in one execution, so having this code as required in the manifests and being executed more than 1 time generated errors, now it can be executed autonomously and remove the relationship with the other manifests, in order to install the product that is need only in each deployment.

The next step that generated errors was the installation of IWazuh indexer, because it did not have the required certificates:

root@ip-172-31-2-205:~# puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for ip-172-31-2-205.us-west-1.compute.internal
Info: Applying configuration version '1675712932'
Notice: /Stage[main]/Wazuh::Indexer_dist/File_line[Insert line limits nofile for wazuh-indexer]/ensure: created
Info: /Stage[main]/Wazuh::Indexer_dist/File_line[Insert line limits nofile for wazuh-indexer]: Scheduling refresh of Service[wazuh-indexer]
Notice: /Stage[main]/Wazuh::Indexer_dist/File_line[Insert line limits memlock for wazuh-indexer]/ensure: created
Info: /Stage[main]/Wazuh::Indexer_dist/File_line[Insert line limits memlock for wazuh-indexer]: Scheduling refresh of Service[wazuh-indexer]
Notice: /Stage[main]/Wazuh::Repo/Apt::Key[wazuh]/Apt_key[wazuh]/ensure: created
Notice: /Stage[main]/Apt/File[preferences]/ensure: created
Info: /Stage[main]/Apt/File[preferences]: Scheduling refresh of Class[Apt::Update]
Notice: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/content: 
--- /etc/apt/apt.conf.d/15update-stamp  2022-04-25 14:02:40.000000000 +0000
+++ /tmp/puppet-file20230206-5234-10n9q46   2023-02-06 19:48:54.326629078 +0000
@@ -1 +1,2 @@
+// This file is managed by Puppet. DO NOT EDIT.
 APT::Update::Post-Invoke-Success {"touch /var/lib/apt/periodic/update-success-stamp 2>/dev/null || true";};

Notice: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/content: content changed '{sha256}174cdb519fd06372847e23c20db869b2ff4b593252c0d2a6274d770eae2d92c9' to '{sha256}2e6eb1f5f20262bfc6b7dfb26a302f00b4ab5fee803abd9e07ad8378cce067d5'
Info: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]: Scheduling refresh of Class[Apt::Update]
Notice: /Stage[main]/Wazuh::Repo/Apt::Source[wazuh]/Apt::Setting[list-wazuh]/File[/etc/apt/sources.list.d/wazuh.list]/ensure: defined content as '{sha256}c100aabaf95d9dbf9bfa75c436548000f7b82a1f1cabf3ccc31217dd9464c75d'
Info: /Stage[main]/Wazuh::Repo/Apt::Source[wazuh]/Apt::Setting[list-wazuh]/File[/etc/apt/sources.list.d/wazuh.list]: Scheduling refresh of Class[Apt::Update]
Info: Class[Apt::Update]: Scheduling refresh of Exec[apt_update]
Notice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 event
Notice: /Stage[main]/Wazuh::Indexer_dist/Package[wazuh-indexer]/ensure: created
Info: /Stage[main]/Wazuh::Indexer_dist/Package[wazuh-indexer]: Scheduling refresh of Exec[set recusive ownership of /etc/wazuh-indexer]
Info: /Stage[main]/Wazuh::Indexer_dist/Package[wazuh-indexer]: Scheduling refresh of Exec[set recusive ownership of /usr/share/wazuh-indexer]
Info: /Stage[main]/Wazuh::Indexer_dist/Package[wazuh-indexer]: Scheduling refresh of Exec[set recusive ownership of /var/lib/wazuh-indexer]
Notice: /Stage[main]/Wazuh::Indexer_dist/Exec[ensure full path of /etc/wazuh-indexer/certs]/returns: executed successfully
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs]/owner: owner changed 'root' to 'wazuh-indexer'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs]/group: group changed 'root' to 'wazuh-indexer'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs]/mode: mode changed '0755' to '0500'
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-$(indexer_node_name).pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///tmp/wazuh-certificates/indexer-$(indexer_node_name).pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-$(indexer_node_name)-key.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///tmp/wazuh-certificates/indexer-$(indexer_node_name)-key.pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/root-ca.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///tmp/wazuh-certificates/root-ca.pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///tmp/wazuh-certificates/admin.pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin-key.pem]: Could not evaluate: Could not retrieve information from environment production source(s) file:///tmp/wazuh-certificates/admin-key.pem
Notice: /Stage[main]/Wazuh::Indexer_dist/File[configuration file]/content: 
--- /etc/wazuh-indexer/opensearch.yml   2023-01-13 17:18:35.000000000 +0000
+++ /tmp/puppet-file20230206-5234-1fr3q29   2023-02-06 19:49:44.947726936 +0000
@@ -1,17 +1,17 @@
-network.host: "0.0.0.0"
-node.name: "node-1"
+network.host: "172.31.2.205"
+node.name: "node1"
 cluster.initial_master_nodes:
-- "node-1"
-#- "node-2"
-#- "node-3"
+- "$(indexer_node1_name)"
+- "$(indexer_node2_name)"
+- "$(indexer_node3_name)"
 cluster.name: "wazuh-cluster"
-#discovery.seed_hosts:
-#  - "node-1-ip"
-#  - "node-2-ip"
-#  - "node-3-ip"
-node.max_local_storage_nodes: "3"
-path.data: /var/lib/wazuh-indexer
-path.logs: /var/log/wazuh-indexer
+discovery.seed_hosts:
+- "$(indexer_node1_name)"
+- "$(indexer_node2_name)"
+- "$(indexer_node3_name)"
+node.max_local_storage_nodes: "1"
+path.data: "/var/lib/wazuh-indexer"
+path.logs: "/var/log/wazuh-indexer"

 plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
 plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
@@ -28,15 +28,15 @@
 plugins.security.check_snapshot_restore_write_privileges: true
 plugins.security.enable_snapshot_restore_privilege: true
 plugins.security.nodes_dn:
-- "CN=node-1,OU=Wazuh,O=Wazuh,L=California,C=US"
-#- "CN=node-2,OU=Wazuh,O=Wazuh,L=California,C=US"
-#- "CN=node-3,OU=Wazuh,O=Wazuh,L=California,C=US"
+- "CN=$(indexer_node1_name),OU=Wazuh,O=Wazuh,L=California,C=US"
+- "CN=$(indexer_node2_name),OU=Wazuh,O=Wazuh,L=California,C=US"
+- "CN=$(indexer_node3_name),OU=Wazuh,O=Wazuh,L=California,C=US"
 plugins.security.restapi.roles_enabled:
 - "all_access"
 - "security_rest_api_access"

 plugins.security.system_indices.enabled: true
-plugins.security.system_indices.indices: [".plugins-ml-model", ".plugins-ml-task", ".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opensearch-notifications-*", ".opensearch-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]
+plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

 ### Option to allow Filebeat-oss 7.10.2 to work ###
-compatibility.override_main_response_version: true
\ No newline at end of file
+compatibility.override_main_response_version: true

Notice: /Stage[main]/Wazuh::Indexer_dist/File[configuration file]/content: content changed '{sha256}d95d40b8ee093f122d8015d4a267eddbd92ba3e323c70f2ac7ab7d8ff9e584fe' to '{sha256}01d7d4777271fc86f3441fc80c37e43714ebe7ed06cd56258942f89d423948ca'
Info: /Stage[main]/Wazuh::Indexer_dist/File[configuration file]: Scheduling refresh of Service[wazuh-indexer]
Notice: /Stage[main]/Wazuh::Indexer_dist/Exec[set recusive ownership of /etc/wazuh-indexer]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Wazuh::Indexer_dist/Exec[set recusive ownership of /etc/wazuh-indexer]: Scheduling refresh of Service[wazuh-indexer]
Notice: /Stage[main]/Wazuh::Indexer_dist/Exec[set recusive ownership of /usr/share/wazuh-indexer]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Wazuh::Indexer_dist/Exec[set recusive ownership of /usr/share/wazuh-indexer]: Scheduling refresh of Service[wazuh-indexer]
Notice: /Stage[main]/Wazuh::Indexer_dist/Exec[set recusive ownership of /var/lib/wazuh-indexer]: Triggered 'refresh' from 1 event
Info: /Stage[main]/Wazuh::Indexer_dist/Exec[set recusive ownership of /var/lib/wazuh-indexer]: Scheduling refresh of Service[wazuh-indexer]
Error: Systemd start for wazuh-indexer failed!
journalctl log for wazuh-indexer:
-- Logs begin at Mon 2023-02-06 13:20:09 UTC, end at Mon 2023-02-06 19:49:48 UTC. --
Feb 06 19:49:45 ip-172-31-2-205 systemd[1]: Starting Wazuh-indexer...
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager will be removed in a future release
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager will be removed in a future release
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: uncaught exception in thread [main]
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: Likely root cause: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:983)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:235)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:394)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:256)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:177)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:263)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.node.Node.<init>(Node.java:426)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.node.Node.<init>(Node.java:353)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.cli.Command.main(Command.java:101)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Feb 06 19:49:48 ip-172-31-2-205 systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Feb 06 19:49:48 ip-172-31-2-205 systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Feb 06 19:49:48 ip-172-31-2-205 systemd[1]: Failed to start Wazuh-indexer.

Error: /Stage[main]/Wazuh::Indexer_dist/Service[wazuh-indexer]/ensure: change from 'stopped' to 'running' failed: Systemd start for wazuh-indexer failed!
journalctl log for wazuh-indexer:
-- Logs begin at Mon 2023-02-06 13:20:09 UTC, end at Mon 2023-02-06 19:49:48 UTC. --
Feb 06 19:49:45 ip-172-31-2-205 systemd[1]: Starting Wazuh-indexer...
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Feb 06 19:49:46 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager will be removed in a future release
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: A terminally deprecated method in java.lang.System has been called
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.4.1.jar)
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Feb 06 19:49:47 ip-172-31-2-205 systemd-entrypoint[6459]: WARNING: System::setSecurityManager will be removed in a future release
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: uncaught exception in thread [main]
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: Likely root cause: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.checkPath(DefaultSecurityKeyStore.java:983)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.resolve(DefaultSecurityKeyStore.java:235)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:394)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:256)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:177)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:263)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:730)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:532)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:195)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.node.Node.<init>(Node.java:426)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.node.Node.<init>(Node.java:353)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.cli.Command.main(Command.java:101)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Feb 06 19:49:48 ip-172-31-2-205 systemd-entrypoint[6459]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Feb 06 19:49:48 ip-172-31-2-205 systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Feb 06 19:49:48 ip-172-31-2-205 systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Feb 06 19:49:48 ip-172-31-2-205 systemd[1]: Failed to start Wazuh-indexer.

Notice: /Stage[main]/Wazuh::Indexer_dist/Service[wazuh-indexer]: Triggered 'refresh' from 6 events
Info: Class[Wazuh::Indexer_dist]: Unscheduling all events on Class[Wazuh::Indexer_dist]
Info: Stage[main]: Unscheduling all events on Stage[main]
Notice: Applied catalog in 55.11 seconds
root@ip-172-31-2-205:~#

It is necessary to verify that the opensearch.yml file is being generated correctly with the data that is being sent with the variables.

vcerenu commented 1 year ago

We are having a lot of problems with passing files from the master node to the Puppet agents

Access to the Puppet server directories is not complete, so the configuration we had is not correct:

source => "$indexer_master_node:///tmp/wazuh-certificates/${certfile}",

Since you don't have access to all the servers, the first option we use is to share the Wazuh module files with the other agents:

source => "puppet:///modules/wazuh/${certfile}",

With this configuration we had errors:

Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-$(indexer_node_name).pem]: Could not evaluate: Could not retrieve information from environment production source(s) puppet :///modules/wazuh/indexer-$(indexer_node_name).pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-$(indexer_node_name)-key.pem]: Could not evaluate: Could not retrieve information from environment production source(s ) puppet:///modules/wazuh/indexer-$(indexer_node_name)-key.pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/root-ca.pem]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:// /modules/wazuh/root-ca.pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin.pem]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules /wazuh/admin.pem
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin-key.pem]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:// /modules/wazuh/admin-key.pem

It is not finding the certificate files we moved, step added to the certificate creation manifest certificates_dist.pp:

file { 'Copy all certificates into module':
     ensure => 'directory',
     source => '/tmp/wazuh-certificates/',
     recurse => 'remote',
     path => '/etc/puppetlabs/code/environments/production/modules/wazuh/files/',
     owner => 'root',
     group => 'root',
     mode => '0700',
   }

We proceeded to investigate other ways to share files with the agents, we found the option to create a mountpoint, which is configured within the fileserver.conf file:

  [certificates]
    path /tmp/wazuh-certificates

We proceeded to configure it and add it to the code for its use:

source => "puppet:///certificates/${certfile}",

Errors were generated in the copy but apparently they are due to problems with access permissions to the mountpoint by the agents:

Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node1.pem]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Not authorized to call search on /file_metadata/certificates/indexer-node1.pem with {:rest=>"certificates/indexer-node1.pem", :recurse=>true, :max_files=>0, :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node1.pem]: Could not evaluate: Could not retrieve file metadata for puppet:///certificates/indexer-node1.pem: Error 500 on SERVER: Server Error: Not authorized to call find on /file_metadata/certificates/indexer-node1.pem with {:rest=>"certificates/indexer-node1.pem", :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node1-key.pem]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Not authorized to call search on /file_metadata/certificates/indexer-node1-key.pem with {:rest=>"certificates/indexer-node1-key.pem", :recurse=>true, :max_files=>0, :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node1-key.pem]: Could not evaluate: Could not retrieve file metadata for puppet:///certificates/indexer-node1-key.pem: Error 500 on SERVER: Server Error: Not authorized to call find on /file_metadata/certificates/indexer-node1-key.pem with {:rest=>"certificates/indexer-node1-key.pem", :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/root-ca.pem]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Not authorized to call search on /file_metadata/certificates/root-ca.pem with {:rest=>"certificates/root-ca.pem", :recurse=>true, :max_files=>0, :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/root-ca.pem]: Could not evaluate: Could not retrieve file metadata for puppet:///certificates/root-ca.pem: Error 500 on SERVER: Server Error: Not authorized to call find on /file_metadata/certificates/root-ca.pem with {:rest=>"certificates/root-ca.pem", :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin.pem]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Not authorized to call search on /file_metadata/certificates/admin.pem with {:rest=>"certificates/admin.pem", :recurse=>true, :max_files=>0, :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin.pem]: Could not evaluate: Could not retrieve file metadata for puppet:///certificates/admin.pem: Error 500 on SERVER: Server Error: Not authorized to call find on /file_metadata/certificates/admin.pem with {:rest=>"certificates/admin.pem", :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin-key.pem]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Not authorized to call search on /file_metadata/certificates/admin-key.pem with {:rest=>"certificates/admin-key.pem", :recurse=>true, :max_files=>0, :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}
Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin-key.pem]: Could not evaluate: Could not retrieve file metadata for puppet:///certificates/admin-key.pem: Error 500 on SERVER: Server Error: Not authorized to call find on /file_metadata/certificates/admin-key.pem with {:rest=>"certificates/admin-key.pem", :links=>"manage", :checksum_type=>"sha256", :source_permissions=>"ignore"}

We proceed to continue investigating the assignment of permits to the mountpoints by the agents

vcerenu commented 1 year ago

I continued to troubleshoot to move the files from the master node to the agents.

After declaring a new mountpoint, permissions were assigned to it within the auth.conf file.

         {
             # Allow limited access to files in /tmp/wazuh-credentials:
             match-request: {
                 path: "/etc/puppetlabs/code/environments/production/modules/wazuh/files"
                 type: path
                 method: get
             }
             allow: "*"
             sort-order: 400
             name: "certificates for cluster"
         },

The agent was rerun but it still had permission problems.

The path was changed again so that it takes the files from the Wazuh module but we continued with the previous problems.

The logic was modified and it was decided to move the entire files directory within the Wazuh module, which is where the files that the module is going to share should be left. We continued with errors:

Error: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Permission denied - /etc/ puppetlabs/code/environments/production/modules/wazuh/files/dashboard.pem

Unlike the previous errors, it now apparently recognized files that were inside the files directory.

Going through the official Puppet documentation and forums I found a case with the same error, one of the answers was that if the module had a permissions problem in its construction, that caused it to not be able to move the files. We proceeded to review the other installed Puppet modules, which are Wazuh dependencies and one of them is the "archive" module. I proceeded to take the certificate files to the files directory, inside the archive module location, the code of the wazuh::indexer_dist class was modified to take this new module and managed to take the files inside the destination directory:

root@ip-172-31-12-65:~# puppet agent -t
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for ip-172-31-12-65.us-west-1.compute.internal
Info: Applying configuration version '1675885735'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin-key.pem]/ensure: defined content as '{sha256}a77b9a6fdbb261e226ac72f55a3f3df0c38d5b96f8ba13328afe013138b06331'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/admin.pem]/ensure: defined content as '{sha256}a89afb845d808239034075ee0b4055922c968941b8da5c2b14e67cde91c0e941'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/dashboard-key.pem]/ensure: defined content as '{sha256}9a6a4e33b4aaca011f9cdd4a929690209fb1b8141d490ff8c6f1dc7e884aba56'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/dashboard.pem]/ensure: defined content as '{sha256}b637354b9ecd1e7be2f80617a4e0e34c96efe6248d7821aebcafb2f93150cf74'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node1-key.pem]/ensure: defined content as '{sha256}3f883c9b77d34c072cba4bce828e65850a222af82da7c91691320af060e9f314'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node1.pem]/ensure: defined content as '{sha256}4725dca7fcdb9590e309c2edbc54c21e6bc587a33b6cb8b051548db5d40080d1'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node2-key.pem]/ensure: defined content as '{sha256}4ae962df0704e69996cde8427a40828be3f880df3ab1dc20243e0596b5386906'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node2.pem]/ensure: defined content as '{sha256}533e6ee81b68bbecd677297acb04addf802c7da881c26aa347f5b95ea3fb5d47'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node3-key.pem]/ensure: defined content as '{sha256}af40ff38c209b5f887f3125299e872c360f79df041b04c27dbda0fd05de53bcb'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/indexer-node3.pem]/ensure: defined content as '{sha256}aa575011c79905e1863857289a2210031a43b9b5f12b5f107fc528d16cf36112'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/manager-master-key.pem]/ensure: defined content as '{sha256}d017c2fc4b755b841be2209fcc3f51b01371dd643da99f26e8542674dbd1ac3e'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/manager-master.pem]/ensure: defined content as '{sha256}3c1d289b5ab0921a840c74997093c8ba8ccd0be66ccba2d307855b793f5a5e66'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/manager-worker-key.pem]/ensure: defined content as '{sha256}03bebda187b6f003ec8b7a4060c90a67bbf97e474d7fb19c7737fc193a04bf0b'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/manager-worker.pem]/ensure: defined content as '{sha256}c40bbbf1613dcaf4d24dc04bcdc9b9baf859707f1a6c9606e6dbfb712ba94af4'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/root-ca.key]/ensure: defined content as '{sha256}3253fee031765062d18de25ccc3cd1cbf381e0195f687fc85c07b3c716e4305d'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/root-ca.pem]/ensure: defined content as '{sha256}6be42858dfde9d9044f796e9df32b22c99a492c246d6ded5ba351f0a426ec750'
Notice: /Stage[main]/Wazuh::Indexer_dist/File[/etc/wazuh-indexer/certs/test.zip]/ensure: defined content as '{sha256}6da36d7c637d2d3455f695d647585dfcc013279762d096377157b97de23d66fb'
Notice: /Stage[main]/Wazuh::Indexer_dist/Service[wazuh-indexer]/ensure: ensure changed 'stopped' to 'running'
Info: /Stage[main]/Wazuh::Indexer_dist/Service[wazuh-indexer]: Unscheduling refresh on Service[wazuh-indexer]
Notice: Applied catalog in 43.35 seconds
root@ip-172-31-12-65:~#

Verified that the certificates were in your directory:

root@ip-172-31-6-144:~# ls -ltr /etc/wazuh-indexer/certs/
total 68
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1708 Feb 8 19:48 admin-key.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1119 Feb 8 19:48 admin.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1704 Feb 8 19:48 dashboard-key.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1237 Feb 8 19:48 dashboard.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1708 Feb 8 19:48 indexer-node1-key.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1245 Feb 8 19:48 indexer-node1.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1708 Feb 8 19:48 indexer-node2-key.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1245 Feb 8 19:48 indexer-node2.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1704 Feb 8 19:48 indexer-node3-key.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1245 Feb 8 19:48 indexer-node3.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1704 Feb 8 19:48 manager-master-key.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1245 Feb 8 19:48 manager-master.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1704 Feb 8 19:48 manager-worker-key.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1245 Feb 8 19:48 manager-worker.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1704 Feb 8 19:48 root-ca.key
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 1204 Feb 8 19:48 root-ca.pem
-rw-r--r-- 1 wazuh-indexer wazuh-indexer 170 Feb 8 19:48 test.zip
root@ip-172-31-6-144:~#

The activity of starting the Wazuh indexer cluster continues, first verifying that each part is correctly generated (certificates, wazuh installation on all nodes, securityadmin)

vcerenu commented 1 year ago

The configuration of the distributed environment of Wazuh through Puppet is finished.

For the installation, the following manifest must be executed, adding the names of the Puppet agent nodes to each one and adding the necessary IPs.

$node1host   = '172.31.2.205'
$node2host   = '172.31.12.65'
$node3host   = '172.31.6.144'
$masterhost    = '172.31.2.205'
$workerhost    = '172.31.12.65'
$dashboardhost = '172.31.6.144'
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'
$cluster_size = '3'
# Define stage for order execution
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}

node "ip-172-31-18-215.us-west-1.compute.internal" {
 class { 'wazuh::certificates_dist': 
  stage => certificates,
 }
}
node "ip-172-31-2-205.us-west-1.compute.internal" {
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  indexer_node_name => "$indexer_node1_name",
  indexer_network_host => "$node1host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  stage => indexerdeploy,
 }
 class { 'wazuh::securityadmin':
 stage => securityadmin,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-master',
  ossec_cluster_node_type => 'master',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  manage_repos => false,
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
 class { 'wazuh::filebeat_oss_dist':
  filebeat_oss_indexer_ip => "$node1host",
  stage => manager,
 }
}
node "ip-172-31-12-65.us-west-1.compute.internal" {
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  indexer_node_name => "$indexer_node2_name",
  indexer_network_host => "$node2host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  stage => indexerdeploy,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-worker',
  ossec_cluster_node_type => 'worker',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  manage_repos => false,
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
}
node "ip-172-31-6-144.us-west-1.compute.internal" {
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  indexer_node_name => "$indexer_node3_name",
  indexer_network_host => "$node3host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  stage => indexerdeploy,
 }
 class { 'wazuh::dashboard_dist':
  indexer_server_ip  => "$node1host",
  manager_api_host   => "$masterhost",
  stage => dashboard,
 }
}

The manifest has several stages, in which the certificates are created in the Puppet server, they are moved to the directory of one of the modules, which will serve to later move the certificates to the necessary agents, then proceed to install Wazuh indexer on each node, securityadmin is run and then Wazuh manager (master and worker) and Wazuh dashboard are installed.

vcerenu commented 1 year ago

I did a test installation with the classes for distributed installation and found several problems:

It was necessary to declare each of the nodes and they were fixed, so I opted to modify the code of the indexer_dist.pp class so that it does not configure any parameters of the opensearch.yml template except the default values and that they are assigned in the installation manifest

The config.yml file for the creation of the certificates has fixed values assigned and I am investigating how to be able to assign these values of each node by means of an array, which should have a pair of values for each item (node name, ip) So I continue with the investigation to review how to assign these values to the creation template of the config.yml file.

Modifying the indexer_dist.yml manifest allowed me to deploy an AIO environment, but it requires a different configuration of the install manifest than what we currently have:

$discovery_type = 'single'
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}
node "ip-172-31-35-35.us-east-2.compute.internal" {
 class { 'wazuh::certificates_dist': 
  stage => certificates,
 }
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  stage => indexerdeploy,
 }
 class { 'wazuh::manager':
  manage_repos => false,
  stage => manager,
 }
 class { 'wazuh::filebeat_oss_dist':
  stage => manager,
 }
 class { 'wazuh::dashboard_dist':
  stage => dashboard,
 }
}
vcerenu commented 1 year ago

The configuration template of the config.yml file for the creation of the certificates was modified to add different servers according to an array of names and IPs, in addition to dividing the creation of certificates for Wazuh manager master and workers, being able to add the number of nodes that are necessary and not having a fixed template without the possibility of scaling:

nodes:
  indexer:
<% @indexer_certs.each do |node| -%>
      - name: indexer-<%= node[0] %>
        ip: <%= node[1] %>
<% end -%>
  server:
<% if @manager_certs -%>
<% @manager_certs.each do |node| -%>
      - name: manager-<%= node[0] %>
        ip: <%= node[1] %>
<% end -%>
<% end -%>
<% if @manager_master_certs -%>
<% @manager_master_certs.each do |node| -%>
      - name: manager-<%= node[0] %>
        ip: <%= node[1] %>
        node_type: master
<% end -%>
<% @manager_worker_certs.each do |node| -%>
      - name: manager-<%= node[0] %>
        ip: <%= node[1] %>
        node_type: worker
<% end -%>
<% end -%>
  dashboard:
<% @dashboard_certs.each do |node| -%>
      - name: dashboard
        ip: <%= node %>
<% end -%>

In addition, the manifests for the deployment were modified:

Multi node:

$node1host   = '172.31.33.220'
$node2host   = '172.31.37.227'
$node3host   = '172.31.36.228'
$masterhost    = '172.31.33.220'
$workerhost    = '172.31.37.227'
$dashboardhost = '172.31.36.228'
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'
$cluster_size = '3'
$indexer_discovery_hosts = [$node1host, $node2host, $node3host]
$indexer_cluster_initial_master_nodes = [$node1host, $node2host, $node3host]
$indexer_cluster_CN = [$indexer_node1_name, $indexer_node2_name, $indexer_node3_name]
# Define stage for order execution
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}
node "ip-172-31-34-223.us-east-2.compute.internal" { #Puppet server
 class { 'wazuh::certificates_dist': 
  indexer_certs => [["$indexer_node1_name","$node1host" ],["$indexer_node2_name","$node2host" ],["$indexer_node3_name","$node3host" ]],
  manager_master_certs => [['master',"$masterhost"]],
  manager_worker_certs => [['worker',"$workerhost"]],
  dashboard_certs => ["$dashboardhost"],
  stage => certificates,
 }
}
node "ip-172-31-33-220.us-east-2.compute.internal" { #node1 with Wazuh manager master
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  indexer_node_name => "$indexer_node1_name",
  indexer_network_host => "$node1host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::securityadmin':
 stage => securityadmin,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-master',
  ossec_cluster_node_type => 'master',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  manage_repos => false,
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
 class { 'wazuh::filebeat_oss_dist':
  filebeat_oss_indexer_ip => "$node1host",
  stage => manager,
 }
}
node "ip-172-31-37-227.us-east-2.compute.internal" { # node2 with Wazuh manager worker
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  indexer_node_name => "$indexer_node2_name",
  indexer_network_host => "$node2host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-worker',
  ossec_cluster_node_type => 'worker',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  manage_repos => false,
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
}
node "ip-172-31-36-228.us-east-2.compute.internal" { # node 3 with Wazuh dashboard
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  indexer_node_name => "$indexer_node3_name",
  indexer_network_host => "$node3host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::dashboard_dist':
  indexer_server_ip  => "$node1host",
  manager_api_host   => "$masterhost",
  stage => dashboard,
 }
}

Single node:

$discovery_type = 'single'
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}
node "ip-172-31-34-223.us-east-2.compute.internal" {
 class { 'wazuh::certificates_dist': 
  indexer_certs => [['node-1','127.0.0.1']],
  manager_certs => [['master','127.0.0.1']],
  dashboard_certs => ['127.0.0.1'],
  stage => certificates,
 }
}
node "ip-172-31-35-35.us-east-2.compute.internal" {
 class { 'wazuh::repo_use':
 stage => repo,
 }
 class { 'wazuh::indexer_dist':
  stage => indexerdeploy,
 }
 class { 'wazuh::manager':
  manage_repos => false,
  stage => manager,
 }
 class { 'wazuh::filebeat_oss_dist':
  stage => manager,
 }
 class { 'wazuh::dashboard_dist':
  stage => dashboard,
 }
}
vcerenu commented 1 year ago

The newly created classes were compared and replaced with the first ones, taking into account that no previous functionality is lost

The main changes that were made are the following:

Modified the installation manifests with the final class names and tested both without errors:

Multi-node:

$node1host   = '172.31.33.220'
$node2host   = '172.31.37.227'
$node3host   = '172.31.36.228'
$masterhost    = '172.31.33.220'
$workerhost    = '172.31.37.227'
$dashboardhost = '172.31.36.228'
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'
$cluster_size = '3'
$indexer_discovery_hosts = [$node1host, $node2host, $node3host]
$indexer_cluster_initial_master_nodes = [$node1host, $node2host, $node3host]
$indexer_cluster_CN = [$indexer_node1_name, $indexer_node2_name, $indexer_node3_name]
# Define stage for order execution
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}
node "ip-172-31-34-223.us-east-2.compute.internal" { #Puppet server
 class { 'wazuh::certificates': 
  indexer_certs => [["$indexer_node1_name","$node1host" ],["$indexer_node2_name","$node2host" ],["$indexer_node3_name","$node3host" ]],
  manager_master_certs => [['master',"$masterhost"]],
  manager_worker_certs => [['worker',"$workerhost"]],
  dashboard_certs => ["$dashboardhost"],
  stage => certificates,
 }
}
node "ip-172-31-33-220.us-east-2.compute.internal" { #node1 with Wazuh manager master
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node1_name",
  indexer_network_host => "$node1host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::securityadmin':
 stage => securityadmin,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-master',
  ossec_cluster_node_type => 'master',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
 class { 'wazuh::filebeat_oss':
  filebeat_oss_indexer_ip => "$node1host",
  stage => manager,
 }
}
node "ip-172-31-37-227.us-east-2.compute.internal" { # node2 with Wazuh manager worker
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node2_name",
  indexer_network_host => "$node2host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-worker',
  ossec_cluster_node_type => 'worker',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
}
node "ip-172-31-36-228.us-east-2.compute.internal" { # node 3 with Wazuh dashboard
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node3_name",
  indexer_network_host => "$node3host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::dashboard':
  indexer_server_ip  => "$node1host",
  manager_api_host   => "$masterhost",
  stage => dashboard,
 }
}

Single node:

$discovery_type = 'single-node'
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard]
Exec {
 timeout => 0,
}
node "ip-172-31-34-223.us-east-2.compute.internal" {
 class { 'wazuh::certificates': 
  indexer_certs => [['node-1','127.0.0.1']],
  manager_certs => [['master','127.0.0.1']],
  dashboard_certs => ['127.0.0.1'],
  stage => certificates,
 }
}
node "ip-172-31-35-35.us-east-2.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  stage => indexerdeploy,
 }
 class { 'wazuh::manager':
  stage => manager,
 }
 class { 'wazuh::filebeat_oss':
  stage => manager,
 }
 class { 'wazuh::dashboard':
  stage => dashboard,
 }
}

The changes in the documentation for the use of the new installation modes and revision of the implementation date continue.

vcerenu commented 1 year ago

Modified the deployment manifest to align with the distributed Wazuh deployment, deploying each Wazuh indexernode to a different server and Wazuh manager master, worker, and Wazuh dashboard nodes to other servers.

A successful deployment test was performed. The deployment was done with version 4.4.0 of Wazuh.

The deployment of Wazuh agent was also added within the Wazuh indexer, Wazuh dashboard and Puppet server servers, to have all the nodes involved in monitoring.

image

After verifying the correct deployment of all Wazuh components, we proceeded to bump the branch version and apply the upgrade to 4.4.0. The upgrade of all the components worked, but within Wazuh dashboard it added a new index .kibana_2 which it was necessary to delete by hand for Wazuh dashboard to start correctly, this error is pending review.

After that executed worksround, it started correctly and all the updated agents were shown

image

image

I add the updated manifest with all the changes and the addition of the Wazuh agent deployment in the corresponding nodes:

$node1host   = '172.31.21.111'
$node2host   = '172.31.18.61'
$node3host   = '172.31.30.54'
$masterhost    = '172.31.18.242'
$workerhost    = '172.31.20.247'
$dashboardhost = '172.31.23.39'
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'
$cluster_size = '3'
$indexer_discovery_hosts = [$node1host, $node2host, $node3host]
$indexer_cluster_initial_master_nodes = [$node1host, $node2host, $node3host]
$indexer_cluster_CN = [$indexer_node1_name, $indexer_node2_name, $indexer_node3_name]
# Define stage for order execution
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
stage { 'agent': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard] -> Stage[agent]
Exec {
 timeout => 0,
}
node "puppet-server.us-east-2.compute.internal" {
 class { 'wazuh::certificates': 
  indexer_certs => [["$indexer_node1_name","$node1host" ],["$indexer_node2_name","$node2host" ],["$indexer_node3_name","$node3host" ]],
  manager_master_certs => [['master',"$masterhost"]],
  manager_worker_certs => [['worker',"$workerhost"]],
  dashboard_certs => ["$dashboardhost"],
  stage => certificates,
 }
 class { 'wazuh::repo':
 stage => repo,
 }
 class { "wazuh::agent":
    wazuh_register_endpoint => "$masterhost",
    wazuh_reporting_endpoint => "$masterhost"
 }
}
node "puppet-wazuh-indexer-node1.us-east-2.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node1_name",
  indexer_network_host => "$node1host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::securityadmin':
 stage => securityadmin,
 }
 class { "wazuh::agent":
    wazuh_register_endpoint => "$masterhost",
    wazuh_reporting_endpoint => "$masterhost"
 }
}
node "puppet-wazuh-indexer-node2.us-east-2.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node2_name",
  indexer_network_host => "$node2host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { "wazuh::agent":
    wazuh_register_endpoint => "$masterhost",
    wazuh_reporting_endpoint => "$masterhost"
 }
}
node "puppet-wazuh-indexer-node3.us-east-2.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node3_name",
  indexer_network_host => "$node3host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { "wazuh::agent":
    wazuh_register_endpoint => "$masterhost",
    wazuh_reporting_endpoint => "$masterhost"
 }
}
 node "puppet-wazuh-manager-master.us-east-2.compute.internal" { 
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-master',
  ossec_cluster_node_type => 'master',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
 class { 'wazuh::filebeat_oss':
  filebeat_oss_indexer_ip => "$node1host",
  stage => manager,
 }
}
 node "puppet-wazuh-manager-worker.us-east-2.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-worker',
  ossec_cluster_node_type => 'worker',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
}
node "puppet-wazuh-dashboard.us-east-2.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::dashboard':
  indexer_server_ip  => "$node1host",
  manager_api_host   => "$masterhost",
  stage => dashboard,
 }
 class { "wazuh::agent":
    wazuh_register_endpoint => "$masterhost",
    wazuh_reporting_endpoint => "$masterhost"
 }
}
vcerenu commented 1 year ago

Made all documentation changes in the https://github.com/wazuh/wazuh-documentation/tree/puppet-distributed-deploy branch and created the PR https://github.com/wazuh/wazuh-documentation/ pull/6186 for approval.

vcerenu commented 1 year ago

The commits of this change were taken to merge to the 4.8.0 branch, a complete deployment was tested with 3 Wazuh indexer nodes, 1 Wazuh dashboard, 1 Wazuh manager master and 1 Wazuh manager worker and the deployment was carried out successfully.

$node1host   = '172.31.23.191'
$node2host   = '172.31.28.121'
$node3host   = '172.31.24.156'
$masterhost    = '172.31.28.245'
$workerhost    = '172.31.28.97'
$dashboardhost = '172.31.31.95'
$indexer_node1_name = 'node1'
$indexer_node2_name = 'node2'
$indexer_node3_name = 'node3'
$cluster_size = '3'
$indexer_discovery_hosts = [$node1host, $node2host, $node3host]
$indexer_cluster_initial_master_nodes = [$node1host, $node2host, $node3host]
$indexer_cluster_CN = [$indexer_node1_name, $indexer_node2_name, $indexer_node3_name]
# Define stage for order execution
stage { 'certificates': }
stage { 'repo': }
stage { 'indexerdeploy': }
stage { 'securityadmin': }
stage { 'dashboard': }
stage { 'manager': }
stage { 'agent': }
Stage[certificates] -> Stage[repo] -> Stage[indexerdeploy] -> Stage[securityadmin] -> Stage[manager] -> Stage[dashboard] -> Stage[agent]
Exec {
 timeout => 0,
}
node "ip-172-31-26-67.us-west-1.compute.internal" {
 class { 'wazuh::certificates': 
  indexer_certs => [["$indexer_node1_name","$node1host" ],["$indexer_node2_name","$node2host" ],["$indexer_node3_name","$node3host" ]],
  manager_master_certs => [['master',"$masterhost"]],
  manager_worker_certs => [['worker',"$workerhost"]],
  dashboard_certs => ["$dashboardhost"],
  stage => certificates,
 }
 class { 'wazuh::repo':
 stage => repo,
 }
}
node "ip-172-31-23-191.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node1_name",
  indexer_network_host => "$node1host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
 class { 'wazuh::securityadmin':
 stage => securityadmin,
 }
}
node "ip-172-31-28-121.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node2_name",
  indexer_network_host => "$node2host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
}
node "ip-172-31-24-156.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::indexer':
  indexer_node_name => "$indexer_node3_name",
  indexer_network_host => "$node3host",
  indexer_node_max_local_storage_nodes => "$cluster_size",
  indexer_discovery_hosts => $indexer_discovery_hosts,
  indexer_cluster_initial_master_nodes => $indexer_cluster_initial_master_nodes,
  indexer_cluster_CN => $indexer_cluster_CN,
  stage => indexerdeploy,
 }
}
 node "ip-172-31-28-245.us-west-1.compute.internal" { 
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-master',
  ossec_cluster_node_type => 'master',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
}
 node "ip-172-31-28-97.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::manager':
  ossec_cluster_name => 'wazuh-cluster',
  ossec_cluster_node_name => 'wazuh-worker',
  ossec_cluster_node_type => 'worker',
  ossec_cluster_key => '01234567890123456789012345678912',
  ossec_cluster_bind_addr => "$masterhost",
  ossec_cluster_nodes => ["$masterhost"],
  ossec_cluster_disabled => 'no',
  stage => manager,
 }
}
node "ip-172-31-31-95.us-west-1.compute.internal" {
 class { 'wazuh::repo':
 stage => repo,
 }
 class { 'wazuh::dashboard':
  indexer_server_ip  => "$node1host",
  manager_api_host   => "$masterhost",
  stage => dashboard,
 }
}

In addition, the documentation was modified so that the distributed example does not try to install Wazuh agent on the nodes and the commits were moved to merge to the 4.8.0 branch of wazuh-documentation.