4.3.0 Splunk installation steps doesn't finish with a successfully Splunk-Wazuh environment

[alberpilot]

After following the guide we noticed that something is not working on the guide. The following issue comment https://github.com/wazuh/wazuh/issues/12913#issuecomment-1084742999 shows the problems encountered:

• The Splunk indexer installation component should be placed first and then, the Manager and forwarder.
• Not always is possible to download the Splunk software from the provided link. Maybe this is not our concern, but it happened.
• Installing and configuring Splunk forwarder finished with a port overlapping with indexer.
• No alerts were possible to see after the installation.

[rauldpm]

Update report

• Researching the issue and getting familiar with splunk

[xrisbarney]

Documentation update

1. The Splunk documentation has been updated to use the format that works V3.0 here.
2. I believe the failure to download Splunk may be as a result of country embargoes. (Do we need to mention this in the documentation?)
3. See wazuh alerts: Trying to replicate this issue. I will provide update.

[xrisbarney]

Update on issue 6. See wazuh alerts:

It looks like splunk is unable to retrieve the alerts when port 9997 is unreachable, occupied or has not been opened.

I am adding an instruction line to advise users to switch to an alternative port for forwarding.

[xrisbarney]

Update on issue 6. See wazuh alerts:

Included a troubleshooting section to ask users to check if port 9997 is open and listening, or switch to a new port from the forwarding and receiving section of Splunk.

[rauldpm]

Update report - 8.1.4 single node (debian)

• As @xrisbarney mentions, the port 9997 (used by defaults) is not opened (tested two times), had to open a new port and restart the services after adding the new port in Forwarding and receiving
• Restarting the splunkforwarder splunk binary, a port error show (need to change it):

ERROR: mgmt port [8089] - port is already bound. Splunk needs to use this port.

• Still getting Unable to retrieve results. It may be due to a connection problem with the Splunk forwarder, please try restarting this service.. Testing 8.2.2 to check if is a problem of 8.1.4 version
• Same behavior in 8.2.2, investigating the 8089 port error, i see that the splunkforwarded tries to use the same port as the splunk package

root@ubuntu20:/vagrant# netstat -an | grep 8089
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN   
root@ubuntu20:/vagrant# fuser 8089/tcp
8089/tcp:            41817
root@ubuntu20:/vagrant# ls -l /proc/41817/exe 
lrwxrwxrwx 1 root root 0 Apr  7 18:22 /proc/41817/exe -> /opt/splunk/bin/splunkd

• Testing different packages and versions, investigating demo splunk tasks, the --auto-ports options when starting the forwarder splunk package, assign the 8090 port, the same port changed manually, if splunk its restarted before the forwarded splunk (not forwarded and then splunk), alerts are received
• I have found that the change of the indexer port (troubleshooting) is mandatory in all test i have done. We need to change the default 9997 port since the beginning to avoid this, i also found that this troubleshouting affects only to splunk forwarding outputs.conf but the splunk outputs.conf still has the 9997 port, changing the default port seem show logs in the forwarded splunk and splunk splunkd.log files messages like:

AutoLoadBalancedConnectionStrategy [46723 TcpOutEloop] - Found currently active indexer. Connected to idx=192.168.57.118:12000, reuse=1.

Before config change

After config change

[xrisbarney]

Docu update

I have found that the change of the indexer port (troubleshooting) is mandatory in all test i have done. We need to change the default 9997 port since the beginning to avoid this.

I have moved the troubleshooting section to the install and configure splunk indexer instructions as items 4-8. This way, once Splunk is configured before the forwarder is configured with the indexer port, we are already using an unused port and there should be no conflicts in outputs.conf.

[rauldpm]

Update report

• Following the new modified guide in single instance, I still get the message:

` Unable to retrieve results. It may be due to a connection problem with the Splunk forwarder, please try restarting this service `

• I see that with this guide, the /opt/splunkforwarder/etc/system/local/outputs.conf is empty when it should contain data
• Adding a new forwarded server, the file is updated with the new server/port indicated without the need to restart the service, this means that the first add forward-server command does not work as it should

[rauldpm]

Update report

• After doing research and talking with Christian, we have seen that the guide is designed to build the environment in two nodes (one with splunk enterprise and the other with splunk forwarder), the titles lead to confusion, making us believe that the entire deployment is done on the same node when it is not, it works for deployment on two nodes as Christian has been able to verify
• For the AIO style deployment, some extra steps must be added, I have been doing tests and the basic and minimum configuration to have this type of deployment functional would be following these steps:

AIO deployment for centos (should work using the apt simils)

``` 1. yum install curl -y 2. rpm --import https://packages-dev.wazuh.com/key/GPG-KEY-WAZUH 3. cat > /etc/yum.repos.d/wazuh.repo << EOF [wazuh] gpgcheck=1 gpgkey=https://packages-dev.wazuh.com/key/GPG-KEY-WAZUH enabled=1 name=EL-\$releasever - Wazuh baseurl=https://packages-dev.wazuh.com/pre-release/yum/ protect=1 EOF 4. yum install -y wazuh-manager 5. systemctl start wazuh-manager 6. yum install splunk-8.2.2.rpm -y 7. curl -so /opt/splunk/etc/system/local/indexes.conf https://raw.githubusercontent.com/wazuh/wazuh-splunk/v4.3.0-8.2.2/setup/indexer/indexes.conf 8. curl -so /opt/splunk/etc/system/local/inputs.conf https://raw.githubusercontent.com/wazuh/wazuh-splunk/v4.3.0-8.2.2/setup/indexer/inputs.conf 9. /opt/splunk/bin/splunk start (in AIO deplyment its seems mandatory change the default 9997 port) 10. add new port in UI -> eg: 1025 ```  ``` 11. /opt/splunk/bin/splunk login 12. /opt/splunk/bin/splunk restart 13. yum install splunkforwarder-8.2.2.rpm -y 14. curl -so /opt/splunkforwarder/etc/system/local/props.conf https://raw.githubusercontent.com/wazuh/wazuh-splunk/v4.3.0-8.2.2/setup/forwarder/props.conf 15. curl -so /opt/splunkforwarder/etc/system/local/inputs.conf https://raw.githubusercontent.com/wazuh/wazuh-splunk/v4.3.0-8.2.2/setup/forwarder/inputs.conf 16. sed -i "s:MANAGER_HOSTNAME:$(hostname):g" /opt/splunkforwarder/etc/system/local/inputs.conf 17. /opt/splunkforwarder/bin/splunk start 18. /opt/splunkforwarder/bin/splunk add forward-server 192.168.57.102:1025 19. /opt/splunkforwarder/bin/splunk restart 20. curl -o SplunkAppForWazuh.tar.gz https://packages-dev.wazuh.com/pre-release/ui/splunk/wazuh_splunk-4.3.0_8.2.2-1.tar.gz 21. /opt/splunk/bin/splunk install app SplunkAppForWazuh.tar.gz 22. /opt/splunk/bin/splunk restart 23. /opt/splunkforwarder/bin/splunk restart 24. Configure API (sing in UI) (Go wazuh plugin) --> username and passwords seems to be already saved with the splunk values (password must be changed to wazuh. Maybe related to splunk login) (save api configuration) (access overview -> security events) (The message Unable to retrieve results. It may be due to a connection problem with the Splunk forwarder, please try restarting this service will pop up, but we should get events) ``` 

• Since the current documentation indicates this type of deployment, we must add the AIO type deployment to the guide with these steps, note that the main difference is the use of splunk login, the initial port change and the multiple necessary restarts

• Generally, from what I have seen, this type of deployment is not recommended in the splunk forums due to performance issues, but for testing they should be useful. Also, although the documentation indicates that this deployment is possible, there are users who indicate that certain commands have problems and step on each other, such as boot start

• https://community.splunk.com/t5/Getting-Data-In/Running-a-Universal-Forwarder-on-the-same-server-as-the/m-p/127676/highlight/true#M26246

• https://community.splunk.com/t5/Getting-Data-In/Install-both-Universal-Forwarder-and-Splunk-Enterprise-on-on/m-p/211042/highlight/true#M41547

• Looking in splunk enterprise log, i can see warnings when entering in events

04-08-2022 22:40:31.155 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.156 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.157 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.157 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.159 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.533 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.535 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.536 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.538 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.540 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.551 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.553 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.555 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.557 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.567 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"
04-08-2022 22:40:31.569 +0000 WARN  SearchResultsFiles - Unable to parse site_label, label=invalid due to err="Invalid site id: invalid"

[rauldpm]

TODO

• [x] Tests versions
  • [x] 8.1.4 -> Receive alerts AIO
  • [x] 8.2.2 -> Receive alerts AIO
  • [x] 8.2.4 -> Receive alerts AIO
• [ ] Test API Alias - wazuh-splunk/issues/1145
• [ ] RBAC

[xrisbarney]

Update

TODO

Guides

• [x] Add a prerequisites section stating there should be 2 servers for the single instance splunk installation.
• [x] Add AIO guide.

[xrisbarney]

Update

@rauldpm's Steps worked for the all-in-one architecture.

We now have 3 methods of installing Wazuh with Splunk:

• All-in-one server architecture
• Single-instance multi-tier server architecture. A pre-requisites section was added stating there should be 2 servers.
• Multi-instance cluster architecture.

[okynos]

Hello,

Following the documentation guide I have to remark some points that sounds bad for me:

• The inputs.conf file provided in the documentation doesn't work for me. I have to modified like this:

  [splunktcp://9997]
  disabled = 0
  connection_host = ip

• The path described in documentation doesn't let me provide the environment like the path followed in the Indexer/dashboard version (documentation problem).
• Some commands are different from Indexer/dashboard section. Lacks curl to perform the next step.

Considerations: I have set up two nodes:

1. Splunk indexer
2. Wazuh manager + Splunk forwarder. The forwarder needs specific configuration.

[okynos]

• [x] Test API Alias - WORK AS EXPECTED
• [x] RBAC Test -> https://github.com/wazuh/wazuh-documentation/issues/5057 RBAC seems to not work but we will need more research.

[okynos]

RBAC successfully configured and tested.

[xrisbarney]

Update

TODO

Guides

• [x] Add RBAC section to the guide. RBAC section has been added here.

[rauldpm]

Update report

• Tested again AIO, some steps can be removed
• I keep saying that we should stop using the term single-instance as it can confuse the user with AIO
• In the wazuh cluster deployment, we could add a cluster health check
• I cannot make a single-instance (two nodes - manager+forwarder and enterprise+app) works. API does not show dialogs to connect the api. Steps:
  • Install wazuh manager on node1
  • Install splunk enterprise on node2
  • Install splunk forwarder on node1
  • Install wazuh app on node2
• We should redefine the deployment methods that we are giving to the user:
  • AIO
  • minimal splunk distributed architecture (wazuh cluster or single manager?) The image suggest (wazuh single node with forwarder (node1)) and (splunk enterprise with wazuh app (node2))
  • distributed architecture (wazuh cluster with forwarders, indexer cluster, heads cluster, desployer)