Splunk doesn't appear to be configured by default to listen on port 9997 - has nothing set.

[aglerj]

• Operating System Version: mine - win10 pro/ detectionlab - Logger system.
• Provider (VirtualBox/VMWare): VirtualBox
• Vagrant Version: 2.2.3
• Packer Version: n/a
• Is the issue reproducible or intermittent? Reproducable.

Description of the issue:

There's no configuration already in Splunk for receiving on port 9997 as the script installs/configures all the splunk forwarders to send to on 192.168.38.105.

Noticed no data/logs coming into Splunk after starting up... Forwarding and receiving » Receive data » Add new Encountered the following error while trying to save: Parameter name: TCP port 9997 is not available. Parameter name: TCP port 9997 is not available.

[aglerj]

I created one for 9998, however all of the forwarders are installed/set to use 9997.

[aglerj]

ignore - errors related to port allocation.

[aglerj]

it happens because the splunk UF allocates 8089, then when the splunk server install kicks off, it's already taken... causing the rest of the splunk install to fail. I'm still trying to find the right spot to either change or disable the local logger UF mgt port.

[clong]

https://github.com/clong/DetectionLab/blob/ef566fc5efcc69fb39e66a2974af15b0102e0f22/Vagrant/bootstrap.sh#L88

[aglerj]

Thanks, the issue title is misleading now. I looked through the logs and basically both the splunk UF and the splunk server install want to use port 8089 for their management ports on the logger system. T makes the splunk install fail etc. I'm not sure how others haven't already seen that though? I've recreated the vms and am using the cloned repository. I'll try to copy/paste the details in a bit.

[clong]

Hmm, I'm not sure I understand but post the details and I'll take a look!

[aglerj]

See the example error msgs below

logger: md64.deb 99%[==================> ] 249.85M 22.9MB/s eta 0s
logger: splunk-7.2.1-be11b2 100%[===================>] 250.65M 23.2MB/s in 9.7s
logger: 2019-02-25 18:47:07 (25.8 MB/s) - ‘splunk-7.2.1-be11b2c46e23-linux-2.6-amd64.deb’ saved [262825000/262825000] logger: Selecting previously unselected package splunk. logger: (Reading database ... 136987 files and directories currently installed.) logger: Preparing to unpack splunk-7.2.1-be11b2c46e23-linux-2.6-amd64.deb ... logger: Unpacking splunk (7.2.1) ... logger: Setting up splunk (7.2.1) ... logger: complete logger: Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'. logger: Generating RSA private key, 2048 bit long modulus logger: ................................+++++ logger: ...+++++ logger: e is 65537 (0x10001) logger: writing RSA key logger: Generating RSA private key, 2048 bit long modulus logger: ..+++++ logger: .........................................................................................+++++ logger: e is 65537 (0x10001) logger: writing RSA key logger: Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modu les'. logger: logger: This appears to be your first time running this version of Splunk. logger: logger: Splunk> Australian for grep. logger: logger: Checking prerequisites... logger: Checking http port [8000]: logger: open logger: Checking mgmt port [8089]: logger: Would you like to change ports? [y/n]: logger: Exiting due to --no-prompt. logger: not available logger: ERROR: mgmt port [8089] - port is already bound. Splunk needs to use this port. logger: The server's splunkd port has been changed. logger: IndexerService is not initialized for a universal forwarder logger: IndexerService is not initialized for a universal forwarder logger: IndexerService is not initialized for a universal forwarder logger: IndexerService is not initialized for a universal forwarder logger: IndexerService is not initialized for a universal forwarder logger: IndexerService is not initialized for a universal forwarder logger: IndexerService is not initialized for a universal forwarder logger: IndexerService is not initialized for a universal forwarder logger: App '/vagrant/resources/splunk_forwarder/splunk-add-on-for-microsoft-windows_500.tgz' installed logger: App '/vagrant/resources/splunk_server/add-on-for-microsoft-sysmon_800.tgz' installed logger: App '/vagrant/resources/splunk_server/asn-lookup-generator_012.tgz' installed logger: App '/vagrant/resources/splunk_server/force-directed-app-for-splunk_200.tgz' installed logger: App '/vagrant/resources/splunk_server/punchcard-custom-visualization_130.tgz' installed logger: App '/vagrant/resources/splunk_server/sankey-diagram-custom-visualization_130.tgz' installed logger: App '/vagrant/resources/splunk_server/threathunting_11.tgz' installed logger: cp: cannot create regular file '/opt/splunk/etc/apps/ThreatHunting/local': No such file or directory logger: /tmp/vagrant-shell: line 94: /opt/splunk/etc/apps/search/local/inputs.conf: No such file or directory logger: cp: cannot create regular file '/opt/splunk/etc/apps/search/local/': Not a directory logger: cp: cannot create regular file '/opt/splunk/etc/apps/search/local/': Not a directory logger: splunkd is not running. logger: logger: Splunk> Australian for grep. logger: logger: Checking prerequisites... logger: Checking http port [8000]: logger: open logger: Checking mgmt port [8089]: logger: not available logger: ERROR: mgmt port [8089] - port is already bound. Splunk needs to use this port. logger: tcgetattr: Inappropriate ioctl for device logger: WARNING: error changing terminal modes - password will echo! logger: Would you like to change ports? [y/n]: tcgetattr: Inappropriate ioctl for device logger: WARNING: error changing terminal modes - password will echo! logger: Would you like to change ports? [y/n]: logger: Init script installed at /etc/init.d/splunk. logger: Init script is configured to run at boot. logger: FATAL: Unable to read the job status.

[aglerj]

Here is the part before it where it installs the splunk UF on the logger system and acquires port 8089.

logger: Saving to: ‘splunkforwarder-7.2.1-be11b2c46e23-linux-2.6-amd64.deb’ logger: logger: splunkfor 0%[ ] 0 --.-KB/s
logger: splunkforw 25%[====> ] 4.32M 21.6MB/s
logger: splunkforwa 50%[=========> ] 8.71M 20.4MB/s
logger: splunkforwar 66%[============> ] 11.49M 18.3MB/s
logger: splunkforward 99%[==================> ] 17.13M 20.6MB/s
logger: splunkforwarder-7.2 100%[===================>] 17.16M 20.6MB/s in 0.8s
logger: logger: 2019-02-25 18:42:21 (20.6 MB/s) - ‘splunkforwarder-7.2.1-be11b2c46e23-linux-2.6-amd64.deb’ saved [17993338/17 993338] logger: Selecting previously unselected package splunkforwarder. logger: (Reading database ... 134975 files and directories currently installed.) logger: Preparing to unpack splunkforwarder-7.2.1-be11b2c46e23-linux-2.6-amd64.deb ... logger: Unpacking splunkforwarder (7.2.1) ... logger: Setting up splunkforwarder (7.2.1) ... logger: complete logger: Created symlink from /etc/systemd/system/multi-user.target.wants/splunk.service to /lib/systemd/system/splunk.ser vice. logger: logger: This appears to be your first time running this version of Splunk. logger: logger: Splunk> Australian for grep. logger: logger: Checking prerequisites... logger: Checking mgmt port [8089]: logger: open logger: New certs have been generated in '/opt/splunkforwarder/etc/auth'. logger: Creating: /opt/splunkforwarder/var/lib/splunk logger: Creating: /opt/splunkforwarder/var/run/splunk logger: Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n logger: Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css logger: Creating: /opt/splunkforwarder/var/run/splunk/upload logger: Creating: /opt/splunkforwarder/var/spool/splunk logger: Creating: /opt/splunkforwarder/var/spool/dirmoncache logger: Creating: /opt/splunkforwarder/var/lib/splunk/authDb logger: Creating: /opt/splunkforwarder/var/lib/splunk/hashDb logger: Checking conf files for problems... logger: Done logger: Checking default conf files for edits... logger: Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.2.1-be11b2c46e23-linux-2. 6-x86_64-manifest' logger: All installed files intact. logger: Done logger: All preliminary checks passed. logger: logger: Starting splunk server daemon (splunkd)...
logger: Done logger: Cloning into '/opt/splunkforwarder/etc/apps/TA-bro_json'... logger: Added forwarding to: 192.168.38.105:9997. ==> logger: Running provisioner: reload... ==> logger: Attempting graceful shutdown of VM... ==> logger: Checking if box 'bento/ubuntu-16.04' version '201812.27.0' is up to date... ==> logger: Clearing any previously set forwarded ports... ==> logger: Clearing any previously set network interfaces... ==> logger: Preparing network interfaces based on configuration... logger: Adapter 1: nat logger: Adapter 2: hostonly ==> logger: Forwarding ports... logger: 22 (guest) => 2200 (host) (adapter 1) ==> logger: Running 'pre-boot' VM customizations... ==> logger: Booting VM...

[clong]

@aglerj I see the messages about the management port which is 8089, but I don't understand what that has to do with the input listening on 9997.

It also sounds like you're attempting to install a Splunk UF on logger which already has Splunk Enterprise installed on it, which would be redundant and explain the reason behind the port conflicts. You don't need a forwarder on a host that already has Splunk on it.

I looked through the most recent CI run and this is not an issue with the stock bootstrap.sh:

    logger: This appears to be your first time running this version of Splunk.
    logger: 
    logger: Splunk> Finding your faults, just like mom.
    logger: 
    logger: Checking prerequisites...
    logger:     Checking http port [8000]: 
    logger: open
    logger:     Checking mgmt port [8089]: 
    logger: open

Since it sounds like you have customized the bootstrap.sh by installing splunkforwarder-7.2.1-be11b2c46e23-linux-2.6-amd64.deb, you're on your own here for working out these issues.

[aglerj]

I cloned from git....so I'll try to figure it out myself/repost. I only tweaked the bootstrap.sh after receiving the errors to try to fix/understand why it was broken. I didn't custom add splunk UF to the logger server. Thanks for looking.

[aglerj]

Oops - cloned from dlee35/DetectionLab ... that's why.

Rebuilding now with clong version.