Closed marcellodesales closed 8 years ago
Looks like the config you received from Ops does not do what you thought it would do. See this snippet from your inputs.conf:
[monitor:///var/log/messages]
That stanza instructs a Splunk forwarder to tail a file on disk, not listen on a TCP/UDP port. As documented here, to listen on a UDP port, you'll need to do this:
[udp://514]
I suggest going back to your Ops folks to confirm the settings before baking them in.
To your questions:
If I understand what you are asking, yes you can mount a docker volume to effect configuration changes. Mount to the whole $SPLUNK_HOME/etc folder.
Troubleshooting steps. If you have access to Splunk, then you can search it for host=pe2enpmas300*
, and failing that, host=pe2enpmas300* index=_internal
. If the former sees nothing, you are not collecting data. If the latter sees nothing, then the forwarder is not forwarding data at all, not even the internal debug logs.
If you don't have Splunk access, fix that, because Splunk is cool. In the meantime, inspect the container's /var volume. /var/log/splunk/splunkd.log
maps to what you should be seeing in that _internal line above.
Note that our best practice is to not use Splunk as a syslog server, even though it works fine. For production, we suggest setting up a syslog server (could be in a container...), and have a Splunk Forwarder ingest the syslog files on disk, so going back to using a line like the monitor://...
stuff. This excellent blog post lays out the reasons why.
@marcellodesales at least one issue I see, please take a look on README.md
Splunk processes are running under splunk user.
We are using 1514 instead of standard 514 syslog port because ports below 1024 are reserved for root access only. See Run Splunk Enterprise as a different or non-root user.
I understand that you are running your docker daemon with root, but in the container I start splunk with splunk user. if you really want to use port 514 inside of the container - you should start splunk as root as well, for that you can specify environment variables SPLUNK_USER=root
, SPLUNK_GROUP=root
.
Also you have not enabled listening on port 514, see http://docs.splunk.com/Documentation/Splunk/latest/Data/Monitornetworkports
Another small issue - you do not need to specify SPLUNK_FORWARD_SERVER
as you specify them on your own in outputs.conf. SPLUNK_FORWARD_SERVER
exists only for people who does not have conf files and they want to configure it with this variable.
About the configuration files there are few options:
FROM outcoldman/splunk:forwarder-6.2
COPY inputs.conf $SPLUNK_HOME/etc/system/local/
COPY outputs.conf $SPLUNK_HOME/etc/system/loca/
.... and so on ...
$SPLUNK_HOME/etc/
, don't worry about default settings - they will be copied over on startup. Debugging
For debugging I would recommend to look inside of the splunkd.log
, see
docker exec tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log
@halr9000 and @outcoldman, thank you so much for all the response here! I will go through today and I will place comments about our findings...
@halr9000 [udp://514]
This one of the reasons why it might not have worked... So, let me try that...
@outcoldman SPLUNK_USER=root, SPLUNK_GROUP=root
I will be using the following on my ops-services-docker-compose.yml
splunkforwarder:
image: outcoldman/splunk:6.2.4-forwarder
restart: always
ports:
- 514:514/udp
environment:
- SPLUNK_USER=root
- SPLUNK_GROUP=root
Is that all I need? The processes are now run by root
.
[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@a59c0fdf35e9:/opt/splunk#
root@a59c0fdf35e9:/opt/splunk# ps aux | grep splunk
root 66 0.1 0.6 209580 99612 ? Sl 20:19 0:03 splunkd -p 8089 start
root 67 0.0 0.0 53640 7768 ? Ss 20:19 0:00 [splunkd pid=66] splunkd -p 8089 start [process-runner]
root 97 0.0 0.0 47580 1852 ? S 20:19 0:00 sudo -HEu root tail -f /opt/splunk/var/log/splunk/splunkd_stderr.log
root 98 0.0 0.0 5952 616 ? S 20:19 0:00 tail -f /opt/splunk/var/log/splunk/splunkd_stderr.log
root 140 0.0 0.0 10460 940 ? S+ 21:06 0:00 grep --color=auto splunk
I will monitor the splunk logs inside the container... I see that the container connected to the index servers:
$ docker exec -ti npmoserver_splunkforwarder_1 sh
[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@a59c0fdf35e9:/opt/splunk# tail -f $SPLUNK_HOME/var/log/splunk/splunkd.log
01-11-2016 20:19:32.399 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/splunkd_stdout.log'.
01-11-2016 20:19:32.400 +0000 INFO WatchedFile - Will begin reading at offset=183 for file='/opt/splunk/var/log/splunk/splunkd_stderr.log'.
01-11-2016 20:19:32.412 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
01-11-2016 20:19:32.432 +0000 INFO WatchedFile - Will begin reading at offset=955414 for file='/opt/splunk/var/log/splunk/metrics.log'.
I can see it connected to the hosts...
01-10-2016 14:56:43.229 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-10-2016 14:57:14.093 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
01-10-2016 14:57:43.309 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
However, I see a few errors around SSL connectivity. Do I need to have the CERTS? or can I disable it?
01-11-2016 20:19:31.787 +0000 INFO ServerConfig - My hostname is "a59c0fdf35e9".
01-11-2016 20:19:31.797 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-11-2016 20:19:31.797 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-11-2016 20:19:31.797 +0000 INFO ServerConfig - Setting HTTP server compression state=on
01-11-2016 20:19:31.797 +0000 INFO ServerConfig - Setting HTTP client compression state=0 (false)
01-11-2016 20:19:31.797 +0000 INFO ServerConfig - Default output queue for file-based input: parsingQueue.
01-11-2016 20:19:31.801 +0000 INFO LicenseMgr - Initing LicenseMgr
Thanks for any help!
Hi @halr9000 @outcoldman
Good news! The setup worked!!!
My question now is more related to the events... There's a mix of txt
and json
output... Is there a way to identify and differenciate among the events in syslog?
Thanks a lot
Nice! Should we close this issue now?
@outcoldman Actually I have to take it back... There was a splunkforwarder
instance running on the host... :'( Here's what I have in the containers so far...
Mounts the dir /var/log
containing the /var/log/messages
file. This HOST is writing the docker container logs to /var/log/messages
... This is volume-mounted in the docker container splunkforwarder
.
splunkforwarderData:
image: busybox
volumes:
- ./monitor/splunk:/opt/splunk/etc/system/local
- /var/log:/var/log
splunkforwarder:
image: outcoldman/splunk:6.2.4-forwarder
restart: always
environment:
- SPLUNK_USER=root
- SPLUNK_GROUP=root
volumes_from:
- "splunkforwarderData"
frontdoor:
build: roles/registry
restart: always
env_file: .env
ports:
- "8080:8080"
log_driver: "syslog"
log_opt:
syslog-tag: "frontdoor"
Can you confirm that this output below is good? I cannot see new messages going to splunk anymore :( Is there any other logs I should be looking at? I see in the bottom of the logs that the splunk forwarder is indeed connected to the 2 hosts...
root@f750b9a56059:/opt/splunk# ls -la /var/log/messages
-rw------- 1 root root 2842029 Jan 12 02:14 /var/log/messages
root@f750b9a56059:/opt/splunk# vim
bash: vim: command not found
root@f750b9a56059:/opt/splunk# vi $SPLUNK_HOME/var/log/splunk/splunkd.log
01-12-2016 02:14:18.906 +0000 INFO ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 02:14:18.907 +0000 INFO ServerConfig - My newly generated GUID is 0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.907 +0000 INFO ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 02:14:18.907 +0000 INFO ServerConfig - Found no site defined in server.conf
01-12-2016 02:14:18.907 +0000 INFO ServerConfig - My hostname is "f750b9a56059".
01-12-2016 02:14:18.917 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFi
nal_ex:bad decrypt.
01-12-2016 02:14:18.917 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-12-2016 02:14:18.917 +0000 INFO ServerConfig - Setting HTTP server compression state=on
01-12-2016 02:14:18.917 +0000 INFO ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 02:14:18.917 +0000 INFO ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 02:14:18.928 +0000 INFO LicenseMgr - Initing LicenseMgr
01-12-2016 02:14:18.928 +0000 INFO LMConfig - serverName=pe2enpmas300.corp.company.net guid=0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.929 +0000 INFO LMConfig - connection_timeout=30
01-12-2016 02:14:18.929 +0000 INFO LMConfig - send_timeout=30
01-12-2016 02:14:18.929 +0000 INFO LMConfig - receive_timeout=30
01-12-2016 02:14:18.929 +0000 INFO LMConfig - squash_threshold=2000
01-12-2016 02:14:18.929 +0000 INFO LMConfig - strict_pool_quota=1
01-12-2016 02:14:18.929 +0000 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 02:14:18.929 +0000 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 02:14:18.929 +0000 INFO LMStackMgr - closing stack mgr
01-12-2016 02:14:18.929 +0000 INFO LMSlaveInfo - all slaves cleared
01-12-2016 02:14:18.929 +0000 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 02:14:18.929 +0000 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 02:14:18.929 +0000 INFO ServerRoles - Declared role=license_master.
01-12-2016 02:14:18.929 +0000 INFO LMStackMgr - init completed [0E7035A5-7F3B-42ED-8735-7BEA3435BA74,Forwarder,runContext_splunkd=true]
01-12-2016 02:14:18.929 +0000 INFO LicenseMgr - StackMgr init complete...
01-12-2016 02:14:18.929 +0000 INFO LMTracker - init'ing slaveId=0E7035A5-7F3B-42ED-8735-7BEA3435BA74 label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 02:14:18.929 +0000 INFO LMTracker - enabling implicit feature set
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 02:14:18.929 +0000 INFO LMTracker - attempting to ping master=self from slave=0E7035A5-7F3B-42ED-8735-7BEA3435BA74
01-12-2016 02:14:18.930 +0000 INFO LMSlaveInfo - new slave='0E7035A5-7F3B-42ED-8735-7BEA3435BA74' created
01-12-2016 02:14:18.930 +0000 INFO LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Tue Jan 12 00:00:00 2016), meaning that the master has a
lready rolled over. Ignore slave persisted usage.
"/opt/splunk/var/log/splunk/splunkd.log" 167 lines, 19388 characters
01-12-2016 02:14:19.128 +0000 INFO IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (No such fi
le or directory). This is normal when splunk is first starting up.
01-12-2016 02:14:19.130 +0000 WARN DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 02:14:19.130 +0000 INFO IndexProcessor - running splunkd specific init
01-12-2016 02:14:19.131 +0000 INFO loader - Initializing from configuration
01-12-2016 02:14:19.132 +0000 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 02:14:19.134 +0000 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 02:14:19.134 +0000 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 02:14:19.282 +0000 INFO TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 02:14:19.282 +0000 INFO ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 02:14:19.287 +0000 INFO TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 02:14:19.287 +0000 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 02:14:19.287 +0000 INFO PipelineComponent - Launching the pipelines.
01-12-2016 02:14:19.288 +0000 INFO loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 02:14:19.288 +0000 INFO loader - Limiting REST HTTP server to 2646 threads
01-12-2016 02:14:19.293 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFi
nal_ex:bad decrypt.
01-12-2016 02:14:19.293 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 02:14:19.293 +0000 ERROR HTTPServer - SSL will not be enabled
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - TailWatcher initializing...
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 02:14:19.341 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 02:14:19.342 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 02:14:19.342 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 02:14:19.342 +0000 INFO TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 02:14:19.342 +0000 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 02:14:19.371 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
01-12-2016 02:14:49.131 +0000 WARN AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-12-2016 02:14:50.006 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
01-12-2016 02:15:50.067 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-12-2016 02:16:20.096 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
01-12-2016 02:17:19.158 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
@marcellodesales looks good for me. I see that you are monitoring /var/log/messages, also see that you could connect to the indexers. Try to less on var/log/messages and see that when you get new messages here you will get them in splunk as well.
I do not thing that it is good idea to mount /var/log into container in the same path, who knows what container can expect from this path (I do not run anything, which depends on it, but I cannot be 100% sure about that).
@outcoldman I changed the settings to the following...
splunkforwarderData:
image: busybox
volumes:
- ./monitor/splunk:/opt/splunk/etc/system/local
- /var/log/messages:/var/log/messages
That only loads the /var/log/messages... However, I still don't see anything in Splunk... The files under /opt/splunk/etc/system/local
are loaded, correct? Because we see the dir being watched...
From inside the container, I see the logs coming to the /var/log/messages if I tail it... However, nothing is showing up in Splunk, although it says connected... :( Is there anything else I should be looking at?
root@be6fe6af3549:/opt/splunk# tail -f /var/log/messages
Jan 11 19:30:01 pe2enpmas300 systemd: Starting user-0.slice.
Jan 11 19:30:01 pe2enpmas300 systemd: Started Session 1186 of user root.
Jan 11 19:30:01 pe2enpmas300 systemd: Starting Session 1186 of user root.
Jan 11 19:30:01 pe2enpmas300 systemd: Removed slice user-0.slice.
Jan 11 19:30:01 pe2enpmas300 systemd: Stopping user-0.slice.
Jan 11 19:30:01 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:01.996Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:04 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:03.999Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:04 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:04.559Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:03:30:04 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:05 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:05.921Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:03:30:05 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:06 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:06.000Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:08 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:08.003Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:09 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:09.592Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:03:30:09 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:10 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:10.008Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:10 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:10.874Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:03:30:10 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 11 19:30:12 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:12.011Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:14 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:14.012Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:14 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:14.588Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:03:30:14 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:15 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T03:30:15.965Z","hostname":"7207e35f9e32","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:03:30:15 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 11 19:30:16 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:16.015Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 11 19:30:18 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T03:30:18.018Z: 0 queued changes, 0 open requests, 0 retries pending
The docker-compose logs show that the container tries to create the certs and initializes...
postgres_1 | taching: configured logging reader does not support reading
splunkforwarder_1 |
splunkforwarder_1 | This appears to be your first time running this version of Splunk.
splunkforwarder_1 |
splunkforwarder_1 | Splunk> Now with more code!
splunkforwarder_1 |
splunkforwarder_1 | Checking prerequisites...
nginx_1 | taching: configured logging reader does not support reading
splunkforwarder_1 | Checking mgmt port [8089]: New certs have been generated in '/opt/splunk/etc/auth'.
elasticsearch_1 | taching: configured logging reader does not support reading
splunkforwarder_1 | Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1 | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1 | Generating a 1024 bit RSA private key
splunkforwarder_1 | ...............................++++++
splunkforwarder_1 | ........................................++++++
auth_1 | taching: configured logging reader does not support reading
splunkforwarder_1 | writing new private key to 'privKeySecure.pem'
redis_1 | taching: configured logging reader does not support reading
splunkforwarder_1 | -----
splunkforwarder_1 | Signature ok
splunkforwarder_1 | subject=/CN=be6fe6af3549/O=SplunkUser
splunkforwarder_1 | Getting CA Private Key
splunkforwarder_1 | writing RSA key
splunkforwarder_1 | open
splunkforwarder_1 | Creating: /opt/splunk/var/lib/splunk
splunkforwarder_1 | Creating: /opt/splunk/var/run/splunk
splunkforwarder_1 | Creating: /opt/splunk/var/run/splunk/appserver/i18n
splunkforwarder_1 | Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
splunkforwarder_1 | Creating: /opt/splunk/var/run/splunk/upload
splunkforwarder_1 | Creating: /opt/splunk/var/spool/splunk
splunkforwarder_1 | Creating: /opt/splunk/var/spool/dirmoncache
splunkforwarder_1 | Creating: /opt/splunk/var/lib/splunk/authDb
splunkforwarder_1 | Creating: /opt/splunk/var/lib/splunk/hashDb
splunkforwarder_1 | Checking conf files for problems...
splunkforwarder_1 | Done
splunkforwarder_1 | All preliminary checks passed.
splunkforwarder_1 |
splunkforwarder_1 | Starting splunk server daemon (splunkd)...
splunkforwarder_1 | Done
splunkforwarder_1 |
splunkforwarder_1 | 2016-01-12 03:18:38.574 +0000 splunkd started (build 271043)
Then, I can see a few errors in the container
root@be6fe6af3549:/opt/splunk# grep -R "ERROR" var/log/splunk/splunkd.log
01-12-2016 03:18:38.572 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 03:18:38.572 +0000 ERROR ServerConfig - Couldn't initialize SSL Context for HTTPClient in ServerConfig
01-12-2016 03:18:38.931 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 03:18:38.931 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 03:18:38.931 +0000 ERROR HTTPServer - SSL will not be enabled
Maybe this is the reason the container cannot forward the logs?
@marcellodesales sorry, I don't see anything unusual.
Just few recommendations:
@outcoldman
The only delta I found was the SSL
errors shown on the container... Here's the logs from the HOST without ANY ERROR...
[root@pe2enpmas300 npmo-server]# grep -R "ERROR" /opt/splunkforwarder/var/log/splunk/splunkd.log | wc -l
0
01-08-2016 20:01:27.070 -0800 INFO ServerConfig - Will generate GUID, as none found on this server.
01-08-2016 20:01:27.070 -0800 INFO ServerConfig - My newly generated GUID is 6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.071 -0800 INFO ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-08-2016 20:01:27.071 -0800 INFO ServerConfig - Found no site defined in server.conf
01-08-2016 20:01:27.071 -0800 INFO ServerConfig - My hostname is "pe2enpmas300.corp.company.net".
01-08-2016 20:01:27.080 -0800 INFO ServerConfig - Setting HTTP server compression state=on
01-08-2016 20:01:27.081 -0800 INFO ServerConfig - Setting HTTP client compression state=0 (false)
01-08-2016 20:01:27.081 -0800 INFO ServerConfig - Default output queue for file-based input: parsingQueue.
01-08-2016 20:01:27.084 -0800 INFO LicenseMgr - Initing LicenseMgr
01-08-2016 20:01:27.084 -0800 INFO LMConfig - serverName=pe2enpmas300.corp.company.net guid=6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.084 -0800 INFO LMConfig - connection_timeout=30
01-08-2016 20:01:27.084 -0800 INFO LMConfig - send_timeout=30
01-08-2016 20:01:27.084 -0800 INFO LMConfig - receive_timeout=30
01-08-2016 20:01:27.084 -0800 INFO LMConfig - squash_threshold=2000
01-08-2016 20:01:27.084 -0800 INFO LMConfig - strict_pool_quota=1
01-08-2016 20:01:27.084 -0800 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-08-2016 20:01:27.084 -0800 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-08-2016 20:01:27.084 -0800 INFO LMStackMgr - closing stack mgr
01-08-2016 20:01:27.084 -0800 INFO LMSlaveInfo - all slaves cleared
01-08-2016 20:01:27.140 -0800 INFO LMConfig - created default pool=auto_generated_pool_forwarder for stack=forwarder
01-08-2016 20:01:27.140 -0800 INFO LMStackMgr - added default pool=auto_generated_pool_forwarder for stack=forwarder
01-08-2016 20:01:27.142 -0800 INFO LMConfig - created default pool=auto_generated_pool_free for stack=free
01-08-2016 20:01:27.142 -0800 INFO LMStackMgr - added default pool=auto_generated_pool_free for stack=free
01-08-2016 20:01:27.142 -0800 INFO ServerRoles - Declared role=license_master.
01-08-2016 20:01:27.142 -0800 INFO LMStackMgr - init completed [6748789D-54FE-4389-B00D-F144248A0683,Forwarder,runContext_splunkd=true]
01-08-2016 20:01:27.142 -0800 INFO LicenseMgr - StackMgr init complete...
01-08-2016 20:01:27.142 -0800 INFO LMTracker - init'ing slaveId=6748789D-54FE-4389-B00D-F144248A0683 label=pe2enpmas300.corp.company.net [30,30,self]
01-08-2016 20:01:27.143 -0800 INFO LMTracker - enabling implicit feature set
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - attempting to ping master=self from slave=6748789D-54FE-4389-B00D-F144248A0683
01-08-2016 20:01:27.143 -0800 INFO LMSlaveInfo - new slave='6748789D-54FE-4389-B00D-F144248A0683' created
01-08-2016 20:01:27.143 -0800 INFO LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Fri Jan 8 00:00:00 2016), meaning that the master has a
lready rolled over. Ignore slave persisted usage.
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=AdvancedSearchCommands state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=AdvancedXML state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=CustomRoles state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=DeployClient state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=DeployServer state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=GuestPass state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=KVStore state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=LDAPAuth state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=LocalSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=MultisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=NontableLookups state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=RcvData state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=ResetWarnings state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=RollingWindowAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=ScheduledAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=ScheduledReports state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=ScheduledSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=SearchheadPooling state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=SigningProcessor state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=SplunkWeb state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=SyslogOutputProcessor state=ENABLED (featureStatus=1)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - Setting feature=UnisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-08-2016 20:01:27.143 -0800 INFO LMTracker - setting masterGuid='6748789D-54FE-4389-B00D-F144248A0683'
01-08-2016 20:01:27.143 -0800 INFO LMTracker - attempting to contact master=self from slave=6748789D-54FE-4389-B00D-F144248A0683 success
01-08-2016 20:01:27.143 -0800 INFO LicenseMgr - Tracker init complete...
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: virtual address space size: unlimited
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: data segment size: unlimited
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: resident memory size: unlimited
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: core file size: 0 bytes [hard maximum: unlimited]
01-08-2016 20:01:27.146 -0800 WARN ulimit - Core file generation disabled
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: data file size: unlimited
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: open files: 4096 files
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: user processes: 63441 processes
01-08-2016 20:01:27.146 -0800 INFO ulimit - Limit: cpu time: unlimited
01-08-2016 20:01:27.147 -0800 INFO ulimit - Linux transparent hugetables support, enabled="always" defrag="always"
01-08-2016 20:01:27.150 -0800 INFO loader - Splunkd starting (build 271043).
01-08-2016 20:01:27.150 -0800 INFO loader - System info: Linux, pe2enpmas300.corp.company.net, 3.10.0-327.3.1.el7.x86_64, #1 SMP Fri Nov 20 05:40:26 EST 2015, x86_64.
01-08-2016 20:01:27.150 -0800 INFO loader - Detected 4 (virtual) CPUs, 4 CPU cores, and 15880MB RAM
01-08-2016 20:01:27.150 -0800 INFO loader - Maximum number of threads (approximate): 7940
01-08-2016 20:01:27.150 -0800 INFO loader - Arguments are: "-p" "8089" "start"
01-08-2016 20:01:27.150 -0800 INFO loader - Getting configuration data from: /opt/splunkforwarder/etc/myinstall/splunkd.xml
01-08-2016 20:01:27.151 -0800 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunkforwarder/etc/modules
01-08-2016 20:01:27.151 -0800 INFO loader - loading modules from /opt/splunkforwarder/etc/modules
01-08-2016 20:01:27.152 -0800 INFO loader - Writing out composite configuration file: /opt/splunkforwarder/var/run/splunk/composite.xml
01-08-2016 20:01:27.156 -0800 INFO ServerRoles - Declared role=universal_forwarder.
01-08-2016 20:01:27.156 -0800 INFO BundlesSetup - Setup stats for /opt/splunkforwarder/etc: wallclock_elapsed_msec=6, cpu_time_used=0.004996, shared_services_generation=
1, shared_services_population=1
01-08-2016 20:01:27.178 -0800 INFO loader - Setting SSL configuration.
01-08-2016 20:01:27.178 -0800 INFO loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
01-08-2016 20:01:27.178 -0800 INFO loader - Using cipher suite TLSv1+HIGH:@STRENGTH
01-08-2016 20:01:27.178 -0800 INFO loader - ECDH curve not configured
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "WinEventLog://" with 30 parameters: start_from, current_only, checkpointInter
val, disabled, evt_resolve_ad_obj, evt_dc_name, evt_dns_name, index, whitelist, blacklist, whitelist1, whitelist2, whitelist3, whitelist4, whitelist5, whitelist6, whiteli
st7, whitelist8, whitelist9, blacklist1, blacklist2, blacklist3, blacklist4, blacklist5, blacklist6, blacklist7, blacklist8, blacklist9, suppress_text, renderXml
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "WinNetMon://" with 15 parameters: remoteAddress, process, user, addressFamily
, packetType, direction, protocol, readInterval, driverBufferSize, userBufferSize, mode, multikvMaxEventCount, multikvMaxTimeMs, disabled, index
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "WinRegMon://" with 7 parameters: proc, hive, type, baseline, baseline_interva
l, disabled, index
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "admon://" with 7 parameters: targetDc, startingNode, monitorSubtree, disabled
, index, printSchema, baseline
01-08-2016 20:01:27.356 -0800 INFO SpecFiles - Found external scheme definition for stanza "perfmon://" with 11 parameters: object, counters, instances, interval, mode,
samplingInterval, stats, disabled, index, showZeroValue, useEnglishOnly
01-08-2016 20:01:27.357 -0800 WARN ClusteringMgr - Ignoring clustering configuration, the active license disables this feature.
01-08-2016 20:01:27.358 -0800 INFO SHPoolingMgr - initing shpooling with: ht=60 rf=3 ct=60 st=60 rt=60 rct=5 rst=5 rrt=10 rmst=600 rmrt=600 pe=1 im=0 is=0 mor=5 pb=5 rep
_port= pptr=10
01-08-2016 20:01:27.358 -0800 INFO SHPoolingMgr - shpooling disabled
01-08-2016 20:01:27.358 -0800 INFO DS_DC_Common - Initializing the PubSub system.
01-08-2016 20:01:27.358 -0800 INFO DS_DC_Common - Initializing core facilities of PubSub system.
01-08-2016 20:01:27.366 -0800 INFO DC:DeploymentClient - target-broker clause is missing.
01-08-2016 20:01:27.366 -0800 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-08-2016 20:01:27.366 -0800 INFO DS_DC_Common - Deployment Client not initialized.
01-08-2016 20:01:27.366 -0800 INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-08-2016 20:01:27.367 -0800 INFO IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-08-2016 20:01:27.367 -0800 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-08-2016 20:01:27.367 -0800 INFO IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunkforwarder/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory). This is normal when splunk is first starting up.
01-08-2016 20:01:27.369 -0800 WARN DistributedPeerManager - feature=DistSearch not enabled for your license level
01-08-2016 20:01:27.369 -0800 INFO IndexProcessor - running splunkd specific init
01-08-2016 20:01:27.370 -0800 INFO loader - Initializing from configuration
01-08-2016 20:01:27.372 -0800 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-08-2016 20:01:27.373 -0800 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-08-2016 20:01:27.373 -0800 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
01-08-2016 20:01:27.496 -0800 INFO TcpOutputProc - Initializing with fwdtype=lwf
01-08-2016 20:01:27.496 -0800 INFO ServerRoles - Declared role=lightweight_forwarder.
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-08-2016 20:01:27.500 -0800 INFO TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-08-2016 20:01:27.501 -0800 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-08-2016 20:01:27.501 -0800 INFO PipelineComponent - Launching the pipelines.
01-08-2016 20:01:27.502 -0800 INFO loader - Limiting REST HTTP server to 1365 sockets
01-08-2016 20:01:27.502 -0800 INFO loader - Limiting REST HTTP server to 1365 threads
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - TailWatcher initializing...
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-08-2016 20:01:27.546 -0800 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-08-2016 20:01:27.547 -0800 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/etc/splunk.version.
01-08-2016 20:01:27.547 -0800 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/log/splunk.
01-08-2016 20:01:27.547 -0800 INFO TailingProcessor - Adding watch on path: /opt/splunkforwarder/var/spool/splunk.
01-08-2016 20:01:27.547 -0800 INFO TailingProcessor - Adding watch on path: /var/log/messages.
01-08-2016 20:01:27.547 -0800 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
01-08-2016 20:01:27.600 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
01-08-2016 20:01:57.371 -0800 WARN AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-08-2016 20:01:57.703 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:02:28.450 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
01-08-2016 20:02:57.479 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:03:27.509 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
01-08-2016 20:03:57.538 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:04:27.560 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
01-08-2016 20:04:57.576 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:05:27.593 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
01-08-2016 20:06:57.638 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-08-2016 20:07:27.652 -0800 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
HOST
: Works with the same configuration shared before... CONTAINER
: DOES NOT WORK so far... Is there any Debug statement I can turn on in the container?
Hi @outcoldman
I got a Docker image based on CENTOS that works like the following:
We have a cached version of the rpm stored at http://sds-repo-int.qdc.company.com:8081/nexus/content/repositories/CTO.OPS-releases/com/company/CTO/OPS-releases/splunkforwarder/6.2.4/splunkforwarder-6.2.4-271043-lin ux-2.6-x86_64.rpm
FROM richxsl/rhel7
MAINTAINER marcello.desales@gmail.com
ENV SPLUNK_HOME /opt/splunkforwarder
ENV SPLUNK_GROUP splunk
ENV SPLUNK_USER splunk
ENV SPLUNK_BACKUP_DEFAULT_ETC /var/opt/splunk
RUN groupadd -r ${SPLUNK_GROUP} \
&& useradd -r -m -g ${SPLUNK_GROUP} ${SPLUNK_USER}
COPY ./INTU-LATEST.repo /etc/yum.repos.d/
RUN yum remove -y subscription-manager
ENV LANG en_US.utf8
ADD http://sds-repo-int.qdc.company.com:8081/nexus/content/repositories/CTO.OPS-releases/com/company/CTO/OPS-releases/splunkforwarder/6.2.4/splunkforwarder-6.2.4-271043-lin
ux-2.6-x86_64.rpm .
RUN yum install -y splunkforwarder-6.2.4-271043-linux-2.6-x86_64.rpm && /opt/splunkforwarder/bin/splunk enable boot-start --accept-license --answer-yes && rm -f splunkfor
warder-6.2.4-271043-linux-2.6-x86_64.rpm
COPY entrypoint.sh /sbin/entrypoint.sh
RUN chmod +x /sbin/entrypoint.sh
WORKDIR /opt/splunkforwarder
# Configurations folder, var folder for everyting (indexes, logs, kvstore)
VOLUME [ "/opt/splunkforwarder/etc", "/opt/splunkforwarder/var" ]
ENTRYPOINT ["/sbin/entrypoint.sh"]
#!/bin/bash
set -e
${SPLUNK_HOME}/bin/splunk start --nodaemon --accept-license --answer-yes --no-prompt
Is there anything that this setup differs from yours? I noticed you do other things while setting up the server in entrypoint.sh
.
Here's the logs of the container itself:
splunkforwarder_1 |
splunkforwarder_1 | Splunk> Now with more code!
splunkforwarder_1 |
splunkforwarder_1 | Checking prerequisites...
splunkforwarder_1 | Checking mgmt port [8089]: New certs have been generated in '/opt/splunkforwarder/etc/auth'.
splunkforwarder_1 | Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1 | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1 | Generating a 1024 bit RSA private key
splunkforwarder_1 | .....++++++
splunkforwarder_1 | .......................++++++
splunkforwarder_1 | writing new private key to 'privKeySecure.pem'
splunkforwarder_1 | -----
splunkforwarder_1 | Signature ok
splunkforwarder_1 | subject=/CN=pe2enpmas300.corp.company.net/O=SplunkUser
splunkforwarder_1 | Getting CA Private Key
splunkforwarder_1 | writing RSA key
splunkforwarder_1 | 2016-01-12 08:53:24.785 +0000 splunkd started (build 271043)
Another false alarm... It is NOT working... I had Splunk running in the host
.
[deploy@pe2enpmas300 ~]$ sudo su
[root@pe2enpmas300 deploy]# cd /npmo-data/npmo-server/
[root@pe2enpmas300 npmo-server]# service splunk status
Splunk status:
splunkd is running (PID: 25381).
splunk helpers are running (PIDs: 25382).
[root@pe2enpmas300 npmo-server]# service splunk stop
I restarted all the containers again and splunk stopped recieving events...
Both docker images present this same problem: https://answers.splunk.com/answers/39497/unable-to-access-the-webui-and-im-getting-this-error-in-my-splunkd-log-anyone-know-how-to-fix-this.html
I'm not sure if this is caused by Docker images or if we need to share the server's cert
.
Other Potential solution is to automate the creation of the SSL Certs... Maybe?
@marcellodesales in your server.conf you have sslKeysfilePassword setup, do you actually map any your own certificates? Could you try to remove this line from conf?
@outcoldman Let me try that! I don't think I do...
@outcoldman I get the same exact error I reported before and they are the same between the 2 docker images (yours and ours)...
splunkforwarder_1 |
splunkforwarder_1 | Splunk> Now with more code!
splunkforwarder_1 |
splunkforwarder_1 | Checking prerequisites...
splunkforwarder_1 | Checking mgmt port [8089]: New certs have been generated in '/opt/splunkforwarder/etc/auth'.
splunkforwarder_1 | Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
splunkforwarder_1 | Couldn't initialize SSL Context for HTTPClient in ServerConfig
splunkforwarder_1 | Generating a 1024 bit RSA private key
splunkforwarder_1 | ..............................................................++++++
splunkforwarder_1 | ..........................++++++
splunkforwarder_1 | writing new private key to 'privKeySecure.pem'
splunkforwarder_1 | -----
splunkforwarder_1 | Signature ok
splunkforwarder_1 | subject=/CN=pe2enpmas300.corp.company.net/O=SplunkUser
splunkforwarder_1 | Getting CA Private Key
splunkforwarder_1 | writing RSA key
splunkforwarder_1 | 2016-01-12 15:20:40.571 +0000 splunkd started (build 271043)
01-12-2016 15:20:40.922 +0000 INFO loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:20:40.922 +0000 INFO loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:20:40.927 +0000 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=101077092 error:06065064:digital envelope rout
ines:EVP_DecryptFinal_ex:bad decrypt.
01-12-2016 15:20:40.927 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
01-12-2016 15:20:40.927 +0000 ERROR HTTPServer - SSL will not be enabled
@marcellodesales and please remove pass4SymmKey as well
@outcoldman ok Let me try it...
@outcoldman It finally solved the ERROR events with your image
... I'm recreating all the containers and verify...
[root@pe2enpmas300 npmo-server]# docker exec -ti npmoserver_splunkforwarder_1 bash
root@cd9d03dfce3b:/opt/splunk# vi var/log/
introspection/ splunk/
root@cd9d03dfce3b:/opt/splunk# vi var/log/splunk/splunkd.log
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - My newly generated GUID is C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - Found no site defined in server.conf
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - My hostname is "cd9d03dfce3b".
01-12-2016 15:45:34.600 +0000 INFO ServerConfig - Setting HTTP server compression state=on
01-12-2016 15:45:34.600 +0000 INFO ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 15:45:34.600 +0000 INFO ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 15:45:34.604 +0000 INFO LicenseMgr - Initing LicenseMgr
01-12-2016 15:45:34.604 +0000 INFO LMConfig - serverName=pe2enpmas300.corp.company.net guid=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.604 +0000 INFO LMConfig - connection_timeout=30
01-12-2016 15:45:34.604 +0000 INFO LMConfig - send_timeout=30
01-12-2016 15:45:34.604 +0000 INFO LMConfig - receive_timeout=30
01-12-2016 15:45:34.604 +0000 INFO LMConfig - squash_threshold=2000
01-12-2016 15:45:34.604 +0000 INFO LMConfig - strict_pool_quota=1
01-12-2016 15:45:34.604 +0000 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 15:45:34.604 +0000 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - closing stack mgr
01-12-2016 15:45:34.604 +0000 INFO LMSlaveInfo - all slaves cleared
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 15:45:34.604 +0000 INFO ServerRoles - Declared role=license_master.
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - init completed [C3A35716-9824-4BDC-83D8-4C67FE8077DD,Forwarder,runContext_splunkd=true]
01-12-2016 15:45:34.604 +0000 INFO LicenseMgr - StackMgr init complete...
01-12-2016 15:45:34.604 +0000 INFO LMTracker - init'ing slaveId=C3A35716-9824-4BDC-83D8-4C67FE8077DD label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 15:45:34.605 +0000 INFO LMTracker - enabling implicit feature set
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - attempting to ping master=self from slave=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.605 +0000 INFO LMSlaveInfo - new slave='C3A35716-9824-4BDC-83D8-4C67FE8077DD' created
01-12-2016 15:45:34.605 +0000 INFO LMSlaveInfo - Detected that masterTimeFromSlave(ZERO_TIME) < lastRolloverTime(Tue Jan 12 00:00:00 2016), meaning that the mas
ter has already rolled over. Ignore slave persisted usage.
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
:1
01-12-2016 15:45:34.805 +0000 INFO DS_DC_Common - Initializing core facilities of PubSub system.
01-12-2016 15:45:34.813 +0000 INFO DC:DeploymentClient - target-broker clause is missing.
01-12-2016 15:45:34.813 +0000 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-12-2016 15:45:34.813 +0000 INFO DS_DC_Common - Deployment Client not initialized.
01-12-2016 15:45:34.813 +0000 INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-12-2016 15:45:34.813 +0000 INFO IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-12-2016 15:45:34.813 +0000 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-12-2016 15:45:34.814 +0000 INFO IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory). This is normal when splunk is first starting up.
01-12-2016 15:45:34.815 +0000 WARN DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 15:45:34.815 +0000 INFO IndexProcessor - running splunkd specific init
01-12-2016 15:45:34.816 +0000 INFO loader - Initializing from configuration
01-12-2016 15:45:34.817 +0000 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 15:45:34.819 +0000 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 15:45:34.819 +0000 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 15:45:34.966 +0000 INFO TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 15:45:34.966 +0000 INFO ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Launching the pipelines.
01-12-2016 15:45:34.973 +0000 INFO loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:45:34.973 +0000 INFO loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:45:35.018 +0000 INFO TailingProcessor - TailWatcher initializing...
01-12-2016 15:45:35.018 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:45:35.031 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
:1
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - Will generate GUID, as none found on this server.
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - My newly generated GUID is C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - My server name is "pe2enpmas300.corp.company.net".
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - Found no site defined in server.conf
01-12-2016 15:45:34.589 +0000 INFO ServerConfig - My hostname is "cd9d03dfce3b".
01-12-2016 15:45:34.600 +0000 INFO ServerConfig - Setting HTTP server compression state=on
01-12-2016 15:45:34.600 +0000 INFO ServerConfig - Setting HTTP client compression state=0 (false)
01-12-2016 15:45:34.600 +0000 INFO ServerConfig - Default output queue for file-based input: parsingQueue.
01-12-2016 15:45:34.604 +0000 INFO LicenseMgr - Initing LicenseMgr
01-12-2016 15:45:34.604 +0000 INFO LMConfig - serverName=pe2enpmas300.corp.company.net guid=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.604 +0000 INFO LMConfig - connection_timeout=30
01-12-2016 15:45:34.604 +0000 INFO LMConfig - send_timeout=30
01-12-2016 15:45:34.604 +0000 INFO LMConfig - receive_timeout=30
01-12-2016 15:45:34.604 +0000 INFO LMConfig - squash_threshold=2000
01-12-2016 15:45:34.604 +0000 INFO LMConfig - strict_pool_quota=1
01-12-2016 15:45:34.604 +0000 INFO LMConfig - key=pool_suggestion not found in licenser stanza of server.conf, defaulting=''
01-12-2016 15:45:34.604 +0000 INFO LicenseMgr - Initing LicenseMgr runContext_splunkd=true
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - closing stack mgr
01-12-2016 15:45:34.604 +0000 INFO LMSlaveInfo - all slaves cleared
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - added pool auto_generated_pool_forwarder to stack forwarder
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - added pool auto_generated_pool_free to stack free
01-12-2016 15:45:34.604 +0000 INFO ServerRoles - Declared role=license_master.
01-12-2016 15:45:34.604 +0000 INFO LMStackMgr - init completed [C3A35716-9824-4BDC-83D8-4C67FE8077DD,Forwarder,runContext_splunkd=true]
01-12-2016 15:45:34.604 +0000 INFO LicenseMgr - StackMgr init complete...
01-12-2016 15:45:34.604 +0000 INFO LMTracker - init'ing slaveId=C3A35716-9824-4BDC-83D8-4C67FE8077DD label=pe2enpmas300.corp.company.net [30,30,self]
01-12-2016 15:45:34.605 +0000 INFO LMTracker - enabling implicit feature set
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=Acceleration state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=AdvancedSearchCommands state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=AdvancedXML state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=CustomRoles state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=GuestPass state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=KVStore state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=LDAPAuth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=MultisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=NontableLookups state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=RollingWindowAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=ScheduledAlerts state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=ScheduledReports state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=SearchheadPooling state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=UnisiteClustering state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - attempting to ping master=self from slave=C3A35716-9824-4BDC-83D8-4C67FE8077DD
01-12-2016 15:45:34.605 +0000 INFO LMSlaveInfo - new slave='C3A35716-9824-4BDC-83D8-4C67FE8077DD' created
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=Acceleration state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=AdvancedSearchCommands state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=AdvancedXML state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=Alerting state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=AllowDuplicateKeys state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=Auth state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=CanBeRemoteMaster state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=CustomRoles state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=DistSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=FwdData state=ENABLED (featureStatus=1)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=GuestPass state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=KVStore state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=LDAPAuth state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=RcvSearch state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=ScheduledAlerts state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LMTracker - Setting feature=UnisiteClustering state=DISABLED_DUE_TO_LICENSE (featureStatus=2)
01-12-2016 15:45:34.605 +0000 INFO LicenseMgr - Tracker init complete...
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: virtual address space size: unlimited
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: data segment size: unlimited
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: resident memory size: unlimited
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: stack size: 8388608 bytes [hard maximum: unlimited]
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: core file size: unlimited
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: data file size: unlimited
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: open files: 1048576 files
01-12-2016 15:45:34.609 +0000 INFO ulimit - Limit: user processes: 1048576 processes
01-12-2016 15:45:34.611 +0000 INFO loader - Splunkd starting (build 271043).
01-12-2016 15:45:34.611 +0000 INFO loader - System info: Linux, cd9d03dfce3b, 3.10.0-327.3.1.el7.x86_64, #1 SMP Fri Nov 20 05:40:26 EST 2015, x86_64.
01-12-2016 15:45:34.611 +0000 INFO loader - Detected 4 (virtual) CPUs, 4 CPU cores, and 15880MB RAM
01-12-2016 15:45:34.611 +0000 INFO loader - Maximum number of threads (approximate): 7940
01-12-2016 15:45:34.611 +0000 INFO loader - Arguments are: "-p" "8089" "start"
01-12-2016 15:45:34.611 +0000 INFO loader - Getting configuration data from: /opt/splunk/etc/myinstall/splunkd.xml
01-12-2016 15:45:34.612 +0000 INFO loader - SPLUNK_MODULE_PATH environment variable not found - defaulting to /opt/splunk/etc/modules
01-12-2016 15:45:34.612 +0000 INFO loader - loading modules from /opt/splunk/etc/modules
01-12-2016 15:45:34.613 +0000 INFO loader - Writing out composite configuration file: /opt/splunk/var/run/splunk/composite.xml
01-12-2016 15:45:34.617 +0000 INFO ServerRoles - Declared role=universal_forwarder.
01-12-2016 15:45:34.643 +0000 INFO loader - Setting SSL configuration.
01-12-2016 15:45:34.643 +0000 INFO loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
01-12-2016 15:45:34.643 +0000 INFO loader - Using cipher suite TLSv1+HIGH:@STRENGTH
01-12-2016 15:45:34.643 +0000 INFO loader - ECDH curve not configured
01-12-2016 15:45:34.803 +0000 INFO SpecFiles - Found external scheme definition for stanza "MonitorNoHandle://" with 2 parameters: disabled, index
01-12-2016 15:45:34.803 +0000 INFO SpecFiles - Found external scheme definition for stanza "WinHostMon://" with 4 parameters: type, interval, disabled, index
01-12-2016 15:45:34.803 +0000 INFO SpecFiles - Found external scheme definition for stanza "WinPrintMon://" with 4 parameters: type, baseline, disabled, index
01-12-2016 15:45:34.804 +0000 WARN ClusteringMgr - Ignoring clustering configuration, the active license disables this feature.
01-12-2016 15:45:34.805 +0000 INFO SHPoolingMgr - shpooling disabled
01-12-2016 15:45:34.805 +0000 INFO DS_DC_Common - Initializing the PubSub system.
01-12-2016 15:45:34.805 +0000 INFO DS_DC_Common - Initializing core facilities of PubSub system.
01-12-2016 15:45:34.813 +0000 INFO DC:DeploymentClient - target-broker clause is missing.
01-12-2016 15:45:34.813 +0000 WARN DC:DeploymentClient - DeploymentClient explicitly disabled through config.
01-12-2016 15:45:34.813 +0000 INFO DS_DC_Common - Deployment Client not initialized.
01-12-2016 15:45:34.813 +0000 INFO DS_DC_Common - Deployment Server not available on a dedicated forwarder.
01-12-2016 15:45:34.813 +0000 INFO IntrospectionGenerator:disk_objects - Enabled: indexes|volumes|dispatch=false fishbucket=true partitions=false
01-12-2016 15:45:34.813 +0000 INFO IntrospectionGenerator:disk_objects - I-data gathering (Disk Objects) starting; period=600s
01-12-2016 15:45:34.814 +0000 INFO IntrospectionGenerator:disk_objects - Unable to getSizeOnDisk of='/opt/splunk/var/lib/splunk/fishbucket/splunk_private_db' (N
o such file or directory). This is normal when splunk is first starting up.
01-12-2016 15:45:34.815 +0000 WARN DistributedPeerManager - feature=DistSearch not enabled for your license level
01-12-2016 15:45:34.815 +0000 INFO IndexProcessor - running splunkd specific init
01-12-2016 15:45:34.816 +0000 INFO loader - Initializing from configuration
01-12-2016 15:45:34.817 +0000 INFO PipelineComponent - Pipeline fifo disabled in default-mode.conf file
01-12-2016 15:45:34.819 +0000 INFO TcpInputConfig - SSL clause not found or servercert not provided - SSL ports will not be available
01-12-2016 15:45:34.819 +0000 INFO TcpInputProc - Registering metrics callback for: tcpin_connections
01-12-2016 15:45:34.966 +0000 INFO TcpOutputProc - Initializing with fwdtype=lwf
01-12-2016 15:45:34.966 +0000 INFO ServerRoles - Declared role=lightweight_forwarder.
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : forwardedindex.0.whitelist
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : forwardedindex.1.blacklist
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : forwardedindex.2.whitelist
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg310.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg311.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to oe2esstlg312.corp.company.net:9997
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - tcpout group primary_indexers using Auto load balanced forwarding
01-12-2016 15:45:34.971 +0000 INFO TcpOutputProc - Group primary_indexers initialized with maxQueueSize=512000 in bytes.
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Pipeline merging disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Pipeline typing disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Pipeline vix disabled in default-mode.conf file
01-12-2016 15:45:34.971 +0000 INFO PipelineComponent - Launching the pipelines.
01-12-2016 15:45:34.973 +0000 INFO loader - Limiting REST HTTP server to 349525 sockets
01-12-2016 15:45:34.973 +0000 INFO loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:45:35.018 +0000 INFO TailingProcessor - TailWatcher initializing...
01-12-2016 15:45:35.018 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:45:35.019 +0000 INFO TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:45:35.019 +0000 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:45:35.031 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
@outcoldman
Nothing yet... Although I don't see any errors in the logs...
01-12-2016 15:55:02.774 +0000 INFO loader - Limiting REST HTTP server to 2646 threads
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - TailWatcher initializing...
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Parsing configuration stanza: batch://$SPLUNK_HOME/var/spool/splunk/...stash_new.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/etc/splunk.version.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/metrics.log.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor://$SPLUNK_HOME/var/log/splunk/splunkd.log.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Parsing configuration stanza: monitor:///var/log/messages.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/etc/splunk.version.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/log/splunk.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Adding watch on path: /opt/splunk/var/spool/splunk.
01-12-2016 15:55:02.841 +0000 INFO TailingProcessor - Adding watch on path: /var/log/messages.
01-12-2016 15:55:02.841 +0000 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 15:55:02.853 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
01-12-2016 15:55:32.622 +0000 WARN AuthenticationManagerSplunk - Seed file is not present. Defaulting to generic username/pass pair.
01-12-2016 15:55:32.975 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-12-2016 15:56:02.927 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
01-12-2016 15:56:33.254 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
01-12-2016 15:57:03.284 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.129:9997
01-12-2016 15:57:33.315 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.115:9997
The container can see the current logs in the /var/log/messages
...
root@51e26f57eb48:/opt/splunk# tail -f /var/log/messages
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.690Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 301: change for package lodash._isiterateecall"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.693Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 302: change for package lodash.templatesettings"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.696Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 303: change for package lodash.escape"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.697Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 304: change for package lodash._getnative"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.702Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 305: change for package lodash.keys"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.706Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 306: change for package lodash.isarray"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.709Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 307: change for package lodash.isarguments"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.710Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 308: change for package clone-stats"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.729Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 309: change for package clone"}
Jan 12 07:55:10 pe2enpmas300 docker/esfollower[1358]: {"time":"2016-01-12T15:55:10.732Z","hostname":"a8409e87525c","pid":7,"level":"info","name":"mirror-search","message":"seq 310: change for package flagged-respawn"}
Nothing significant found in the logs either... :(
root@51e26f57eb48:/opt/splunk# ls -la var/log/splunk/
total 92
drwx------ 2 root root 4096 Jan 12 16:01 .
drwx--x--x 4 root root 39 Jan 12 15:55 ..
-rw------- 1 root root 140 Jan 12 15:55 audit.log
-rw------- 1 root root 0 Jan 12 15:55 btool.log
-rw------- 1 root root 307 Jan 12 16:00 conf.log
-rw------- 1 root root 64 Jan 12 15:55 first_install.log
-rw------- 1 root root 0 Jan 12 15:55 license_audit.log
-rw------- 1 root root 0 Jan 12 15:55 license_usage.log
-rw------- 1 root root 47883 Jan 12 16:01 metrics.log
-rw------- 1 root root 0 Jan 12 15:55 mongod.log
-rw------- 1 root root 0 Jan 12 15:55 remote_searches.log
-rw------- 1 root root 0 Jan 12 15:55 scheduler.log
-rw------- 1 root root 0 Jan 12 15:55 searchhistory.log
-rw------- 1 root root 0 Jan 12 15:55 splunkd_access.log
-rw------- 1 root root 20262 Jan 12 16:01 splunkd.log
-rw------- 1 root root 61 Jan 12 15:55 splunkd_stderr.log
-rw------- 1 root root 0 Jan 12 15:55 splunkd_stdout.log
-rw------- 1 root root 0 Jan 12 15:55 splunkd_ui_access.log
-rw------- 1 root root 1125 Jan 12 15:55 splunkd-utility.log
root@51e26f57eb48:/opt/splunk# cat var/log/splunk/splunkd_stderr.log
2016-01-12 15:55:02.408 +0000 splunkd started (build 271043)
Maybe turning on DEBUG
mode of the forward can show something else?
@marcellodesales sure we can turn on DEBUG for something, but I have no clue where we lose data.
Another suggestion - try to use splunk standalone instance just to try if you will get these messages indexed inside Splunk without actually forwarding them.
@outcoldman
Suddenly, things started working this time, as I see it picked up the files... The only thing I did was to add a debug line as specified in step 6 of https://answers.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certificates-and-authentication.html... But that should not affect anything I'm guessing...
01-12-2016 16:16:49.475 +0000 INFO BatchReader - State transitioning from 2 to 0 (initOrResume).
01-12-2016 16:16:49.494 +0000 INFO WatchedFile - Will begin reading at offset=19716950 for file='/var/log/messages'.
01-12-2016 16:16:49.497 +0000 INFO WatchedFile - Will begin reading at offset=1125 for file='/opt/splunk/var/log/splunk/splunkd-utility.log'.
01-12-2016 16:16:49.499 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/btoo
l.log'.
01-12-2016 16:16:49.502 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/sear
chhistory.log'.
01-12-2016 16:16:49.503 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_access.log'.
01-12-2016 16:16:49.505 +0000 INFO WatchedFile - Will begin reading at offset=48771 for file='/opt/splunk/var/log/splunk/audit.log'.
01-12-2016 16:16:49.507 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/mong
od.log'.
01-12-2016 16:16:49.509 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/lice
nse_usage.log'.
01-12-2016 16:16:49.510 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/lice
nse_audit.log'.
01-12-2016 16:16:49.513 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/remo
te_searches.log'.
01-12-2016 16:16:49.515 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/sche
duler.log'.
01-12-2016 16:16:49.516 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_ui_access.log'.
01-12-2016 16:16:49.517 +0000 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='/opt/splunk/var/log/splunk/splu
nkd_stdout.log'.
01-12-2016 16:16:49.519 +0000 INFO WatchedFile - Will begin reading at offset=61 for file='/opt/splunk/var/log/splunk/splunkd_stderr.log'.
01-12-2016 16:16:49.558 +0000 INFO TcpOutputProc - Connected to idx=10.153.194.113:9997
The logs event date in Splunk are using a different time...
root@51e26f57eb48:/opt/splunk# date
Tue Jan 12 16:18:49 UTC 2016
root@51e26f57eb48:/opt/splunk# tail -f /var/log/messages
Jan 12 08:21:44 pe2enpmas300 docker/policyfollower[1358]: storing document admn at 640999
Jan 12 08:21:44 pe2enpmas300 docker/policyfollower[1358]: done processing admn at 640999
Jan 12 08:21:45 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:45.186Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:46 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:46.533Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:16:21:46 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:47 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:47.189Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:47 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:47.335Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:16:21:47 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 12 08:21:49 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:49.192Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:51 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:51.195Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:51 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:51.467Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:16:21:51 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:52 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:52.385Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:16:21:52 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 0 ms"}
Jan 12 08:21:53 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:53.197Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:55 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:55.200Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:56 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:56.525Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.5 - - [12/Jan/2016:16:21:56 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Jan 12 08:21:57 pe2enpmas300 docker/policyfollower[1358]: 2016-01-12T16:21:57.203Z: 0 queued changes, 0 open requests, 0 retries pending
Jan 12 08:21:57 pe2enpmas300 docker/frontdoor[1358]: {"time":"2016-01-12T16:21:57.302Z","hostname":"d66aafce5d6d","pid":9,"level":"info","name":"npme","message":"10.137.66.6 - - [12/Jan/2016:16:21:57 +0000] \"GET /_monitor/role HTTP/0.9\" 200 7 \"-\" \"-\" \"-\" 1 ms"}
Where there's of offset in the Web UI of Less 8 hours... We are very close now :+1:
How can I correlate the Time
column with the time displayed in the Event
column???
I noticed that the timezone
in the container is UTC
root@51e26f57eb48:/opt/splunk# date
Tue Jan 12 16:26:24 UTC 2016
@marcellodesales do you mean that it shows you wrong time or not expected TimeZone? I guess the difference of 8 hours is expected if your indexers show you time in your timezone and forwarder collects everything in UTC. If you are in Pacific Time zone - looks like everything is working.
@outcoldman Everything is working!!! The only problem is that we are in the PST timezone, but the Time shown in the Time
column is less 8 hrs... Is there a way to set the forwarder to collect everything in PST instead?
I'm tring to set the time inside the container, but no luck so far... What I tried was:
Mounting the timezone did not work because my host is RHEL7 and I did not find the same files in your container... (not sure...)
@marcellodesales no need to modify anything on system, in logs everything looks good, you have timestaps in UTC timezone and they show the right time.
Please read http://docs.splunk.com/Documentation/Splunk/6.2.8/Data/Configuretimestamprecognition
Take a look on etc/system/default/props.conf
definition for syslog
[syslog]
pulldown_type = true
maxDist = 3
TIME_FORMAT = %b %d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 32
TRANSFORMS = syslog-host
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = False
category = Operating System
description = Output produced by many syslog daemons, as described in RFC3164 by the IETF
As you can see it does not have TZ
, so add props.conf
inside etc/system/default/props.conf
[syslog]
TZ = UTC
And actually you have better time inside of the syslog message - I would recommend to set timezone parsing of these values instead of timestamps from syslog.
@outcoldman Sounds good and thanks for all your help... The deployment on the second host worked... Here's what I have so far:
splunkforwarderData:
image: busybox
volumes:
- ./monitor/splunk:/opt/splunk/etc/system/local
- /var/log/messages:/var/log/messages:ro
- /usr/share/zoneinfo/America/Los_Angeles:/etc/localtime:ro
The server.conf without any SSL setting... Since the container needs to write the values for the server, the volume is mounted with write
permission...
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
[general]
serverName = pe2enpmas300.company.intuit.net
Thanks a lot for more pointers... I will verify with OPS how the proceed with that...
:+1:
Hi there,
I'm new to Splunk and the OPS team gave me the following files under the
/opt/splunkforwarder/etc/system/local/
:According to http://blogs.splunk.com/2015/08/24/collecting-docker-logs-and-stats-with-splunk/, I'd like to configure a single SplunkForwarder container to collect
syslog
for all the containers... I'd like also to disclose:company
below is a BIG Splunk customer, but so far nobody from OPS team supports DockerThis is a host installation with the splunk-forwarder 6.2.x... The content of the files is as follows:
inputs.conf
outputs.conf
server.conf
Questions
For instance, I just copied the files from the splunk forward from the host and I'm mounting in the data container... Let's start with the container in a separate docker-compose...
docker-compose-monitoring.yml
Since I'm running docker as root (with SSL enabled), then I opened port
514
....docker-compose.yml
I just use extensions https://docs.docker.com/compose/extends/ and the ${HOSTNAME} variable substitution https://docs.docker.com/compose/compose-file/#variable-substitution
For the configuration, I'm mounting it...
Here's the file Dockerfile
Docker inspect command shows the mounted settings...
I verified that the file is in the container as well...
Any help is appreciated...