No logs in splunk

[sharmmoh1983]

I have followed all the steps mentioned and it shows that the daemonset is running on each of the nodes, However, when I see the Splunk Dashboard, I see no logs. Any pointers?

How to debug deamonset functionality?

[matthewmodestino]

Please post your the yaml you used to deploy the daemonset. Be sure to scrub any sensitive info.

Are you familiar with configuring/troubleshooting the Splunk UF?

You can exec into the pod, cd to /opt/splunk/bin and run ./splunk list forward-server to see forwarding connection status. You will need the password that was generated for the UF. You can also grep /opt/splunk/var/logs/splunk/splunkd.log for TcpOutputProc to see whats up.

There is a lot of “it depends” here, so please provide more info about what you have configured and from inside the pod.

[sharmmoh1983]

Also there are no log files getting generated

I am getting : Active forwards: None Configured but inactive forwards: input-prd-p-xxxxxxx.cloud.splunk.com:9XXXX

apiVersion: v1
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: splunk-forwarder-config
data:
  cacert.pem: |

  client.pem: |

  limits.conf: |-
    # By default a universal or light forwarder is limited to 256kB/s
    # Either set a different limit in kB/s, or set the value to zero to
    # have no limit.
    # Note that a full speed UF can overwhelm a single indexer.

    [thruput]
    maxKBps = 256
  outputs.conf: |

  inputs.conf: |
     # watch all files in <path>
     [monitor:///var/log/containers/*.log]
     # extract `host` from the first group in the filename
     host_regex = /var/log/containers/(.*)_.*_.*\.log
     # set source type to Kubernetes
     sourcetype = httpeven
`

Also how to give index name as we don in Http Event Collector?

[matthewmodestino]

are you using the certs that come with the UF for cloud? I assume they would be the ones you need for hec?

Your config is inactive, so we should review the config. My guess is ssl config or certs

The log files are in the container.

kubectl -n <namespace> exec -it <pod> bash

cd /opt/splunk/var/log/splunk

[sharmmoh1983]

@matthewmodestino Yes I am using certs but where do i mention which index name to send and also how splunk UF is different from HEC. I dont think certs will work for the same

My main issue is how to give the same parameters as giving for HEC in configuration file:

I have gone to the location and following files are visible:

-rw------- 1 root root 70 Jul 26 02:23 first_install.log -rw------- 1 root root 0 Jul 26 02:23 btool.log -rw------- 1 root root 555 Jul 26 02:23 splunkd-utility.log -rw------- 1 root root 0 Jul 26 02:23 splunkd_ui_access.log -rw------- 1 root root 0 Jul 26 02:23 searchhistory.log -rw------- 1 root root 0 Jul 26 02:23 scheduler.log -rw------- 1 root root 0 Jul 26 02:23 remote_searches.log -rw------- 1 root root 0 Jul 26 02:23 mongod.log -rw------- 1 root root 0 Jul 26 02:23 license_usage_summary.log -rw------- 1 root root 0 Jul 26 02:23 license_usage.log -rw------- 1 root root 0 Jul 26 02:23 splunkd_stdout.log -rw------- 1 root root 67 Jul 26 02:23 splunkd_stderr.log -rw------- 1 root root 296 Jul 26 02:23 conf.log -rw------- 1 root root 907 Jul 26 02:27 splunkd_access.log -rw------- 1 root root 63212 Jul 26 02:33 audit.log -rw------- 1 root root 35427 Jul 26 02:54 splunkd.log -rw------- 1 root root 363869 Jul 26 02:54 metrics.log

I got this following error in the middle of the file:

07-26-2018 02:23:46.769 +0000 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/apps/splunkclouduf/default/client.pem errno=33558530 error:02001002:system library:fopen:No such file or directory 07-26-2018 02:23:46.769 +0000 ERROR TcpOutputProc - Error initializing SSL context - check splunkd.log regarding configuration error for server input-prd-p-xxxxxxx.cloud.splunk.com:99xx 07-26-2018 02:23:46.769 +0000 INFO TcpOutputProc - tcpout group splunkcloud using Auto load balanced forwarding

I am cheking splunkd.log...

I have updated my daemon file

- mountPath: /opt/splunk/etc/apps/splunkclouduf/default

    - mountPath: /var/opt/splunk/etc/apps/search/local
      name: splunk-config

because of the work around described in issues

[matthewmodestino]

disregard my comment anout hec, i have been focused on Splunk Connect for Kubernetes...

This is straight UF forwarding. Based on the logs, the cloud uf app is not in the right spot, i guess... is your outputs.conf really blank in your file or you just scrubbed it?

Your outputs config is definitey appears to be the issue.

index can be set in inputs.conf

[sharmmoh1983]

@matthewmodestino

Thanks for your input.. But I will explain my issue..

When I am running daemonset with following configuration with confg path mount: mountPath: /opt/splunk/etc/apps/splunkclouduf/default

Deamonset always crashes chown: changing ownership of ‘/opt/splunk/etc/system/local/inputs.conf’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/inputs.conf’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038/SPLUNK_FORWARD_SERVER’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..2018_03_15_23_51_19.952137038’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/SPLUNK_FORWARD_SERVER’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local/..data’: Read-only file system chown: changing ownership of ‘/opt/splunk/etc/system/local’: Read-only file system

and as per #70 , when I change the file like following: - mountPath: /var/opt/splunk/etc/apps/search/local Deamonset is running fine but the issue is while making ssl connection , image is looking for ssl certificates in /opt/splunk/etc/apps/splunkclouduf/default which is commented now...

So how to solve the issue

[matthewmodestino]

here is how I mounted apps in one of my kubernetes configs:

            - name: splunk-idxcluster-config
              mountPath: /opt/splunk/etc
            - name: splunk-idxcluster-data
              mountPath: /opt/splunk/var
            - name: k8s-cluster-idx-base-local
              mountPath: /var/opt/splunk/etc/apps/k8s-cluster-idx-base/local
            - name: k8s-cluster-idx-base-metadata
              mountPath: /var/opt/splunk/etc/apps/k8s-cluster-idx-base/metadata
      volumes:
        - name: k8s-cluster-idx-base-local
          configMap:
            name: k8s-cluster-idx-base-local
        - name: k8s-cluster-idx-base-metadata
          configMap:
            name: k8s-cluster-idx-base-metadata

Create a configmap with the contents of the clouduf app, and mount it into /var/opt/splunk/etc/apps/splunkclouduf/, you will have to do this for each dir under the clouduf. I dont have the dir structure handy at the moment, but that how I control my statefulset...should work for the daemonset.

I just ran this for any dir under my splunk app...:

kubectl -n splunk create configmap k8s-cluster-fwd-local --from-file=local
kubectl -n splunk create configmap k8s-cluster-fwd-metadata --from-file=metadata

[sharmmoh1983]

And what path is appearing for the ssl certificates in splunk log files?..Why there are two config map?

[sharmmoh1983]

I have made some progress with your help but now I am seeing following error: Path which I have mentioned for watch is /var/log/containers/

07-26-2018 05:26:29.300 +0000 ERROR TailReader - Unable to resolve path for symlink: /var/log/containers/ Is this is some user access issue?But I am using root as splunk user. How to check messages are successfully posted from logs?

[matthewmodestino]

Before we go any further, can you tell me more about your kubernetes deploy? Are you just testing/learning, or are you looking to monitor a real enviro?

Where is this kubernetes cluster running?

Looks like you didn’t mount the correct volumes is my guess.

We posted a prototype UF deamonset here: https://github.com/splunk/docker-itmonitoring/blob/master/README-k8s.md

I suggest you take a look at the configs there, and we should probably move this thread there as this is beyond the docker image now...

If you are planning on using this in a production scenario you should check out our supported option, Splunk Connect for Kubernetes here: https://github.com/splunk/splunk-connect-for-kubernetes

While the UF will move the logs, this docker image is not officially Splunk supported and you will run into challenges with parsing the json logs and you wont be able to monitor journald. These are use cases we are looking at enhancing with the UF, but as of today, you should only really use the UF in a sidecar pattern or for testing/learning, use connect for kube if this is going to see real enviros

[sharmmoh1983]

@matthewmodestino

Thanks for your support

We are planning to push log data to splunk in real environments and hence trying out different options

I am also analyzing fluentd to do the splunk integration

But I dont know what is the right image to do so.

So according to your opinion , splunk/universalforwarder is not right for real env and we should go for https://github.com/splunk/splunk-connect-for-kubernetes

By the way,I resolved the issue by working out the mount issues. Splunk logs looks all ok, but I don't sees any logs showing data successfully posted to splunk

Do you know is there any log which shows what all data got posted to splunk from the logs which gets watched by daemon

Also is splunk-connect-for-kubernetes enterprise supported or not as we have the support from splunk?

[matthewmodestino]

Yes, Splunk Connect for Kubernetes is our supported option, I strongly recommend you go that route at this time.