Kubernetes Pod name and container ID

[cricketfan5]

I am new to splunk and kubernetes. I am able to setup a kubernetes with universal splunk forwarder. I am able to see logs in the UI. I don't see the name of the pod or container from which it is forwarding. Is there a way to configure to get the pod name and container id in the logs?.

[ae6rt]

One possible way is to use search-time extracted fields in your props.conf file on the indexer:

EXTRACT-sourcefields = /var/log/containers/(?<pod>.*)_(?<namespace>.*)_(?<containerName>.*)-(?<containerID>.*)\.log in source

[cricketfan5]

@ae6rt Thanks for the response. I have followed the kubernetes readme file and I have installed a daemonset as per the document. where would be the props.conf file.

[ae6rt]

If you are starting the indexer as a DaemonSet, you would include, most likely, a props.conf file as part of a ConfigMap for the indexer pod. You would then mount the ConfigMap in a location where the indexer expects its config files.

[cricketfan5]

I followed the same yaml file for creating splunk forwarder as a daemon set. But don't see anything as a indexer in the file. Can you please help me figuring this?. Thanks

[cricketfan5]

I see a props.conf file in the splunkforwarder/apps-k8 folder. Is that the one are you talking about ?.

[ae6rt]

This one: https://github.com/splunk/docker-itmonitoring/blob/master/app-k8s/default/props.conf#L7. Put your extracted fields instruction within this kubernetes sourcetype.

[cricketfan5]

Thanks

[cricketfan5]

I tried updating the values in the props.conf. But still not able to see the host name and pod name and container name anything as such. What might the wrong ?. Thanks

[matthewmodestino]

Hey crickerfan5,

Where are you sending your events to? The Splunk instance in the full demo yaml? Or one outside the cluster? The props/transforms need to be on the indexer, so make sure you are updating them there.

I'll look at updating the repo with the props.conf/transforms to create indextime fields for pod, namespace, container_id, which would look something like this:

in transforms.conf ON YOUR INDEXER:

[k8s_extractions]
SOURCE_KEY = MetaData:Source
DEST_KEY = _meta
REGEX = ^source::/var/log/containers/([^_]+)_([^_]+)_(.*)-([^\-]+)\.log$
FORMAT = $0 pod::$1 namespace::$2 container::$3_$4
WRITE_META = true

in the Props.conf ON YOUR INDEXER add this line pointing to the transform:

TRANSFORMS-k8s_extractions = k8s_extractions

I would suggest you check out our recently released Splunk Connect for Kubernetes here:

https://github.com/splunk/splunk-connect-for-kubernetes

As this is our currently supported method of getting info out of k8s and will take care of said extractions for you.

[cricketfan5]

@matthewmodestino Thanks for the response. Do we need to create a new transforms.conf file?. And I am sending the logs outside the cluster.

[cricketfan5]

@matthewmodestino I have created a transforms.conf and added the text and updated the props.conf file too. But still I don't see any changes in the log in splunk UI. Any help is appreciated. Thanks

[matthewmodestino]

Where are you making these changes? Directly in the container? Making these changes will likely require you to delete the pod and re-deploy.

join us in #kubernetes on slack at splk.it/slack and we can help in more real-time. my username is @mattymo

[cricketfan5]

@matthewmodestino I am not able to find the slack channel. I signup into splk.it/slack. I made changes on the indexer side and created a new image built the daemonset.

[matthewmodestino]

visit splk.it/slack in your browser, it will take you to a google form for you to request access. The actual Slack channel is splunk-usergroups.slack.com

When you say indexer side, where is that indexer? did you only deploy the daemonset? to k8s, or is the indexer inside the cluster too?

[cricketfan5]

I followed docker-itmonitoring/README-k8s.md and deployed a daemonset. And added the transfroms.conf at https://github.com/splunk/docker-itmonitoring/blob/master/app-k8s/default/.

[matthewmodestino]

did you update your props.conf for the kubernetes sourcetype to make use of that transform?

Is you indexer inside kubernetes? or are you sending outside of the cluster? The props.conf and transforms need to be on whatever splunk indexers are receiving logs from the UFs in the daemonset

[cricketfan5]

Yes, I have updated.

[matthewmodestino]

and the app-k8s is installed on your indexers? And you restarted your indexers?

Are you searching in verbose mode in Splunk?

[cricketfan5]

I am don't see any app-k8s folder in the pod after the pod is created. And I am not searching in verbose mode in splunk. Thanks

[matthewmodestino]

what pod???

That app does not affect the UF daemonset. It is for the indexers. Check the complete demo yaml, you will see that we deploy UFs that send data to a full splunk indexer. That is where app-k8s lives.

These Indexed extraction are needed on the indexers you are sending to. If you only installed the logging daemonset and pointed them to a splunk indexer outside of kubernetes, you need to install that app on those indexers.

Or, just create your own searchtime extractions. Either way will work. Indextime is just more performant.

[cricketfan5]

Thanks for your response. I have installed the logging daemonset and pointed them to a splunk indexer outside of kubernetes i.e in the splunk UI. I have created the transforms.conf and the content and I updated the transform extractor in props.conf on indexer side. But I don't any changes in the splunk UI. am I missing something here?. Thanks.

[cricketfan5]

@matthewmodestino i tried as you said , But still no luck , i have below queries i m a newbee for the Splunk... props.conf should be in which pod (ta-k8s-logs, ta-k8s-meta, splunkenterprise). in props.conf index= k8s is by default, here shd i be changing to my index=abc. is there any other change i need to make in the pods configuraiton becaus i can see partial logs. Please let me know ...

[matthewmodestino]

splunkenterprise is where the props.conf and transforms.conf should be. The index can be whatever you like in inputs.conf in the ta-k8s-logs pod.

[cricketfan5]

@matthewmodestino I have added the props.conf and transform.conf in the splunkenterprise and I have index name from k8s to my own abc(index name) and built a new image and deployed a new daemonset. but still I don't see any info regarding meta data in the splunk UI. I see below error in the splunkd log

ERROR ExecProcessor - message from "/opt/splunk/etc/apps/ta-k8s-meta/bin/deployments.sh" jq: error: Cannot iterate over null

And tried executing the below command manually but I did not get any output and see unauthorized as output

curl -sSk -H "Authorization: Bearer $TOKEN" https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_PORT_443_TCP_PORT/apis/apps/v1beta2/daemonsets Am I missing something or doing wrong somewhere. Please help me.

[matthewmodestino]

now we are talking something completely different than indextime fields....

This is not a great forum for the level of assistance you are seeking. Join us on the slack chat and I can try to help.

The issues with the curl command are likely due to your clusterRole config, but it is impossible for me to assist without seeing what you have done at this point.

[cricketfan5]

I have registered in the splk.it/slack. But I haven't added in the channel.

[cricketfan5]

@matthewmodestino Hi , thanks for your support.

Here is what I am getting,

when I am using k8s-splunk-full-demo.yaml file with 3 images. Only one meta pod is creating and I am able to see the metadata in the splunk seperately. Can I get the metadata of the nodes along with the logs in the splunk.

Is this expected behavior or it should create the number of metapods as number of worker nodes and master nodes and is it possible to get the metadata along with the logs in the splunk?.

[matthewmodestino]

you should only need one meta pod, it is not a daemonset, just a deployment.

The metadata of the nodes should show up as a result of the nodes.sh scripted input that the meta pod calls. Make sure that the scripted inputs are running correctly and that them meta pod has the appropriate cluster role and binding to successfully curl the api

[cricketfan5]

@matthewmodestino I am able to see the metadata seperately in the Ui. But I am not able to get metadata along with the log of the container. Is there a way to configure so that if a log is forwarded to Splunk it should have the metadata information such that it should show from which host, which pod and which container the log is coming from ?. Thank you

[matthewmodestino]

well, the idea is that the logs should contain the minimum amount of metadata, being namespace, podname, container_ID. Those are extracted from the “source” field either at search time or via indextime fields(i provided indextime config above), then u can correlate it with the metadata by using those fields. ie create a lookup from the metadata which will automatically add the meta to each event, without having to constantly index it. This saves having to index metadata over and over...

Another option is to check out Splunk Connect for Kubernetes, our supported option for k8s data collection as of today

[cricketfan5]

@matthewmodestino Thanks for your time and support. Appreciated !!

[cricketfan5]

@matthewmodestino one more quick question, is it possible to parse the log as key pair value. If so, how can we do that and where should be add that. Thanks

[matthewmodestino]

depends on the log itself. What kv pairs are you hoping to see?

[halr9000]

If any event contains a KV pair that looks like this: "field=value" or "field = value", then Splunk will automatically extract those at search time without any action on your part. Does that help @cricketfan5 ?

[cricketfan5]

@halr9000 @matthewmodestino Thanks for you responses, after some research I understood that we cannot the parse the logs using universal forwarder ?.

If not, I have to extract some fields from the log and should be add after the log. here is the example

Now I am getting my log as { [-] log: {someinformation of appication here {msg"a":"1","b":"2","c":"3","d":"4" }

I want my log to be appear as so i want to extract the field so that it should appear as below in the splunk ui

{ [-] log: {someinformation of appication here {msg-"a":"1","b":"2","c":"3","d":"4"} } msg-{ a:1 b:2 c:3 d:4 }

[matthewmodestino]

Here is where you really should move over to Splunk Connect for Kubernetes...it unwraps the docker json wrapper from your log, whereas we still have work to do on the UF to have it parse json in that manner. It is possible to use sedcmd-fu to unwrap the log, but it is worth your efforts to go for the more complete, supported solution here:

https://github.com/splunk/splunk-connect-for-kubernetes

[halr9000]

@cricketfan5 what's the use case? If you want to visually see fields extracted alongside each event, then this is simply the "field extraction" feature. There's search-time and index-time field extraction. 99% of the time, search-time is fine, and adding indexed fields has both positive and negative performance considerations that you should understand before getting into that.

The app which @matthewmodestino mentions just above will help you with field extraction, or you can do it yourself using some of the props/transforms also mentioned above. But either way, this sentence of yours:

we cannot the parse the logs using universal forwarder ?.

...is not quite orthogonal. Very long story short, data moves through a pipeline in Splunk, from input to parsing to indexing and then to search. This doc page has a nice picture and detail about the topic. Aha, this pic:

The part you need to understand right now is that parsing is almost never done at the input phase by the Universal Forwarder. In fact, the vast majority of the time, fields are not done at index-time either. There are a few index-time fields you must get right: _time, host, source, sourcetype. Almost everything else is done at search-time as part of what we call "schema at read". Splunk is optimized for this style of data processing--it's a feature. Throw it all in, and make sense of it later. Contrast this to systems with a rigid schema which you have to get right at ingest-time.

With me so far?

Now what @matthewmodestino is saying is that to make these fields work as you expect, you have to place the config files in question on your Splunk server or servers, not the UF. If you have a single Splunk server handling all roles, that's it. If you have a distributed environment with n search heads and n indexers, well, it's more complicated and we can get into that.

HTH

[cricketfan5]

@halr9000 @matthewmodestino Thanks for your response and information. Can you also please help me with below query ?

What should I add in the props.conf file to break the log after each comma. Below is the example of my log 2018-05-30 17:20:01,node=abc123,env=te,location=India,type=node_chk,status=Ready it should break after the comma and should write in the new line

2018-05-30 17:20:01 node=abc123 env=te location=India type=node_chk status=Ready

Thanks