splunk / splunk-connect-for-kubernetes

Helm charts associated with kubernetes plug-ins
Apache License 2.0
343 stars 270 forks source link

Splunk Connect NameSpace indexer #314

Closed Bruslan closed 4 years ago

Bruslan commented 4 years ago

Hi guys, i try to push my Namespaced Loggs into an own index. First i set the annotation in the Namespace with my index Name. According to the docu it should work right away, but it dont. The problem could be the hardcoded value in the value.yaml during the installation? Should i let the field index = *** at value.yaml empty?

Thanks.

Best Bruslan

Bruslan commented 4 years ago

Ok i noticed, if i set the splunk.com/sourcetype annotation to something else, still the default sourcetype is displayed in Splunk. Any ideas why fluend do not read the annotation properly? How could i debug this error?

matthewmodestino commented 4 years ago

@Bruslan Hi! Can you gather kubectl -n yourNamespace get logs | less ?

Please review it as it shows your fluentd config and share any redacted info relevant to the output.conf specifically.

You can also review the first couple hundred lines of logs to see if there are errors and such.

rockb1017 commented 4 years ago

Hello @Bruslan this feature is added to develop branch only. Could you tell me if you are cloning this repo and installing helm chart from it?

Bruslan commented 4 years ago

Hi @rockb1017 , thank you for the answer. We installed the 1.3.0 release version, cause it was documented in the README as a release feature. Could you tell me when you planed to release this feature? ("annotation indexer") We need a stable version on our cluster.

matthewmodestino commented 4 years ago

@rockb1017 i noticed the repo defaults to displaying dev branch by default. Maybe it’s a GitHub setting that defaults to displaying master??

Bruslan commented 4 years ago

Okay now i see, the default branch is on Dev, just noticed it after @matthewmodestino mentioned it.

chaitanyaphalak commented 4 years ago

@Bruslan develop is the default branch, if you want to refer the code for 1.3.0 release go here - https://github.com/splunk/splunk-connect-for-kubernetes/tree/1.3.0

Bruslan commented 4 years ago

@chaitanyaphalak I pulled the DEV branch and installed the charts via helm. The fluentd-logging containers are crash-looping on Openshift 3.11 with this error:

` 2020-01-28 17:02:33 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/fluent.conf" 2020-01-28 17:02:33 +0000 [error]: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Unknown filter plugin 'kubernetes_metadata'. Run 'gem search -rd fluent-plugin' to find plugins"

`

Any Ideas what that could be? This annotaionRouting feature would be very nice.

rockb1017 commented 4 years ago

Could you reploy with this value changed to 1.2.0 ? https://github.com/splunk/splunk-connect-for-kubernetes/blob/ec2b397fe05f4ff43850896e5aab323cb4721917/helm-chart/splunk-connect-for-kubernetes/values.yaml#L323

Bruslan commented 4 years ago

@rockb1017 You are the best. The newer image did the work. The annotationRouting is finally working! The question here would be, do you consider the dev branch as stable right now? Thank you for your help.

chaitanyaphalak commented 4 years ago

@Bruslan our dev branch is stable, but only our release/master branches are recommended for production usage.

Mr-iX commented 4 years ago

It seems that if you change the index annotation, you have to restart the Pod, so that the logs get written in the new index. Is that so normal and desirable?

rockb1017 commented 4 years ago

@Mr-iX it should rake effect immediately. please use latest version. (chart 1.4.3). If you still have an issue, please open a new issue. closing this.