izark1
commented
3 years ago Any help please with this?
Closed izark1 closed 3 years ago
izark1
commented
3 years ago Any help please with this?
luckyj5
commented
3 years ago @izark1 Thanks for reporting this issue. Can you please share, how are you deploying this plugin?
izark1
commented
3 years ago Hi @luckyj5 , I've deployed the SCK https://github.com/splunk/splunk-connect-for-kubernetes using Helm 3, I'm interested in the metrics collection part.
I configured my_values.yaml file with the proper configuration of my Splunk environment then run the command below
helm install my-splunk-connect -f my_values.yaml splunk/splunk-connect-for-kubernetes
but I don't see the metrics in Splunk environment, after that, I got the logs as you see in the issue.
more information:
docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/splunk/k8s-metrics 1.1.5 65b48dd511c3 4 days ago 1.03 GB
docker.io/splunk/fluentd-hec 1.2.5 d2b9528d8c03 4 days ago 1.08 GB
docker.io/httpd 2.4 0b932df43057 3 weeks ago 138 MB
docker.io/httpd latest 0b932df43057 3 weeks ago 138 MB
k8s.gcr.io/kube-apiserver v1.21.0 4d217480042e 3 weeks ago 126 MB
k8s.gcr.io/kube-proxy v1.21.0 38ddd85fe90e 3 weeks ago 122 MB
k8s.gcr.io/kube-controller-manager v1.21.0 09708983cc37 3 weeks ago 120 MB
k8s.gcr.io/kube-scheduler v1.21.0 62ad3129eca8 3 weeks ago 50.6 MB
docker.io/weaveworks/weave-npc 2.8.1 7f92d556d4ff 3 months ago 39.3 MB
docker.io/weaveworks/weave-kube 2.8.1 df29c0a4002c 3 months ago 89 MB
k8s.gcr.io/pause 3.4.1 0f8457a4c2ec 3 months ago 683 kB
k8s.gcr.io/coredns/coredns v1.8.0 296a6d5035e2 6 months ago 42.5 MB
k8s.gcr.io/etcd 3.4.13-0 0369cf4303ff 8 months ago 253 MB
gcr.io/cadvisor/cadvisor v0.36.0 7414b6ed960c 10 months ago 184 MB
Can you please elaborate more what you need from me to do?
izark1
commented
3 years ago Hi @luckyj5 , can you assist please ?!
luckyj5
commented
3 years ago Please share your values.yaml or a copy of the running configmap in the cluster. Also, what version and flavor of K8s?
kubectl get cm kubectl describe cm
izark1
commented
3 years ago Hi @luckyj5 , thanks for your response.
kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:30:03Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
this is the output for first command:
NAME DATA AGE kube-root-ca.crt 1 33m my-splunk-connect-splunk-kubernetes-logging 8 14m my-splunk-connect-splunk-kubernetes-metrics 1 14m my-splunk-connect-splunk-kubernetes-metrics-aggregator 1 14m my-splunk-connect-splunk-kubernetes-objects 1 14m
Name: kube-root-ca.crt
Namespace: default
Labels:
-----BEGIN CERTIFICATE----- MIIC5zCCAc+gAwIBAgIBADANBgkqhkiG9w0BAQsFADAVMRMwEQYDVQQDEwprdWJl cm5ldGVzMB4XDTIxMDUxMDIwMTk1OFoXDTMxMDUwODIwMTk1OFowFTETMBEGA1UE AxMKa3ViZXJuZXRlczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALCU RISOjWA87S0NAANKMkkZSCcbXLopk67eLn+KrRCbCJKH14CsN9SmQtyGxfJDOu0B VbTqxj7RzDaAAI0r20SKWHbVsgJEJAZhNh198ZgX6FnrIrSOmISv4RNZkGyXbAyZ y9O2ZxdXpfhS87vI+JZJd0f6Kpax532qNBYhXSJ0WxHaFv1SpNmR8yXCcdmPjUNi k8jmKRgu54uQV7CYlyUEoBR1JkUEl4t5OwdiBv0Z8JdHg2pJVN//gqVHwJuGAbI4 BJ86Z/TwOFFR4WVVFrly8LzXzqjf4bMi2KH2pjg1S2uvkxzslLgxKOLiJOecA/aJ q9DfdZ+WHfvpVD4bPaECAwEAAaNCMEAwDgYDVR0PAQH/BAQDAgKkMA8GA1UdEwEB /wQFMAMBAf8wHQYDVR0OBBYEFLRWmF6qoOCq7J4w1pkNRr40BTx0MA0GCSqGSIb3 DQEBCwUAA4IBAQBJjoTEvtA4RA4nU5Fvxwuvm8nCiPHkPBnRcuflxuPX9/Aw08gQ A6paPIa25qJeYgqH/qJoQWcBbsqihTapXYpola6qkCIKcRvB56Qer8O/e3d7bxRw sF1u+lIrrc49BkjnV+x8AMDymjgQ2wCc2PxaCeGjn25zSf530iQd4aNZR2CvcSd2 a7LOf5pVd2gJsIFsC5YQhb8ZA2o07LodYLOqLJON2wXGhKFolxnAPJMm8i4hWHHb SaYGqy1zTLd+316AtKOfJ1NJJIkBwCtBIw5JuDlPERf01onq7nTW+Dtmqdw/akru FuCtRUnXUPRFcABQE33FDGZ/iOl/dwRtvlXi -----END CERTIFICATE-----
Events:
Name: my-splunk-connect-splunk-kubernetes-logging Namespace: default Labels: app=splunk-kubernetes-logging app.kubernetes.io/managed-by=Helm chart=splunk-kubernetes-logging-1.4.7 heritage=Helm release=my-splunk-connect Annotations: meta.helm.sh/release-name: my-splunk-connect meta.helm.sh/release-namespace: default
@id tail.file.kube-audit @type tail @label @CONCAT tag tail.file.kube:apiserver-audit path /var/log/kube-apiserver-audit.log pos_file /var/log/splunk-fluentd-kube-audit.pos read_from_head true path_key source
@id journald-docker @type systemd @label @CONCAT tag journald.kube:docker path "/run/log/journal" matches [{ "_SYSTEMD_UNIT": "docker.service" }] read_from_head true
@id journald-kubelet @type systemd @label @CONCAT tag journald.kube:kubelet path "/run/log/journal" matches [{ "_SYSTEMD_UNIT": "kubelet.service" }] read_from_head true
<label @CONCAT>
<filter tail.containers.var.log.containers.dns-controllerdns-controller.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-dnssidecar.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-dnsdnsmasq.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-apiserverkube-apiserver.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-controller-managerkube-controller-manager.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-dns-autoscalerautoscaler.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-proxykube-proxy.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-schedulerkube-scheduler.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true <filter tail.containers.var.log.containers.kube-dnskubedns.log> @type concat key log timeout_label @SPLUNK stream_identity_key stream multiline_start_regexp /^\w[0-1]\d[0-3]\d/ flush_interval 5 separator "" use_first_timestamp true
<match **> @type relabel @label @SPLUNK <label @SPLUNK>
<filter tail.containers.**> @type grep
<filter tail.containers.*> @type kubernetes_metadata annotation_match [ "." ] de_dot false watch true cache_ttl 3600 <filter tail.containers.**> @type record_transformer enable_ruby
<filter tail.containers.**>
@type grep
<exclude>
key exclude_list
pattern /^true$/
</exclude><filter journald.**> @type jq_transformer jq '.record.source = "/run/log/journal/" + .record.source | .record.sourcetype = (.tag | ltrimstr("journald.")) | .record.cluster_name = "cluster_name" | .record.splunk_index = "k8s" |.record'
<filter tail.file.**> @type jq_transformer jq '.record.sourcetype = (.tag | ltrimstr("tail.file.")) | .record.cluster_name = "cluster_name" | .record.index = "k8s" | .record'
<match **> @type splunk_hec protocol http hec_host "10.10.1.100" hec_port 8088 hec_token "#{ENV['SPLUNK_HEC_TOKEN']}" index_key splunk_index insecure_ssl true host "#{ENV['K8S_NODE_NAME']}" source_key source sourcetype_key sourcetype
app_name splunk-kubernetes-logging
app_version 1.4.7
<buffer>
@type memory
chunk_limit_records 100000
chunk_limit_size 20m
flush_interval 5s
flush_thread_count 1
overflow_action block
retry_max_times 5
retry_type periodic
total_limit_size 600m
</buffer>
<format monitor_agent>
@type json
</format>
<format>
# we just want to keep the raw logs, not the structure created by docker or journald
@type single_value
message_key log
add_newline false
</format>@type prometheus
@type forward
@type prometheus_monitor
@type prometheus_output_monitor
#
#
@id containers.log @type tail @label @CONCAT tag tail.containers. path /var/log/containers/.log pos_file /var/log/splunk-fluentd-containers.log.pos path_key source read_from_head true refresh_interval 60
Name: my-splunk-connect-splunk-kubernetes-metrics Namespace: default Labels: app=splunk-kubernetes-metrics app.kubernetes.io/managed-by=Helm chart=splunk-kubernetes-metrics-1.4.7 heritage=Helm release=my-splunk-connect Annotations: meta.helm.sh/release-name: my-splunk-connect meta.helm.sh/release-namespace: default
@type kubernetes_metrics tag kube.* node_name "#{ENV['NODE_NAME']}" kubelet_port 10248 use_rest_client_ssl false insecure_ssl true cluster_name cluster_name interval 15s
<filter kube.node.**> @type record_modifier
<filter kube.pod.**> @type record_modifier
<filter kube.sys-container.**> @type record_modifier
<filter kube.container.**> @type record_modifier
<match kube.**> @type splunk_hec data_type metric metric_name_key metric_name metric_value_key value protocol http hec_host "10.10.1.100" hec_port 8088 hec_token "#{ENV['SPLUNK_HEC_TOKEN']}" host "#{ENV['NODE_NAME']}" index em_metrics source ${tag} insecure_ssl true app_name splunk-kubernetes-metrics app_version 1.4.7
Events:
Name: my-splunk-connect-splunk-kubernetes-metrics-aggregator Namespace: default Labels: app=splunk-kubernetes-metrics app.kubernetes.io/managed-by=Helm chart=splunk-kubernetes-metrics-1.4.7 heritage=Helm release=my-splunk-connect Annotations: meta.helm.sh/release-name: my-splunk-connect meta.helm.sh/release-namespace: default
@type kubernetes_metrics_aggregator tag kube.* interval 15s
<filter kube.cluster.**> @type record_modifier
<filter kube.namespace.**> @type record_modifier
<filter kube.node.**> @type record_modifier
<filter kube.pod.**> @type record_modifier
<filter kube.sys-container.**> @type record_modifier
<filter kube.container.**> @type record_modifier
<match kube.**> @type splunk_hec data_type metric metric_name_key metric_name metric_value_key value protocol http hec_host "10.10.1.100" hec_port 8088 hec_token "#{ENV['SPLUNK_HEC_TOKEN']}" host "#{ENV['NODE_NAME']}" index em_metrics source source insecure_ssl true app_name splunk-kubernetes-metrics app_version 1.4.7
Events:
Name: my-splunk-connect-splunk-kubernetes-objects Namespace: default Labels: app=splunk-kubernetes-objects app.kubernetes.io/managed-by=Helm chart=splunk-kubernetes-objects-1.4.7 heritage=Helm release=my-splunk-connect Annotations: meta.helm.sh/release-name: my-splunk-connect meta.helm.sh/release-namespace: default
@type kubernetes_objects tag kube.objects.* api_version "v1" insecure_ssl false
<filter kube.**> @type jq_transformer
gsub jq filter, it becomes '\\'.jq '.record.source = "namespace:(env.MY_NAMESPACE)/pod:(env.MY_POD_NAME)" | .record.sourcetype = (.tag | gsub("\\."; ":")) | .record'
<filter kube.**> @type jq_transformer jq '.record.cluster_name = "cluster_name" | .record'
<match kube.**> @type splunk_hec protocol http hec_host "10.10.1.100" hec_port 8088 hec_token "#{ENV['SPLUNK_HEC_TOKEN']}" host "#{ENV['NODE_NAME']}" source_key source sourcetype_key sourcetype index k8s insecure_ssl true
app_name splunk-kubernetes-objects app_version 1.4.7
Events:
izark1
commented
3 years ago @luckyj5 : values.yaml
global: logLevel: info splunk: hec:
host: 10.10.1.100
# port to HEC, optional, default 8088
port: 8088
# token is required and should be provided by user
token: ad4df02b-d141-4297-b890-24ae31745e47
# protocol has two options: "http" and "https", default is "https"
protocol: http
# indexName tells which index to use, this is optional. If it's not present, will use "main".
indexName: k8s
# insecureSSL is a boolean, it indicates should it allow insecure SSL connection (when protocol is "https"). Default is false.
insecureSSL: true
# The PEM-format CA certificate for this client.
# NOTE: The content of the certificate itself should be used here, not the file path.
# The certificate will be stored as a secret in kubernetes.
clientCert:
# The private key for this client.
# NOTE: The content of the key itself should be used here, not the file path.
# The key will be stored as a secret in kubernetes.
clientKey:
# The PEM-format CA certificate file.
# NOTE: The content of the file itself should be used here, not the file path.
# The file will be stored as a secret in kubernetes.
caFile:
# For object and metrics
indexRouting:kubernetes:
clusterName: "cluster_name"prometheus_enabled: true monitoring_agent_enabled: true
splunk-kubernetes-logging chart to a kubernetessplunk-kubernetes-logging: enabled: true
logLevel:
fluentd:
path: /var/log/containers/*.log
# paths of logfiles to exclude. object type is array as per fluentd specification:
# https://docs.fluentd.org/input/tail#exclude_path
exclude_path:
# - /var/log/containers/kube-svc-redirect*.log
# - /var/log/containers/tiller*.log
# - /var/log/containers/*_kube-system_*.log (to exclude `kube-system` namespace)containers:
path: /var/log
# Final volume destination of container log symlinks
pathDest: /var/lib/docker/containers
# Log format type, "json" or "cri"
logFormatType: json
# Specify the logFormat for "cri" logFormatType - provide time format
# For example "%Y-%m-%dT%H:%M:%S.%N%:z" for openshift, "%Y-%m-%dT%H:%M:%S.%NZ" for IBM IKS
# Default for "cri": "%Y-%m-%dT%H:%M:%S.%N%:z"
logFormat:
# Specify the interval of refreshing the list of watch file.
refreshInterval:k8sMetadata:
podLabels:
- app
- k8s-app
- release
watch: true
cache_ttl: 3600sourcetypePrefix: "kube"
rbac:
# This should be set to `false` if either:
# a) RBAC is not enabled in the cluster, or
# b) you want to create RBAC resources by yourself.
create: true
# If you are on OpenShift and you want to run the a privileged pod
# you need to have a ClusterRoleBinding for the system:openshift:scc:privileged
# ClusterRole. Set to `true` to create the ClusterRoleBinding resource
# for the ServiceAccount.
openshiftPrivilegedSccBinding: falseserviceAccount:
create: true
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name:
# This flag specifies if the user wants to use a secret for creating the serviceAccount,
# which will be used to get the images from a private registry
usePullSecrets: falsepodSecurityPolicy:
# This should be set to `false` if either:
# a) Pod Security Policies is not enabled in the cluster, or
# b) you want to create Pod Security Policy resources by yourself.
create: false
# Specifies whether AppArmor profile should be applied.
# if set to true, this will add two annotations to PodSecurityPolicy:
# apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
# apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
# set to false if AppArmor is not available
apparmor_security: true
# apiGroup can be set to "extensions" for Kubernetes < 1.10.
apiGroup: policysplunk:
hec:
# host is required and should be provided by user
host:
# port to HEC, optional, default 8088
port:
# token is required and should be provided by user
token:
# protocol has two options: "http" and "https", default is "https"
protocol:
# indexName tells which index to use, this is optional. If it's not present, will use "main".
indexName:
# insecureSSL is a boolean, it indicates should it allow insecure SSL connection (when protocol is "https"). Default is false.
insecureSSL:
# The PEM-format CA certificate for this client.
# NOTE: The content of the certificate itself should be used here, not the file path.
# The certificate will be stored as a secret in kubernetes.
clientCert:
# The private key for this client.
# NOTE: The content of the key itself should be used here, not the file path.
# The key will be stored as a secret in kubernetes.
clientKey:
# The PEM-format CA certificate file.
# NOTE: The content of the file itself should be used here, not the file path.
# The file will be stored as a secret in kubernetes.
caFile:
# Configurations for Ingest API
ingest_api:
# serviceClientIdentifier is a string, the client identifier is used to make requests to the ingest API with authorization.
serviceClientIdentifier:
# serviceClientSecretKey is a string, the client identifier is used to make requests to the ingest API with authorization.
serviceClientSecretKey:
# tokenEndpoint is a string, it indicates which endpoint should be used to get the authorization token used to make requests to the ingest API.
tokenEndpoint:
# ingestAuthHost is a string, it indicates which url/hostname should be used to make token auth requests to the ingest API.
ingestAuthHost:
# ingestAPIHost is a string, it indicates which url/hostname should be used to make requests to the ingest API.
ingestAPIHost:
# tenant is a string, it indicates which tenant should be used to make requests to the ingest API.
tenant:
# eventsEndpoint is a string, it indicates which endpoint should be used to make requests to the ingest API.
eventsEndpoint:
# debugIngestAPI is a boolean, it indicates whether user wants to debug requests and responses to ingest API. Default is false.
debugIngestAPI:secret: create: true name:
journalLogPath: /run/log/journal
#
# charEncodingUtf8: false
logs defines the source of logs, multiline support, and their sourcetypes.#
#
#
journald logs, unit is required for filtering using _SYSTEMD_UNIT, example:#
file logs, path is required for specifying where is the log files. Log files are expected in /var/log, example:#
container logs, pod name is required. You can also provide the container name, if it's not provided, the name of this source will be used as the container name:#
#
timestampExtraction defines how to extract timestamp from logs. This only works for file source.timestampExtraction you need to define both:regexp: the Regular Expression used to find the timestamp from a log entry.time named group. E.g.format: a format string defintes how to parse the timestamp, e.g. "%Y-%m-%d %H:%M:%S".#
multiline options provide basic multiline support. Two options:firstline: a Regular Expression used to detect the first line of a multiline log.flushInterval: The number of seconds after which the last received event log will be flushed, default value: 5s.#
sourcetype field.sourcetype is not defined, name will be used.#
logs: docker: from: journald: unit: docker.service timestampExtraction: regexp: time="(?
image:
registry: docker.io
# The name of the image to pull
name: splunk/fluentd-hec
# The tag of the image to pull
tag: 1.2.5
# The policy that specifies when the user wants the images to be pulled
pullPolicy: IfNotPresent
# Indicates if the image should be pulled using authentication from a secret
usePullSecret: false
# The name of the pull secret to attach to the respective serviceaccount used to pull the image
pullsecretName:environmentVar:
resources:
# cpu: 100m
# memory: 200Mi
requests:
cpu: 100m
memory: 200Miresources.limits.memory is set,buffer: "@type": memory total_limit_size: 600m chunk_limit_size: 20m chunk_limit_records: 100000 flush_interval: 5s flush_thread_count: 1 overflow_action: block retry_max_times: 5 retry_type: periodic
sendAllMetadata: false
tolerations:
key: node-role.kubernetes.io/master effect: NoSchedule
nodeSelector: beta.kubernetes.io/os: linux
affinity: {}
priorityClassName:
kubernetes:
clusterName:
securityContext: false
customMetadata:
customMetadataAnnotations:
customFilters defines the custom filters to be used.#
#
#
#
#
#
customFilters: {}
#
#
indexFields defines the fields from the fluentd record to be indexed.#
indexFields: []
splunk-kubernetes-objects chart to a kubernetessplunk-kubernetes-objects: enabled: true
logLevel:
rbac:
# This should be set to `false` if either:
# a) RBAC is not enabled in the cluster, or
# b) you want to create RBAC resources by yourself.
create: trueserviceAccount:
create: true
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name:
# This flag specifies if the user wants to use a secret for creating the serviceAccount,
# which will be used to get the images from a private registry
usePullSecrets: falsepodSecurityPolicy:
# This should be set to `false` if either:
# a) Pod Security Policies is not enabled in the cluster, or
# b) you want to create Pod Security Policy resources by yourself.
create: false
# Specifies whether AppArmor profile should be applied.
# if set to true, this will add two annotations to PodSecurityPolicy:
# apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
# apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
# set to false if AppArmor is not available
apparmor_security: true
# apiGroup can be set to "extensions" for Kubernetes < 1.10.
apiGroup: policykubernetes:
url:
# if insecureSSL is set to true, insecure HTTPS API call is allowed, default false
insecureSSL: false
# Path to the certificate file for this client.
clientCert:
# Path to the private key file for this client.
clientKey:
# Path to the CA file.
caFile:
# Path to the file contains the API token. By default it reads from the file "token" in the `secret_dir`.
bearerTokenFile:
# Path of the location where pod's service account's credentials are stored. Usually you don't need to care about this config, the default value should work in most cases.
secretDir:
# The cluster name used to tag cluster metrics from the aggregator. Default is cluster_name
clusterName:
# Add privileged access to containers for openshift compatibility
openshift: false#
#
objectDefinition has the following fields:pods, namespaces.daemonsets,_, so daemonsets becomes daemon_sets.#
objects: core: v1:
name: events mode: watch
checkpointFile:
name: kubernetes-objects.pos
#
volume:
splunk: hec:
host:
token:
protocol:
indexName:
insecureSSL:
clientCert:
clientKey:
caFile:
caPath:
indexRouting:
secret: create: true name:
image:
registry: docker.io
name: splunk/kube-objects
tag: 1.1.5
pullPolicy: IfNotPresent
usePullSecret: false
pullSecretName:
environmentVar:
resources:
requests: cpu: 100m memory: 200Mi
resources.limits.memory is set,buffer: "@type": memory total_limit_size: 600m chunk_limit_size: 20m chunk_limit_records: 10000 flush_interval: 5s flush_thread_count: 1 overflow_action: block retry_max_times: 5 retry_type: periodic
nodeSelector: beta.kubernetes.io/os: linux
tolerations: []
affinity: {}
customFilters defines the custom filters to be used.#
#
#
#
#
#
customFilters: {} #
#
#
indexFields defines the fields from the fluentd record to be indexed.#
indexFields: []
splunk-kubernetes-metrics chart to a kubernetessplunk-kubernetes-metrics: enabled: true
logLevel:
rbac:
# This should be set to `false` if either:
# a) RBAC is not enabled in the cluster, or
# b) you want to create RBAC resources by yourself.
create: trueserviceAccount:
create: true
# The name of the ServiceAccount to use.
# If not set and create is true, a name is generated using the fullname template
name:
# This flag specifies if the user wants to use a secret for creating the serviceAccount,
# which will be used to get the images from a private registry
usePullSecrets: falsepodSecurityPolicy:
# This should be set to `false` if either:
# a) Pod Security Policies is not enabled in the cluster, or
# b) you want to create Pod Security Policy resources by yourself.
create: false
# Specifies whether AppArmor profile should be applied.
# if set to true, this will add two annotations to PodSecurityPolicy:
# apparmor.security.beta.kubernetes.io/allowedProfileNames: 'runtime/default'
# apparmor.security.beta.kubernetes.io/defaultProfileName: 'runtime/default'
# set to false if AppArmor is not available
apparmor_security: true
# apiGroup can be set to "extensions" for Kubernetes < 1.10.
apiGroup: policysplunk:
hec:
# hostname/ip of HEC, REQUIRED.
host:
# port to HEC, OPTIONAL. Default value: 8088
port:
# the HEC token, REQUIRED.
token:
# protocol has two options: "http" and "https". Default value: "https"
protocol:
# indexName tells which index to use, OPTIONAL. If it's not present, will use "main".
indexName: em_metrics
# insecureSSL is a boolean, it indicates should it allow insecure SSL connection (when protocol is "https"). Default value: false
insecureSSL:
# The PEM-format CA certificate for this client.
# NOTE: The content of the certificate itself should be used here, not the file path.
# The certificate will be stored as a secret in kubernetes.
clientCert:
# The private key for this client.
# NOTE: The content of the key itself should be used here, not the file path.
# The key will be stored as a secret in kubernetes.
clientKey:
# The PEM-format CA certificate file.
# NOTE: The content of the file itself should be used here, not the file path.
# The file will be stored as a secret in kubernetes.
caFile:secret: create: true name:
image:
registry: docker.io
# The name of the image to pull
name: splunk/k8s-metrics
# The tag of the image to pull
tag: 1.1.5
# The policy that specifies when the user wants the images to be pulled
pullPolicy: IfNotPresent
# Indicates if the image should be pulled using authentication from a secret
usePullSecret: false
# The name of the pull secret to attach to the respective serviceaccount used to pull the image
pullsecretName:imageAgg:
registry: docker.io
# The name of the image to pull
name: splunk/k8s-metrics-aggr
# The tag of the image to pull
tag: 1.1.5
# The policy that specifies when the user wants the images to be pulled
pullPolicy: IfNotPresent
# Indicates if the image should be pulled using authentication from a secret
usePullSecret: false
# The name of the pull secret to attach to the respective serviceaccount used to pull the image
pullsecretName:environmentVar:
environmentVarAgg:
resources: fluent: limits: cpu: 200m memory: 300Mi requests: cpu: 200m memory: 300Mi
resources.sidecar.limits.memory is set,buffer: "@type": memory total_limit_size: 400m chunk_limit_size: 10m chunk_limit_records: 10000 flush_interval: 5s flush_thread_count: 1 overflow_action: block retry_max_times: 5 retry_type: periodic
aggregatorBuffer: "@type": memory total_limit_size: 400m chunk_limit_size: 10m chunk_limit_records: 10000 flush_interval: 5s flush_thread_count: 1 overflow_action: block retry_max_times: 5 retry_type: periodic
metricsInterval: 15s
nodeSelector: beta.kubernetes.io/os: linux
tolerations:
key: node-role.kubernetes.io/master effect: NoSchedule
aggregatorTolerations: {}
priorityClassName:
affinity: {}
kubernetes:
kubeletAddress:
kubeletPort: 10248
kubeletPortAggregator:
useRestClientSSL: false
insecureSSL: true
caFile:
secret_dir.bearerTokenFile:
secretDir:
clusterName:
openshift: false
customFilters defines the custom filters to be used.#
#
#
#
#
#
customFilters: {}
izark1
commented
3 years ago Hi @luckyj5 ,
commands output attached: get_cm.txt describe_cm.txt
values.yaml: note: renamed the extension to .txt my_values.yaml.txt
izark1
commented
3 years ago I want to add here, that even direct call to the endpoint results in Error 404 not found.
[root@docker1 k8s]# curl http://10.10.1.80:10248/stats/summary
404 page not found
[root@docker1 k8s]#
izark1
commented
3 years ago kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:30:03Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:25:06Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
rockb1017
commented
3 years ago Hello, it seems you need to enable the metrics endpoint. Could you try steps in this thread and let us know? https://github.com/splunk/splunk-connect-for-kubernetes/issues/505#issuecomment-754705688
izark1
commented
3 years ago Hi @rockb1017 , the case is the --enable-cadvisor-json-endpoints=true parameter can't be added to the /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf file nor in /var/lib/kubelet/kubeadm-flags.env file.
once add this parameter, so the complete ARG is Environment="KUBELET_KUBECONFIG_ARGS=--enable-cadvisor-json-endpoints=true --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" and attempt to do systemctl restart kubelet.service it immediatly fails..
Do I add it correctly?!
rockb1017
commented
3 years ago what version of kubelet are you using?
rockb1017
commented
3 years ago if you are using version that this option is removed, it won't work for that version. our metrics collector assumes this endpoint to collect metrics.
izark1
commented
3 years ago [root@docker1 k8s]# kubelet --version
Kubernetes v1.21.0
izark1
commented
3 years ago So if it's a version compatibility issue, then what is max supported version of Kubernetes for SCK?
rockb1017
commented
3 years ago the option is only available up to 1.20 https://v1-20.docs.kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
izark1
commented
3 years ago Thanks all for the clarifying the reason.
Hi, I'm deploying this plugin and run it in my environment, but I get this these errors in the my splunk-fluentd-k8s-metrics container:
2021-05-02 23:08:20 +0000 [error]: #0 Unexpected error raised. Stopping the timer. title=:cadvisor_metric_scraper error_class=RestClient::NotFound error="404 Not Found" 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in
exception_with_response' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:inreturn!' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:836:inprocess_result' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:inblock in transmit' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/ruby/net/http.rb:933:instart' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:intransmit' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:inexecute' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:inexecute' 2021-05-02 23:08:20 +0000 [error]: #0 /opt/app-root/src/gem/fluent-plugin-kubernetes-metrics-1.1.5/lib/fluent/plugin/in_kubernetes_metrics.rb:660:inscrape_cadvisor_metrics' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin_helper/timer.rb:80:inon_timer' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:inrun_once' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/cool.io-1.7.1/lib/cool.io/loop.rb:88:inrun' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin_helper/event_loop.rb:93:inblock in start' 2021-05-02 23:08:20 +0000 [error]: #0 /usr/share/gems/gems/fluentd-1.11.5/lib/fluent/plugin_helper/thread.rb:78:inblock in thread_create' 2021-05-02 23:08:20 +0000 [error]: #0 Timer detached. title=:cadvisor_metric_scraperany help please with that ?!