SidGrundfos
commented
4 years ago This is how I deployed
helm install humio humio/humio-helm-charts --namespace logging --set humio-fluentbit.token=*** -f humio-agent.yaml
`
humio-agent.yaml
humio-fluentbit:
enabled: true
humioHostname: cloud.humio.com
humioRepoName: gic
es:
tls: true
tls_verify: true
autodiscovery: false
nameOverride: ""
fullnameOverride: ""
tokenSecretName: ""
tokenSecretKeyName: token
resources:
limits:
cpu: 100m
memory: 128Mi
requests:
cpu: 100m
memory: 128Mi
serviceConfig: |-
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
inputConfig: |-
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag kube.*
Refresh_Interval 5
Mem_Buf_Limit 20MB
Skip_Long_Lines On
[INPUT]
Name tail
Path /var/log/containers/*.log
Parser docker
Tag audit.*
Refresh_Interval 5
Mem_Buf_Limit 20MB
Skip_Long_Lines On
filterConfig: |-
[FILTER]
Name kubernetes
Match kube.*
Kube_Tag_Prefix kube.var.log.containers.
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_under log_processed
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_under Properties
[FILTER]
Name grep
Match kube.*
Exclude Category Audit
[FILTER]
Name kubernetes
Match audit.*
Kube_Tag_Prefix kube.var.log.containers.
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Merge_Log On
Merge_Log_Key log_audits
K8S-Logging.Parser On
K8S-Logging.Exclude On
[FILTER]
Name nest
Match audit.*
Operation lift
Nested_under log_audits
[FILTER]
Name nest
Match audit.*
Operation lift
Nested_under Properties
[FILTER]
Name grep
Match audit.*
Regex Category Audit
outputConfig: |-
[OUTPUT]
Name es
Match kube.*
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
tls ${FLUENT_ELASTICSEARCH_TLS}
tls.verify ${FLUENT_ELASTICSEARCH_TLS_VERIFY}
HTTP_User ${HUMIO_REPO_NAME}
HTTP_Passwd ${HUMIO_INGEST_TOKEN}
Logstash_Format On
Retry_Limit False
Type flb_type
Time_Key @timestamp
Replace_Dots On
Logstash_Prefix FluentBitHelmChart
Buffer_Size 20MB
[OUTPUT]
Name es
Match audit.*
Host ${FLUENT_ELASTICSEARCH_HOST}
Port ${FLUENT_ELASTICSEARCH_PORT}
tls ${FLUENT_ELASTICSEARCH_TLS}
tls.verify ${FLUENT_ELASTICSEARCH_TLS_VERIFY}
HTTP_User ***
HTTP_Passwd None
Logstash_Format On
Retry_Limit False
Type flb_type
Time_Key @timestamp
Replace_Dots On
Logstash_Prefix FluentBitHelmChart
Buffer_Size 20MB
parserConfig: |-
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep Off
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
customFluentBitConfig:
custom.conf: |-
nodeSelector: {}
tolerations: []
affinity: {} `
SaaldjorMike
keatinle
Hello Team, I have this deployed to dev, Test, QA and prod environments and it was running fine till few days ago. suddenly in one of the environments (Test) the pods started restarting and failing continuously with error