frenchviking
commented
3 years ago Hello,
I'm having the same issue.
fluent bit v1.7 deployed as a daemonset on kubernetes.
maxmind database is hosted on a persistent volume and mounted to the pods. The DB is managed and updated by Maxmind's geoipupdate tool, also deployed on the cluster (image maxmindinc/geoipupdate). Mounting the volume on a pod with a terminal I can confirm that the DB files are availables.
Here is the configmap storing the fluent config.
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: kube-logging
labels:
k8s-app: fluent-bit
data:
# Configuration files: server, input, filters and output
# ======================================================
fluent-bit.conf: |
[SERVICE]
Flush 1
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
@INCLUDE input-kubernetes.conf
@INCLUDE input-traefikee.conf
@INCLUDE filter-geoip.conf
@INCLUDE filter-kubernetes.conf
@INCLUDE filter-traefikee.conf
@INCLUDE output-elasticsearch.conf
input-kubernetes.conf: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Exclude_Path /var/log/containers/kube-prod-controller*.log,/var/log/containers/kube-prod-proxy*.log,
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
input-traefikee.conf: |
[INPUT]
Name tail
Tag traefikee.*
Path /var/log/containers/kube-prod-controller*.log,/var/log/containers/kube-prod-proxy*.log
Parser docker
DB /var/log/flb_traefik.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 1
filter-kubernetes.conf: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
filter-traefikee.conf: |
[FILTER]
Name kubernetes
Match traefikee.*
Merge_Log On
Merge_Log_Key traefik
filter-geoip.conf: |
[FILTER]
Name geoip2
Match traefikee.*
Database /geoip/GeoLite2-City.mmdb
Lookup_key traefik.ClientHost
Record traefik.country ClientHost %{country.names.en}
Record traefik.isocode ClientHost %{country.iso_code}
Record traefik.latitude ClientHost %{location.latitude}
Record traefik.longitude ClientHost %{location.longitude}
output-elasticsearch.conf: |
[OUTPUT]
Name es
Match *
Host eshost
Port 19200
HTTP_User ${FLUENT_ELASTICSEARCH_USER}
HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
tls on
tls.verify off
Index kube-
Logstash_prefix kube
Logstash_Format On
Replace_Dots On
Retry_Limit False
parsers.conf: |
[PARSER]
Name apache
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache2
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name apache_error
Format regex
Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name syslog
Format regex
Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%SThe daemonset manifest:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: kube-logging
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
updateStrategy:
rollingUpdate:
maxUnavailable: 6
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "2020"
prometheus.io/path: /api/v1/metrics/prometheus
spec:
containers:
- name: fluent-bit
image: fluent/fluent-bit:1.7
imagePullPolicy: Always
ports:
- containerPort: 2020
env:
#ELASTIC CONFIG
- name: FLUENT_ELASTICSEARCH_USER
valueFrom:
secretKeyRef:
name: fluentd-elastic-secret
key: username
- name: FLUENT_ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: fluentd-elastic-secret
key: password
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
**- name: geoip
mountPath: /geoip**
terminationGracePeriodSeconds: 0
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
**- name: geoip
persistentVolumeClaim:
claimName: geoip
readOnly: true**
serviceAccountName: fluent-bit
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
- operator: "Exists"
effect: "NoExecute"
- operator: "Exists"
effect: "NoSchedule"
Volume manifest:
apiVersion: v1
kind: PersistentVolume
metadata:
name: geoip
labels:
type: azure-file
namespace: kube-logging
spec:
storageClassName: azurefile
capacity:
storage: 1G
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
azureFile:
secretName: azure-storage-geoip-secret
secretNamespace: kube-logging
shareName: geoipdb
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: geoip
namespace: kube-logging
spec:
selector:
matchLabels:
name: geoip
resources:
requests:
storage: 1G
accessModes:
- ReadWriteOnce
storageClassName: azurefile
volumeName: geoip
Not shown on the screenshot but the field traefik.ClientHost used in the config filter-geoip.conf Lookup_key contains the client IP (X-Forwarded-For/X-Real-Ip)
agup006
proffalken
github-actions[bot]
Owemeone
Bug Report
Describe the bug When the GeoIP2 filter is enabled, I cannot get fluent-bit to augment my logs in the way I would expect.
To Reproduce
Configure Fluent-bit as follows:
Expected behavior
All log data should be augmented with
fb_<field name>with the values populated as appropriate.Screenshots
Note: Fields circled in red are from Fluent-bit, fields circled in green are from
fluentdwith the GeoIP Filter. Both solutions point to the sameGEOIP2city.mmdbfile, and use the filteredsourcefield as the IP address to lookup against.Your Environment
Additional context Trying to get the same functionality from Fluent-bit that I currently get from Fluentd to remove Fluentd from my logging stack!