endersonmaia
commented
2 years ago Open Justin-DynamicD opened 3 years ago
endersonmaia
commented
2 years ago endersonmaia
commented
2 years ago The README has references to volumes and volumeMounts as a way to define fluent-bit configuration, I just don't know yet how to mount a volume and get it included as part of the fluet-bit daemon
https://github.com/aws/eks-charts/blob/master/stable/aws-for-fluent-bit/README.md?plain=1#L99-L100
jseiser
commented
2 years ago Just wanted to bump this as well, we are wanting to use this chart to configure Container Insights. AWS Supported pointed us this way like this chart would just be ready but Im not actually seeing how to make this happen.
endersonmaia
commented
2 years ago Just wanted to bump this as well, we are wanting to use this chart to configure Container Insights. AWS Supported pointed us this way like this chart would just
be readybut Im not actually seeing how to make this happen.
It works, @jseiser.
It's just not flexible enough, if you need to make changes to [SERVICE], this issue's topic.
What problem are you facing ?
endersonmaia
commented
2 years ago The README has references to
volumesandvolumeMountsas a way to define fluent-bit configuration, I just don't know yet how to mount a volume and get it included as part of the fluet-bit daemonhttps://github.com/aws/eks-charts/blob/master/stable/aws-for-fluent-bit/README.md?plain=1#L99-L100
you could just override a configmap named fluentbit-config with the configuration you need
it should work, but I don't think it's the better approach for a helm chart; ideally, we could leverage the templates and reuse a base config and extend it as needed
the link I posted in a previous comment shows a better approach, IMHO, having the fluent-bit config section being provided by different template sections, with sane defaults, and allowing you to easily change the sections you need
jseiser
commented
2 years ago @endersonmaia
It works, @jseiser.
It's just not flexible enough, if you need to make changes to
[SERVICE], this issue's topic.What problem are you facing ?
I do not see how to make this chart, reflect what is detailed here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html
endersonmaia
commented
2 years ago @endersonmaia
It works, @jseiser. It's just not flexible enough, if you need to make changes to
[SERVICE], this issue's topic. What problem are you facing ?I do not see how to make this chart, reflect what is detailed here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html
the link you shared is not using the helm chart, that won't work
jseiser
commented
2 years ago @endersonmaia
That link, is detailing how AWS says to configure Fluent-bit for Container Insights. You would think the AWS Chart for Fluent bit, would easily be able to be deployed and configured to match the AWS documentation for deploying fluent bit.
endersonmaia
commented
2 years ago @endersonmaia
That link, is detailing how AWS says to configure Fluent-bit for Container Insights. You would think the AWS Chart for Fluent bit, would easily be able to be deployed and configured to match the AWS documentation for deploying fluent bit.
yep, that should be the case, but there's not only one way to configure a k8s deployment, right ?
and we're talking about AWS, https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws/
jokes aside, if you want to use helm charts, don't follow that link
actually, I'm in the process of using the suggested configMap at this spec and maybe even use this chart instead
jseiser
commented
2 years ago @endersonmaia
actually, I'm in the process of using the suggested configMap at this spec and maybe even use this chart instead
I actually did that last night and it worked fine so far.
serviceAccount:
create: "true"
name: "fluentbit-sa"
annotations:
eks.amazonaws.com/role-arn: ${role_arn}
env:
- name: HOST_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
config:
service: |
[SERVICE]
Flush 5
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
storage.path /var/fluent-bit/state/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M
inputs: |
[INPUT]
Name tail
Tag application.*
Exclude_Path /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
Path /var/log/containers/*.log
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser container_firstline
Parser docker
DB /var/fluent-bit/state/flb_container.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Read_from_Head Off
[INPUT]
Name tail
Tag application.*
Path /var/log/containers/fluent-bit*
Parser docker
DB /var/fluent-bit/state/flb_log.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head Off
[INPUT]
Name tail
Tag application.*
Path /var/log/containers/cloudwatch-agent*
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser cwagent_firstline
Parser docker
DB /var/fluent-bit/state/flb_cwagent.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head Off
filters: |
[FILTER]
Name kubernetes
Match application.*
Kube_URL https://kubernetes.default.svc:443
Kube_Tag_Prefix application.var.log.containers.
Merge_Log On
Merge_Log_Key log_processed
K8S-Logging.Parser On
K8S-Logging.Exclude Off
Labels Off
Annotations Off
[FILTER]
Name modify
Match dataplane.systemd.*
Rename _HOSTNAME hostname
Rename _SYSTEMD_UNIT systemd_unit
Rename MESSAGE message
Remove_regex ^((?!hostname|systemd_unit|message).)*$
[FILTER]
Name aws
Match dataplane.*
imds_version v1
outputs: |
[OUTPUT]
Name cloudwatch_logs
Match application.*
region us-gov-west-1
log_group_name /aws/containerinsights/${cluster_name}/application
log_stream_prefix $${HOST_NAME}-
auto_create_group true
extra_user_agent container-insights
log_retention_days 7
metric_namespace amazon-cloudwatch
customParsers: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
[PARSER]
Name syslog
Format regex
Regex ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
Time_Key time
Time_Format %b %d %H:%M:%S
[PARSER]
Name container_firstline
Format regex
Regex (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
[PARSER]
Name cwagent_firstline
Format regex
Regex (?<log>(?<="log":")\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
extraFile:
dataplane-log.conf: |
[INPUT]
Name systemd
Tag dataplane.systemd.*
Systemd_Filter _SYSTEMD_UNIT=docker.service
Systemd_Filter _SYSTEMD_UNIT=kubelet.service
DB /var/fluent-bit/state/systemd.db
Path /var/log/journal
Read_From_Tail On
[INPUT]
Name tail
Tag dataplane.tail.*
Path /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
Docker_Mode On
Docker_Mode_Flush 5
Docker_Mode_Parser container_firstline
Parser docker
DB /var/fluent-bit/state/flb_dataplane_tail.db
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
Rotate_Wait 30
storage.type filesystem
Read_from_Head Off
[FILTER]
Name modify
Match dataplane.systemd.*
Rename _HOSTNAME hostname
Rename _SYSTEMD_UNIT systemd_unit
Rename MESSAGE message
Remove_regex ^((?!hostname|systemd_unit|message).)*$
[FILTER]
Name aws
Match dataplane.*
imds_version v1
[OUTPUT]
Name cloudwatch_logs
Match dataplane.*
region us-gov-west-1
log_group_name /aws/containerinsights/${cluster_name}/dataplane
log_stream_prefix $${HOST_NAME}-
auto_create_group true
extra_user_agent container-insights
host-log.conf: |
[INPUT]
Name tail
Tag host.dmesg
Path /var/log/dmesg
Parser syslog
DB /var/fluent-bit/state/flb_dmesg.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head Off
[INPUT]
Name tail
Tag host.messages
Path /var/log/messages
Parser syslog
DB /var/fluent-bit/state/flb_messages.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head Off
[INPUT]
Name tail
Tag host.secure
Path /var/log/secure
Parser syslog
DB /var/fluent-bit/state/flb_secure.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
Read_from_Head Off
[FILTER]
Name aws
Match host.*
imds_version v1
[OUTPUT]
Name cloudwatch_logs
Match host.*
region us-gov-west-1
log_group_name /aws/containerinsights/${cluster_name}/host
log_stream_prefix $${HOST_NAME}.
auto_create_group true
extra_user_agent container-insights
daemonSetVolumes:
- name: fluentbitstate
hostPath:
path: /var/fluent-bit/state
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: runlogjournal
hostPath:
path: /run/log/journal
- name: dmesg
hostPath:
path: /var/log/dmesg
daemonSetVolumeMounts:
- name: fluentbitstate
mountPath: /var/fluent-bit/state
- name: varlog
mountPath: /var/log
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: runlogjournal
mountPath: /run/log/journal
readOnly: true
- name: dmesg
mountPath: /var/log/dmesg
readOnly: true
svrviny1324
commented
1 year ago Hi , here I am trying to implement that the logs which are related to same pod i want to send it as single json instead of multiple json into cloudwatch using fluent bit how i can write input,filter,parser? Note: Here iam using configmap same as above. below is my logs present sending it to cloudwatch in this formate { "log": "2023-04-17T12:30:06.484012995Z stdout F ", "kubernetes": { "pod_name": "fca-de-green-kafka-consumer-offset-resetter-6cf9856b8-5jffb", "namespace_name": "", "pod_id": "533d0315-bfe9-4da5-b985-52b8eac07f3b", "host": "", "container_name": "resetter", "docker_id": "1a5d9ab1cc923797f7dadbb83db6de89c996d16455e8f64dd2fe8630bc2f394c", "container_hash": "c53446c49b8a4fe21dcc16b92c4b7baaff194fba37e161b68d342640797aa298", "container_image": "" } } { "log": "2023-04-17T12:30:06.484049686Z stdout F ", "kubernetes": { "pod_name": "fca-de-green-kafka-consumer-offset-resetter-6cf9856b8-5jffb", "namespace_name": "", "pod_id": "533d0315-bfe9-4da5-b985-52b8eac07f3b", "host": "", "container_name": "resetter", "docker_id": "1a5d9ab1cc923797f7dadbb83db6de89c996d16455e8f64dd2fe8630bc2f394c", "container_hash": "c53446c49b8a4fe21dcc16b92c4b7baaff194fba37e161b68d342640797aa298", "container_image": "" } } iam expecting now is { "kubernetes": { "pod_name": "fca-de-green-kafka-consumer-offset-resetter-6cf9856b8-5jffb", "namespace_name": "", "pod_id": "533d0315-bfe9-4da5-b985-52b8eac07f3b", "host": "", "container_name": "resetter", "docker_id": "1a5d9ab1cc923797f7dadbb83db6de89c996d16455e8f64dd2fe8630bc2f394c", "container_hash": "c53446c49b8a4fe21dcc16b92c4b7baaff194fba37e161b68d342640797aa298", "container_image": "" }, { "log": "2023-04-17T12:30:06.484049686Z stdout F " }, { "log": "2023-04-17T12:30:06.484049686Z stdout F " }
The current configmap tempalte is very limited, and doesn't allow any sort of configuration with the [SERVICES] block. For example, it doesn't seem possible to set the below at all:
in the above example, only the parsers file is able to be set. It would be nice that if, in addition to
service.extraParsers, it would be nice if we could define additional values.