Closed kefiras closed 2 months ago
@kefiras this error message alone doesn't tell much about the original problem
2024-04-05 11:29:01 +0000 [warn]: #0 [clusterflow:logging:hosttailer-flow:clusteroutput:logging:syslog] bad chunk is moved to /buffers/backup/worker0/clusterflow_logging_hosttailer-flow_clusteroutput_logging_syslog/61557c20915175b74f5d02915b7386cb.log
logLevel: debug
in fluentd spec)Debug is already enabled
bad chunk
??f?.FsN??time?2024-04-09T10:18:22.776368974Z?message?:Apr 9 10:18:22 aks-prometheus-18130450-vmss000000 kernel: [498058.497065] calico-packet: IN=azve56f4c00502 OUT=azva623c2d61aa MAC=aa:aa:aa:aa:aa:aa:6a:73:f2:79:14:75:08:00 SRC=10.244.3.144 DST=10.244.3.135 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=35709 DF PROTO=TCP SPT=36200 DPT=2020 WINDOW=64240 RES=0x00 SYN URGP=0 ?app?host-tailer?container_image?Lrepo-aks.qa.example.net/example/linux/exm/exm/vendor/fluent/fluent-bit:2.1.8?clustername?aks1kexm1?datacenter?eastus2?env?nonprod?family?logging?mnemonic?exm?hostname?"aks-prometheus-18130450-vmss000000?namespace?logging?pod_id?$2461060d-4eb9-41ec-8fe2-eefcf4bad090?pod_name?filetail-host-tailer-phq7s?service?syslog/ $
I haven't checked receiving side but I doubt anything is send
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions!
We encountered the same error. Is it possible to open this issue again?
@liz-86 can you add some details to this? do you see this error with the latest image versions as well?
Yes, we tested our configuration (much the same as the above mentioned but with tcp transport and not tls) with the latest fluentd image (kube-logging/fluentd-images:v1.16-full). Our ClusterOutput:
apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
name: syslog
namespace: logging
spec:
syslog:
buffer:
flush_thread_count: 16
timekey: 1m
timekey_use_utc: true
timekey_wait: 30s
format:
type: json
host: syslog.example.net
insecure: true
port: 5056
transport: tcp
The created fluentd.conf is the following (from k8s secret loggging-operator-logging-fluentd-app):
<match **>
@type syslog_rfc5424
@id clusterflow:logging:syslog-flow:clusteroutput:logging:syslog-output
host syslog.example.net
insecure true
port 5056
transport tcp
<buffer tag,time>
@type file
chunk_limit_size 8MB
flush_thread_count 16
path /buffers/clusterflow:logging:syslog-flow:clusteroutput:logging:syslog-output.*.buffer
retry_forever true
timekey 1m
timekey_use_utc true
timekey_wait 30s
</buffer>
<format>
@type json
</format>
</match>
Same issue here.
Provider: RKE2 Kubernetes Version: v1.27.12 +rke2r1 Chart: Logging (103.1.1+up4.4.0)
What are your fluentd and fluentbit image versions?
It seems I totally misunderstood the issue originally. I've looked at it once again and it seems that the ruby3 upgrade broke the syslog plugin because of the deprecation and removal of https://blog.saeloun.com/2019/10/07/ruby-2-7-keyword-arguments-redesign/
I've made a change here: https://github.com/pepov/fluent-plugin-syslog_rfc5424/commit/6404b617bc8d5ddd9cf4628cb601cf9b4718e7fb
Then applied on my fork of the fluentd image here: https://github.com/kube-logging/fluentd-images/compare/main...pepov:fluentd-images:main
I didn't have the time to test it with a syslog receiver, could you please give it a try with ghcr.io/pepov/fluentd:v1.16-full
?
Thanks for looking into the issue. I can confirm that with the new image there are no more errors in the fluentd. I need to talk to another team to see if there are getting the desired logs. But it looks good at the moment.
Thanks again!
EDIT: All seems to be working perfectly. The other team's are getting logs. :)
thx for the confirmation, I'm making the PRs to have the fix released asap
The images have been updated with the fix with the 148th build: v1.16-full-build.148 v1.16-full
For logging operator 4.8: v1.16-4.8-full-build.148 v1.16-4.8-full
Describe the bug: Error when using syslog output
Expected behaviour: Logs should be sent to defined syslog cluster output
Steps to reproduce the bug: Configure below resource
Additional context: Fluentd throws errors:
Environment details:
/kind bug