Closed ctrader321 closed 1 year ago
As an aside, are we able to configure image proxy anywhere via the values? I noticed the daemonset that is applied only has image referenced like so:
Image: fluent/fluent-bit:1.9.5
Hey @ctrader321 sorry, but we can't help with supporting old versions and I wasn't able to reproduce with logging operator 4.0.0.
As for your image proxy question: what kind of proxy are you referring to?
closing this for now, feel free to reopen if you still have questions or want to clarify
@pepov I have an same issue. I want to learn logging operator, so I write a demo, but when I apply flow and output config yaml, same error will shows.
helm version: v3.13.2 logging-operator version: 4.4.2(Image: ghcr.io/kube-logging/logging-operator:4.4.2) k3s version: v1.27.6+k3s1
logging yaml:
apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
name: demo-log-collector
namespace: logging
spec:
controlNamespace: logging
watchNamespaces: ["dev"]
fluentd:
disablePvc: true
fluentbitAgent yaml:
apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
name: demo-log-collector
namespace: logging
spec: {}
flow and output yaml:
apiVersion: v1
kind: Secret
metadata:
name: elastic-pwd
namespace: dev
type: Opaque
data:
es-password: xxxxxxxxxxxxxxxxxxxxxxx
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Output
metadata:
name: es-output
namespace: dev
spec:
elasticsearch:
host: xxxxxxx
port: xxxxx
scheme: https
ssl_verify: false
ssl_version: TLSv1_2
user: elastic
password:
valueFrom:
secretKeyRef:
name: "elastic-pwd"
key: "es-password"
buffer:
timekey: 1m
timekey_wait: 30s
timekey_use_utc: true
---
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: exchange
namespace: dev
spec:
filters:
- stdout: {}
localOutputRefs:
- es-output
match:
- select:
labels:
appName: logdemo
demo application yaml, it is a springboot app and just print log schedulely:
apiVersion: apps/v1
kind: Deployment
metadata:
name: logdemo-deployment
labels:
deployName: logdemo
spec:
replicas: 1
selector:
matchLabels:
appName: logdemo
template:
metadata:
labels:
appName: logdemo
spec:
containers:
- name: logdemo
image: swr.cn-north-1.myhuaweicloud.com/suixinliulang_dev/log-demo:1.0
error stack:
fan@k3s-master:~$ kubectl get pod -n logging
NAME READY STATUS RESTARTS AGE
demo-log-collector-fluentd-configcheck-ac2d4553 0/1 Completed 0 3h10m
demo-log-collector-fluentbit-j9mck 1/1 Running 0 122m
logging-operator-5898b6769b-6lllg 1/1 Running 13 (3h39m ago) 3d3h
demo-log-collector-fluentd-0 2/2 Running 0 3h10m
demo-log-collector-fluentd-configcheck-8fba9222 0/1 Error 0 23m
fan@k3s-master:~$ kubectl logs demo-log-collector-fluentd-configcheck-8fba9222 -n logging
fluentd -c /fluentd/etc/fluent.conf --dry-run
/usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:56:in `skip': invalid byte sequence in UTF-8 (ArgumentError)
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:56:in `skip'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/basic_parser.rb:84:in `spacing'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:61:in `parse_element'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:169:in `block in eval_include'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:in `each'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:163:in `eval_include'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:146:in `parse_include'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:105:in `parse_element'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:44:in `parse!'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/v1_parser.rb:33:in `parse'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config.rb:71:in `parse'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config.rb:52:in `build'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:679:in `initialize'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:in `new'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:348:in `<top (required)>'
from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in `<top (required)>'
from /usr/bin/fluentd:25:in `load'
from /usr/bin/fluentd:25:in `<main>'
Can't really tell without being able to check the actual config as is. It can be something with the secret you are using. I would try to see the config without the secret whether it works or not.
Can't really tell without being able to check the actual config as is. It can be something with the secret you are using. I would try to see the config without the secret whether it works or not.
fluentd config:
fan@k3s-master:~$ kubectl get secret demo-log-collector-fluentd-app -n logging -o jsonpath="{.data['fluentd\.conf']}" | base64 --decode
<source>
@type forward
@id main_forward
bind 0.0.0.0
port 24240
</source>
<match **>
@type label_router
@id main
metrics false
</match>
<label @ERROR>
<match **>
@type null
@id main-fluentd-error
</match>
</label>
fluentbit config:
fan@k3s-master:~$ kubectl get secret demo-log-collector-fluentbit -n logging -o jsonpath="{.data['fluent-bit\.conf']}" | base64 --decode
[SERVICE]
Flush 1
Grace 5
Daemon Off
Log_Level info
Parsers_File /fluent-bit/etc/parsers.conf
Coro_Stack_Size 24576
storage.path /buffers
[INPUT]
Name tail
DB /tail-db/tail-containers-state.db
DB.locking true
Mem_Buf_Limit 5MB
Parser cri
Path /var/log/containers/*.log
Refresh_Interval 5
Skip_Long_Lines On
Tag kubernetes.*
[FILTER]
Name kubernetes
Buffer_Size 0
K8S-Logging.Exclude On
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Tag_Prefix kubernetes.var.log.containers
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Token_TTL 600
Kube_URL https://kubernetes.default.svc:443
Match kubernetes.*
Merge_Log On
Use_Kubelet Off
[OUTPUT]
Name forward
Match *
Host demo-log-collector-fluentd.logging.svc.cluster.local.
Port 24240
Retry_Limit False
Describe the bug: When applying a flow and output to a namespace, the configcheck pod errors out with little information. Flow and Output objects have no reported 'Problems'. Logging-Operator logs indicate only that "the current config is invalid."
Expected behaviour: More detailed reason on why config is invalid. Also, which config is invalid? The logging object config? The flow? The output? I do not know.
Steps to reproduce the bug: Install logging-operator via helm template command on chart version 4.1.0-dev.1 with default values.
Apply logging-operator-logging logging object using helm template command on logging-operator-logging chart version version: 4.0.0.
At this point, initial configcheck pod which spins up, completes with no errors:
Apply a barebones namespace scoped flow and output with necessary secrets:
Flow
Output
Additional context: Add any other context about the problem here.
Environment details:
k logs logging-operator-logging-fluentd-configcheck-313b1d93 -n logging
fluentd -c /fluentd/etc/fluent.conf --dry-run /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/basic_parser.rb:56:in'
skip': invalid byte sequence in UTF-8 (ArgumentError) from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/basic_parser.rb:56:in
skip' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/basic_parser.rb:84:inspacing' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:61:in
parse_element' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:169:inblock in eval_include' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:163:in
each' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:163:ineval_include' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:146:in
parse_include' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:105:inparse_element' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:44:in
parse!' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config/v1_parser.rb:33:inparse' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config.rb:58:in
parse' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/config.rb:39:inbuild' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/supervisor.rb:618:in
initialize' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:354:innew' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/lib/fluent/command/fluentd.rb:354:in
<top (required)>' from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:inrequire' from /usr/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in
require' from /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.14.6/bin/fluentd:15:in<top (required)>' from /usr/bin/fluentd:23:in
load' from /usr/bin/fluentd:23:in `