search

fluent / fluent-operator

Operate Fluent Bit and Fluentd in the Kubernetes way - Previously known as FluentBit Operator
Apache License 2.0
587 stars 250 forks source link

Add bearer token auth for loki #1224

Closed raynay-r closed 4 months ago

raynay-r commented 4 months ago

What this PR does / why we need it:

I was searching for an alternative of the openshift-logging-operator. None of the available operators supported bearer token authentication for the Loki output. Since bearer authentication is very useful together wiith Loki and the openshift oauth proxy I decided to make this contribution.

Which issue(s) this PR fixes:

Fixes #1221 (Not my ticket, by coincidence someone else seems to be missing this feature)

Does this PR introduced a user-facing change?

 * [Fluent-Bit] Additional Parameter "bearerToken" for (Cluster) Output of type Loki
 * [Fluent-Bit] Additional Parameter "uri" for (Cluster) Output of type Loki
 * [Fluentd] Additional Parameter "bearerTokenFile" for ClusterOutput of type Loki

Additional documentation, usage docs, etc.:

- [Usage]: https://docs.fluentbit.io/manual/pipeline/outputs/loki
- [Other doc] https://github.com/fluent/fluent-operator/blob/master/docs/plugins/fluentbit/output/loki.md
- [Usage] https://github.com/fluent/fluent-operator/blob/master/docs/plugins/fluentd/output/loki.md
- [Other doc] https://docs.fluentbit.io/manual/pipeline/outputs/loki
raynay-r commented 4 months ago

I had to update the controller-gen version, otherwise the generation step would segfault. This caused a lot of changes. I am not sure how this should be handled.

cw-Guo commented 4 months ago

@raynay-r Thanks so much for you first contribution. But upgrade the contrller-gen version might not be a good practice for this MR. Can you please try to fix segfault? Feel free to share the problem you have when generating code here. Let us know the details so that we can help you.

benjaminhuo commented 4 months ago

image conflicts need to be resolved @raynay-r

cc @wanjunlei

raynay-r commented 4 months ago

Here is the output I get running make build on the main branch. You can also see in the output that I am using go 1.22.3. Is the go version maybe not compatible with the controller-gen version?

❯ go version
go version go1.22.3 linux/amd64
❯ make build
go get k8s.io/code-generator@v0.26.1
go mod download
/home/rene/Projects/fluent-operator/main/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xa054af]

goroutine 354 [running]:
go/types.(*Checker).handleBailout(0xc0004a5400, 0xc001b35d40)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/check.go:367 +0x88
panic({0xbbf300?, 0x12ab8a0?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/runtime/panic.go:770 +0x132
go/types.(*StdSizes).Sizeof(0x0, {0xdb6b18, 0x12b4040})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/sizes.go:228 +0x30f
go/types.(*Config).sizeof(...)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/sizes.go:333
go/types.representableConst.func1({0xdb6b18?, 0x12b4040?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/const.go:76 +0x9e
go/types.representableConst({0xdbcdf0, 0x1280200}, 0xc0004a5400, 0x12b4040, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/const.go:92 +0x192
go/types.(*Checker).arrayLength(0xc0004a5400, {0xdbb140, 0xc001844b60?})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:510 +0x2d3
go/types.(*Checker).typInternal(0xc0004a5400, {0xdb9760, 0xc001ccbe60}, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:299 +0x49d
go/types.(*Checker).definedType(0xc0004a5400, {0xdb9760, 0xc001ccbe60}, 0xc001b35328?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:180 +0x37
go/types.(*Checker).varType(0xc0004a5400, {0xdb9760, 0xc001ccbe60})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:145 +0x25
go/types.(*Checker).structType(0xc0004a5400, 0xc001d056b0, 0xc001d056b0?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/struct.go:113 +0x19f
go/types.(*Checker).typInternal(0xc0004a5400, {0xdb96d0, 0xc001cce408}, 0xc0012c3db0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:316 +0x1345
go/types.(*Checker).definedType(0xc0004a5400, {0xdb96d0, 0xc001cce408}, 0xc8815d?)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/typexpr.go:180 +0x37
go/types.(*Checker).typeDecl(0xc0004a5400, 0xc0012c3db0, 0xc001cd6580, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/decl.go:615 +0x44d
go/types.(*Checker).objDecl(0xc0004a5400, {0xdc23e0, 0xc0012c3db0}, 0x0)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/decl.go:197 +0xa7f
go/types.(*Checker).packageObjects(0xc0004a5400)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/resolver.go:681 +0x425
go/types.(*Checker).checkFiles(0xc0004a5400, {0xc001527a40, 0x3, 0x3})
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/check.go:408 +0x1a5
go/types.(*Checker).Files(...)
        /nix/store/vz8d6wmfcf38l3h3vymwqr6c5zxp5jmp-go-1.22.3/share/go/src/go/types/check.go:372
sigs.k8s.io/controller-tools/pkg/loader.(*loader).typeCheck(0xc000363080, 0xc0007e65c0)
        /home/rene/go/pkg/mod/sigs.k8s.io/controller-tools@v0.11.3/pkg/loader/loader.go:286 +0x36a
sigs.k8s.io/controller-tools/pkg/loader.(*Package).NeedTypesInfo(0xc0007e65c0)
        /home/rene/go/pkg/mod/sigs.k8s.io/controller-tools@v0.11.3/pkg/loader/loader.go:99 +0x39
sigs.k8s.io/controller-tools/pkg/loader.(*TypeChecker).check(0xc0008c1bc0, 0xc0007e65c0)
        /home/rene/go/pkg/mod/sigs.k8s.io/controller-tools@v0.11.3/pkg/loader/refs.go:268 +0x2b7
sigs.k8s.io/controller-tools/pkg/loader.(*TypeChecker).check.func1(0x4f?)
        /home/rene/go/pkg/mod/sigs.k8s.io/controller-tools@v0.11.3/pkg/loader/refs.go:262 +0x53
created by sigs.k8s.io/controller-tools/pkg/loader.(*TypeChecker).check in goroutine 149
        /home/rene/go/pkg/mod/sigs.k8s.io/controller-tools@v0.11.3/pkg/loader/refs.go:260 +0x1c5
make: *** [Makefile:56: generate] Error 2
cw-Guo commented 4 months ago

Here is the output I get running make build on the main branch. You can also see in the output that I am using go 1.22.3. Is the go version maybe not compatible with the controller-gen version?

it's possible. Mine is 1.22.0, and it works well. go version go1.22.0 darwin/arm64