Cannot exec into the fluent-bit container starting 1.0.0

[varun-da]

Unable to exec into the container since version 1.0.0.

kubectl exec -ti fluent-bit-68865  -n logging -c fluent-bit -- sh
rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"sh\": executable file not found in $PATH"

command terminated with exit code 126

kubectl exec -ti fluent-bit-68865 -n logging -c fluent-bit -- bash rpc error: code = 2 desc = oci runtime error: exec failed: container_linux.go:247: starting container process caused "exec: \"bash\": executable file not found in $PATH"

command terminated with exit code 126

No shell available. Execing into the container helps during debugging and making sure the config is correct and doing other things such as a tcpdump to make sure information is flowing through correctly. Works fine in `0.14.9`.

[edsiper]

newer docker images are based in distroless and do not contain a shell.

[varun-da]

ah i see, it was multi stage docker file, i did not look all the way down. thanks!

how do you debug these containers? as i am deploying to k8s from docker hub, i do no compile them from source.

[varun-da]

@edsiper perhaps images with a shell can be published as well with a debug tag.

[edsiper]

yeah, a fluent/fluent-bit:1.x.x-debug would work

[paugay]

I had the same problem, switched to the debug one and worked. Thanks a lot for publishing those! 🥇

Is it possible to know the differences between the debug and non-debug?

I can guess the non-debug skips this command maybe? COPY --from=builder /bin/ /bin/

[varun-da]

@paugay non-debug (regular) image: https://github.com/fluent/fluent-bit-docker-image/blob/1.0/1.0/Dockerfile debug image: https://github.com/fluent/fluent-bit-docker-image/blob/1.0/1.0/Dockerfile.debug

All the debug image does is it has busybox binary, so something like docker exec -ti <container id> sh or kubectl exec -ti fluent-bit -n fluent-bit sh would work. Other than that one difference, it is essentially the non-debug distroless image (gcr.io/distroless/cc).

[varun-da]

thank you @edsiper for the debug images.

[shinebayar-g]

Following images doesn't have sh

fluent/fluent-bit:1.0.6-debug
fluent/fluent-bit:1.0.5-debug

@edsiper

[edsiper]

hmm it works:

$ docker run -ti fluent/fluent-bit:1.0.6-debug sh -c "/fluent-bit/bin/fluent-bit --version"
Fluent Bit v1.0.6

[shinebayar-g]

Ah I found the issue. @edsiper

This works on local docker. But not on k8s?

kubectl exec -it fluentbit-test-564596f88b-jsh62 -- sh     
/ # ls
ls: can't open '.': Value too large for defined data type
/ #

but this works.

~ kubectl exec -it fluentbit-test-564596f88b-jsh62 -- sh -c "/fluent-bit/bin/fluent-bit --version"
Fluent Bit v1.0.6

fluent/fluent-bit:1.0.4-debug image it's working fine.

~ kubectl exec -it fluentbit-test-947b7f845-frfg2 sh
# ls
bin  boot  dev  etc  fluent-bit  home  lib  lib64  proc  root  run  sbin  sys  tmp  usr  var
#

~ kubectl exec -it fluentbit-test-947b7f845-frfg2 -- sh -c "/fluent-bit/bin/fluent-bit --version"
Fluent Bit v1.0.4

[colegatron]

This problem is still around with 1.1.3-debug

$ k exec -ti fluentbit-fluentbit2graylog-5tqbd -c fb -- sh -c "ls"
ls: can't open '.': Value too large for defined data type
command terminated with exit code 1

$ k exec -ti fluentbit-fluentbit2graylog-5tqbd -c fb -- sh -c "/fluent-bit/bin/fluent-bit --version"
Fluent Bit v1.1.3

[HakanL]

I have the "Value too large for defined data type" issue on k8s in AWS EKS as well. It seems to be related to busybox not compiled with support for large file systems. As a quick fix I changed the dockerfile to point to busybox 1.31 (https://busybox.net/downloads/binaries/1.31.0-defconfig-multiarch-musl/) and use that temporarily. That worked for me. I put the image here if anyone needs it: https://cloud.docker.com/u/hakanl/repository/docker/hakanl/fluent-bit

[linbingdouzhe]

i use this command to check my configure file kubectl exec -it fluent-bit-pqxvs -n logging -- sh -c "cat /fluent-bit/etc/fluent-bit-output.conf"

[edsiper]

please try 1.3.2-debug

[bl000b]

@edsiper It's working in 1.3.2. Thanks.