search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.96k stars 9.19k forks source link

Sending logs from Fluentbit to Fluend through HTTPS #10268

Closed ajith2234 closed 2 years ago

ajith2234 commented 2 years ago

Name and Version

bitnami/fluentd 5.0.0

What steps will reproduce the bug?

I want to create an environment where multiple fluent bit instances send logs to a fluentd instance deployed in Kubernetes through HTTPS. I am using an ingress controller to receive traffic from outside the cluster and redirect it to the fluentd aggregator. I am using Azure Kubernetes Service for deploying fluentd and my local machine to use a fluent bit instance to send logs. I have also created a DNS name using Azure DNS Zone

Step 1: Deploy Fluend to Kubernetes Aggregator Configuration in values.yaml

    aggregator:
      ## @param aggregator.enabled Enable Fluentd aggregator statefulset
      ##
      enabled: true
      ## @param aggregator.image.registry Fluentd aggregator image registry override
      ## @param aggregator.image.repository Fluentd aggregator image repository override
      ## @param aggregator.image.tag Fluentd aggregator image tag override (immutable tags are recommended)
      image:
        registry: ""
        repository: ""
        tag: ""
      ## @param aggregator.replicaCount Number of aggregator pods to deploy in the Stateful Set
      ##
      replicaCount: 2
      ## K8s Security Context for Aggregator pods
      ## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
      ## @param aggregator.podSecurityContext.enabled Enable security context for aggregator pods
      ## @param aggregator.podSecurityContext.runAsUser User ID for aggregator's containers
      ## @param aggregator.podSecurityContext.runAsGroup Group ID for aggregator's containers
      ## @param aggregator.podSecurityContext.fsGroup Group ID for aggregator's containers filesystem
      ##
      podSecurityContext:
        enabled: true
        runAsUser: 1001
        runAsGroup: 1001
        fsGroup: 1001
      ## @param aggregator.hostAliases Add deployment host aliases
      ## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
      ##
      hostAliases: []
      ## K8s Security Context for Aggregator containers
      ## https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
      ## @param aggregator.containerSecurityContext.enabled Enable security context for the aggregator container
      ## @param aggregator.containerSecurityContext.privileged Run as privileged
      ## @param aggregator.containerSecurityContext.allowPrivilegeEscalation Allow Privilege Escalation
      ## @param aggregator.containerSecurityContext.readOnlyRootFilesystem Require the use of a read only root file system
      ## @param aggregator.containerSecurityContext.capabilities.drop [array] Drop capabilities for the securityContext
      ##
      containerSecurityContext:
        enabled: true
        privileged: false
        allowPrivilegeEscalation: false
        ## Requires mounting an `extraVolume` of type `emptyDir` into /tmp
        ##
        readOnlyRootFilesystem: false
        capabilities:
          drop:
            - ALL
      ## @param aggregator.terminationGracePeriodSeconds Duration in seconds the pod needs to terminate gracefully
      ## https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/
      ##
      terminationGracePeriodSeconds: 30
      ## @param aggregator.configFile Name of the config file that will be used by Fluentd at launch under the `/opt/bitnami/fluentd/conf` directory
      ##
      configFile: fluentd.conf
      ## @param aggregator.configMap Name of the config map that contains the Fluentd configuration files
      ##
      configMap: ""
      ## @param aggregator.configMapFiles [object] Files to be added to be config map. Ignored if `aggregator.configMap` is set
      ##
      configMapFiles:
        fluentd.conf: |
          # Ignore fluentd own events
          <match fluent.**>
            @type null
          </match>

          @include fluentd-inputs.conf
          @include fluentd-output.conf
          {{- if .Values.metrics.enabled }}
          @include metrics.conf
          {{- end }}
        fluentd-inputs.conf: |
          # HTTP input 
          <source>
            @type http
            bind 0.0.0.0
          </source>
        fluentd-output.conf: |
          # Throw the healthcheck to the standard output
          <match fluentd.healthcheck>
            @type stdout
          </match>

command: helm install -f values.yaml fluentd-aggregator bitnami/fluentd

Step 2: Deploy the ingress controller to Kubernetes https://github.com/kubernetes/ingress-nginx/

command: helm install -f values.yaml fluentd-ingress ingress-nginx/ingress-nginx

Step 3: Configure Ingress ingress.yaml file

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: fluentd
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /$2
        nginx.ingress.kubernetes.io/use-regex: "true"
    spec:
      ingressClassName: nginx
      rules:
      - host: my-dns-name.com
        http:
          paths:
            - pathType: Prefix
              backend:
                service:
                  name: fluentd-aggregator-aggregator 
                  port:
                    number: 9880
              path: /
      tls:
        - hosts:
          - my-dns-name.com
          secretName: tls-secret

command: kubectl apply -f ingress.yaml

I have created a certificate and secret using the documentation given below https://docs.microsoft.com/en-us/azure/aks/ingress-own-tls?tabs=azure-cli

Step 4: Fluent-Bit Configuration

Install fluent bit command: curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

While loop for getting logs from local machine command:

while [ $num -gt 0 ]
do
echo "$num - `date` - Message from `hostname`" >> /home/ajith/device.log
((num++))
sleep 1
done

Send Message to fluentd using tls

/opt/fluent-bit/bin/fluent-bit -i tail -p path=/home/ajith/device.log -o http://my-dns-name:80 -p format=json -p json_date_key=false -p tls=on -p tls.verify=on -p tls.crt_file=./tls.crt -p tls.key_file=./tls.key

What is the expected behavior?

    Fluent Bit v1.9.3
    * Copyright (C) 2015-2022 The Fluent Bit Authors
    * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
    * https://fluentbit.io

    [2022/05/17 17:43:12] [ info] [fluent bit] version=1.9.3, commit=, pid=1357
    [2022/05/17 17:43:12] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
    [2022/05/17 17:43:12] [ info] [cmetrics] version=0.3.1
    [2022/05/17 17:43:12] [ info] [sp] stream processor started
    [2022/05/17 17:43:12] [ info] [output:http:http.0] worker #1 started
    [2022/05/17 17:43:12] [ info] [input:tail:tail.0] inotify_fs_add(): inode=156255 watch_fd=1 name=/home/ajith/device.log
    [2022/05/17 17:43:12] [ info] [output:http:http.0] worker #0 started
    [2022/05/17 17:43:44] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200
    [2022/05/17 17:43:45] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200
    [2022/05/17 17:43:45] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200
    [2022/05/17 17:43:46] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200
    [2022/05/17 17:43:47] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200
    [2022/05/17 17:43:48] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200
    [2022/05/17 17:43:49] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200
    [2022/05/17 17:43:50] [ info] [output:http:http.0] te-fluentd.te.nestdigital.com:80, HTTP status=200

What do you see instead?

    Fluent Bit v1.9.3
    * Copyright (C) 2015-2022 The Fluent Bit Authors
    * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
    * https://fluentbit.io

    [2022/05/17 17:46:37] [ info] [fluent bit] version=1.9.3, commit=, pid=2031
    [2022/05/17 17:46:37] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
    [2022/05/17 17:46:37] [ info] [cmetrics] version=0.3.1
    [2022/05/17 17:46:37] [ info] [sp] stream processor started
    [2022/05/17 17:46:37] [ info] [input:tail:tail.0] inotify_fs_add(): inode=156255 watch_fd=1 name=/home/ajith/device.log
    [2022/05/17 17:46:37] [ info] [output:http:http.0] worker #0 started
    [2022/05/17 17:46:37] [ info] [output:http:http.0] worker #1 started
    [2022/05/17 17:46:39] [error] [tls] error: unexpected EOF
    [2022/05/17 17:46:39] [error] [output:http:http.0] no upstream connections available to te-fluentd.te.nestdigital.com:80
    [2022/05/17 17:46:39] [ warn] [engine] failed to flush chunk '2031-1652789797.924454632.flb', retry in 6 seconds: task_id=0, input=tail.0 > output=http.0 (out_id=0)
    [2022/05/17 17:46:39] [error] [tls] error: unexpected EOF
    [2022/05/17 17:46:39] [error] [output:http:http.0] no upstream connections available to te-fluentd.te.nestdigital.com:80
    [2022/05/17 17:46:39] [ warn] [engine] failed to flush chunk '2031-1652789798.928311173.flb', retry in 10 seconds: task_id=1, input=tail.0 > output=http.0 (out_id=0)
    [2022/05/17 17:46:41] [error] [tls] error: unexpected EOF
    [2022/05/17 17:46:41] [error] [output:http:http.0] no upstream connections available to te-fluentd.te.nestdigital.com:80
    [2022/05/17 17:46:41] [ warn] [engine] failed to flush chunk '2031-1652789799.932216454.flb', retry in 11 seconds: task_id=2, input=tail.0 > output=http.0 (out_id=0)
    [2022/05/17 17:46:42] [error] [tls] error: unexpected EOF
    [2022/05/17 17:46:42] [error] [output:http:http.0] no upstream connections available to te-fluentd.te.nestdigital.com:80
    [2022/05/17 17:46:42] [ warn] [engine] failed to flush chunk '2031-1652789800.936987660.flb', retry in 9 seconds: task_id=3, input=tail.0 > output=http.0 (out_id=0)
    [2022/05/17 17:46:42] [error] [tls] error: unexpected EOF
    [2022/05/17 17:46:42] [error] [output:http:http.0] no upstream connections available to te-fluentd.te.nestdigital.com:80
    [2022/05/17 17:46:42] [ warn] [engine] failed to flush chunk '2031-1652789801.941955732.flb', retry in 6 seconds: task_id=4, input=tail.0 > output=http.0 (out_id=0)

Additional information

I have created a DNS Zone resource in azure and added the IP of the ingress controller to it

javsalgar commented 2 years ago

Hi!

There's something I did not fully understand. If you are using HTTPS via ingress, shouldn't you use port 443 instead of 80?

ajith2234 commented 2 years ago

Thank @javsalgar for the reply, But I tried with port 443 and it's still not working. The only way it works through port 443 is when we mention tls.verify=off

/opt/fluent-bit/bin/fluent-bit -i tail -p path=/home/ajith/device.log -o http://my-dns-name:443 -p format=json -p json_date_key=false -p tls=on -p tls.verify=off.

But I would like it to work when tls.verify=on and we authenticate with the certificates.

ajith2234 commented 2 years ago

I think the problem is with the sender. I also tried to send the logs through Fluentd and was not successful.

carrodher commented 2 years ago

It seems it is not an issue related to the Bitnami FluentD container image or Helm chart but about how the application or environment is being used/configured.

For information regarding the application itself, customization of the content within the application, or questions about the use of technology or infrastructure; we highly recommend checking forums and user guides made available by the project behind the application or the technology.

That said, we will keep this ticket open until the stale bot closes it just in case someone from the community adds some valuable info.

github-actions[bot] commented 2 years ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 2 years ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.