logging-operator-logging-fluentd.logging.svc:24240 (Connection timed out)

[acarsercan]

Hi,

I'm trying to setup logging solution using the logging-operator. Combining fluent-bit, fluentd, S3, loki and grafana

I'm running kubernetes 1.15.9

helm install --namespace logging --name logging banzaicloud-stable/logging-operator

helm install --name loki loki/loki -f loki.yaml --namespace logging

values

config:
  schema_config:
    configs:
    - from: 2020-05-15
      store: aws
      object_store: s3
      schema: v11
      index:
        prefix: loki_index
        period: 336h
  storage_config:
    aws:
      s3: s3://XXXX
      bucketnames: ccuk-kubernetes-loki
      dynamodb:
        dynamodb_url: dynamodb://XXXXX
  table_manager:
    retention_deletes_enabled: true
    retention_period: 336h

Clusterflow

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterFlow
metadata:
  name: clusterflow-sample
spec:
  outputRefs:
    - loki-output
  match:
    - select:
        namespaces:
          - production-dynamo

output

apiVersion: logging.banzaicloud.io/v1beta1
kind: ClusterOutput
metadata:
 name: loki-output
spec:
 loki:
   url: http://loki:3100
   configure_kubernetes_labels: true
   buffer:
     timekey: 1m
     timekey_wait: 30s
     timekey_use_utc: true

Logging

apiVersion: logging.banzaicloud.io/v1beta1
kind: Logging
metadata:
  name: logging-operator-logging
spec:
  fluentd:
    logLevel: debug
    fluentOutLogrotate:
      enabled: true
      path: /fluentd/log/out
      age: "10"
      size: "10485760"
    scaling:
      replicas: 3
  fluentbit:
    inputTail:
      Mem_Buf_Limit: "256MB"
    filterKubernetes:
       Kube_URL: "https://kubernetes.default.svc:443"
       Match: "kube.*"
  controlNamespace: logging

k8s svc in logging namespace

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)               AGE
logging-logging-operator           ClusterIP   None             <none>        8080/TCP              22h
logging-operator-logging-fluentd   ClusterIP   100.69.2.88      <none>        24240/TCP,24240/UDP   5m36s
loki                               ClusterIP   100.70.252.253   <none>        3100/TCP              21h
loki-headless                      ClusterIP   None             <none>        3100/TCP              21h

I was hoping to see logs from production-dynamo namespace, however I don't. Here is the error which I'm failing to solve

logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection timed out)
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection timed out)
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection timed out)
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection timed out)
logging/logging-operator-logging-fluentbit-drntr[fluent-bit]: [2020/08/11 13:29:47] [error] [output:forward:forward.0] no upstream connections available

Replies are appreciated

Regards

[tarokkk]

Could you paste the kubectl get po -n logging output?

[acarsercan]

when i install logging object

logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.4079401,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.4434352,"logger":"controllers.Logging","msg":"still waiting for the configcheck result..."}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.4441767,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.4478495,"logger":"controllers.Logging","msg":"there are pending configcheck pods, need to back off"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.4526553,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.456177,"logger":"controllers.Logging","msg":"there are pending configcheck pods, need to back off"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.4638445,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160248.4671798,"logger":"controllers.Logging","msg":"there are pending configcheck pods, need to back off"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160249.1197636,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160249.1225626,"logger":"controllers.Logging","msg":"there are pending configcheck pods, need to back off"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160249.850328,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160249.8531914,"logger":"controllers.Logging","msg":"there are running configcheck pods, need to back off"}
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: fluentd -c /fluentd/etc/generated.conf --dry-run
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: parsing config file is succeeded path="/fluentd/etc/generated.conf"
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-aws-elasticsearch-service' version '2.4.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-azurestorage' version '0.1.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-cloudwatch-logs' version '0.10.2'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-datadog' version '0.12.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-dedot_filter' version '1.0.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '4.1.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-gcs' version '0.4.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-geoip' version '1.3.2'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-grafana-loki' version '1.2.14'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-kafka' version '0.13.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-kinesis' version '3.2.3'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.5.2'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-label-router' version '0.2.4'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-logdna' version '0.3.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-logzio' version '0.0.20'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-newrelic' version '1.1.8'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-oss' version '0.0.2'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-parser-logfmt' version '0.0.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.8.2'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-remote-syslog' version '1.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.3.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-s3' version '1.4.0'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-splunk-hec' version '1.2.3'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-sumologic_output' version '1.7.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-tag-normaliser' version '0.1.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-throttle' version '0.0.5'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.2.5'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: gem 'fluentd' version '1.11.1'
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: starting fluentd-1.11.1 as dry run mode ruby="2.7.1"
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [warn]: define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: using configuration file: <ROOT>
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:   <source>
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:     @type forward
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:     @id main_forward
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:     bind "0.0.0.0"
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:     port 24240
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:   </source>
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:   <match **>
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:     @type label_router
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:     @id main
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:     metrics false
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:   </match>
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: </ROOT>
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [warn]: parameter 'metrics' in <match **>
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:   @type label_router
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:   @id main
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]:   metrics false
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: </match> is not used.
logging/logging-operator-logging-fluentd-configcheck-c3054702[fluentd]: 2020-08-11 15:37:30 +0000 [info]: finished dry run mode
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160250.8824303,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}
logging/logging-logging-operator-7cf4d8d64c-hg7pk[logging-operator]: {"level":"info","ts":1597160250.903397,"logger":"controllers.Logging","msg":"no flows found, generating empty model"}

[acarsercan]

Once I install the flow

logging/logging-operator-logging-fluentbit-svk8x[fluent-bit]: [2020/08/11 15:39:48] [ info] [engine] flush chunk '1-1597160261.709118263.flb' succeeded at retry 2: task_id=2, input=tail.0 > output=forward.0
logging/logging-operator-logging-fluentbit-svk8x[fluent-bit]: [2020/08/11 15:39:48] [ info] [engine] flush chunk '1-1597160263.550594111.flb' succeeded at retry 2: task_id=2, input=tail.0 > output=forward.0
logging/logging-operator-logging-fluentbit-4vgtb[fluent-bit]: [2020/08/11 15:39:49] [ info] [engine] flush chunk '1-1597160256.306187199.flb' succeeded at retry 2: task_id=2, input=tail.0 > output=forward.0
logging/logging-operator-logging-fluentbit-4vgtb[fluent-bit]: [2020/08/11 15:39:50] [ info] [engine] flush chunk '1-1597160259.289344934.flb' succeeded at retry 2: task_id=2, input=tail.0 > output=forward.0
logging/logging-operator-logging-fluentbit-4vgtb[fluent-bit]: [2020/08/11 15:39:51] [ info] [engine] flush chunk '1-1597160258.336217722.flb' succeeded at retry 2: task_id=2, input=tail.0 > output=forward.0
logging/loki-0[loki]: 2020/08/11 15:39:56 Request.HasNextPage deprecated. Use Pagination type for configurable pagination of API operations
logging/logging-operator-logging-fluentd-0[config-reloader]: 2020/08/11 15:40:35 config map updated
logging/logging-operator-logging-fluentd-0[config-reloader]: 2020/08/11 15:40:35 successfully triggered reload
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection refused)
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection refused)
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection refused)
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection refused)
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [output:forward:forward.0] no upstream connections available
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [io] TCP connection failed: logging-operator-logging-fluentd.logging.svc:24240 (Connection refused)
logging/logging-operator-logging-fluentbit-xgxnd[fluent-bit]: [2020/08/11 15:40:36] [error] [output:forward:forward.0] no upstream connections available

[acarsercan]

This is now resolved. I'm deploying logging-operator and logging object using helm rather than applying a YAML