Could not communicate to OpenSearch

[marcio-pessoa]

Hi dears,

I'm trying to use Fluentd Daemonset for Kubernetes with OpenSearch 1.2.4.

Here are the DaemonSet definition:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1.14.5-debian-opensearch-amd64-1.0
        env:
          - name: K8S_NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name:  FLUENT_OPENSEARCH_HOST
            value: "api.opensearch.svc.cluster.local"
          - name:  FLUENT_OPENSEARCH_PORT
            value: "9200"
          - name: FLUENT_OPENSEARCH_SCHEME
            value: "https"
          - name: FLUENT_OPENSEARCH_SSL_VERIFY
            value: "false"
          - name: FLUENT_OPENSEARCH_SSL_VERSION
            value: "TLSv1_2"
          - name: FLUENT_OPENSEARCH_USER
            value: "fluentd"
          - name: FLUENT_OPENSEARCH_PASSWORD
            value: "password"
          # Logz.io Authentication
          # ======================
          # - name: LOGZIO_TOKEN
          #   value: "ThisIsASuperLongToken"
          # - name: LOGZIO_LOGTYPE
          #   value: "kubernetes"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        # When actual pod logs in /var/lib/docker/containers, the following lines should be used.
        # - name: dockercontainerlogdirectory
        #   mountPath: /var/lib/docker/containers
        #   readOnly: true
        # When actual pod logs in /var/log/pods, the following lines should be used.
        - name: dockercontainerlogdirectory
          mountPath: /var/log/pods
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      # When actual pod logs in /var/lib/docker/containers, the following lines should be used.
      # - name: dockercontainerlogdirectory
      #   hostPath:
      #     path: /var/lib/docker/containers
      # When actual pod logs in /var/log/pods, the following lines should be used.
      - name: dockercontainerlogdirectory
        hostPath:
          path: /var/log/pods

After apply the yaml, I got some logs:

...
  <match **>
    @type opensearch
    @id out_os
    @log_level "info"
    include_tag_key true
    host "api.opensearch.svc.cluster.local"
    port 9200
    path ""
    scheme https
    ssl_verify true
    ssl_version TLSv1_2
    ca_file ""
    user "fluentd"
    password xxxxxx
    client_cert ""
    client_key ""
    client_key_pass xxxxxx
    index_name "fluentd"
  </match>
</ROOT>
2022-04-05 21:50:20 +0000 [info]: starting fluentd-1.14.5 pid=7 ruby="2.7.5"
2022-04-05 21:50:20 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.7.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2022-04-05 21:50:21 +0000 [info]: adding match in @FLUENT_LOG pattern="fluent.**" type="null"
2022-04-05 21:50:21 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2022-04-05 21:50:21 +0000 [info]: adding match pattern="**" type="opensearch"
2022-04-05 21:50:23 +0000 [warn]: #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen -  (Errno::ENOENT)
2022-04-05 21:50:23 +0000 [warn]: #0 [out_os] Remaining retry: 14. Retry to communicate after 2 second(s).
...

Could you help me to solve the warning #0 [out_os] Could not communicate to OpenSearch, resetting connection and trying again. No such file or directory @ rb_sysopen - (Errno::ENOENT)?

The OpenSearch is working fine for other applications, I also tested the user and It's authenticating fine.

Thanks in advance.

[ghost]

Just encountered this

Seems you need to provide FLUENT_OPENSEARCH_CA_PATH and FLUENT_OPENSEARCH_CLIENT_* values in your env variables, or follow guidance in the readme ("Use your configuration) to remove those lines from /fluentd/etc/fluent.conf, even though FLUENT_OPENSEARCH_SSL_VERIFY is false.

[marcio-pessoa]

Your tip really help me @jm3dedalus Thank you very much!

After the change Fluentd connects to OpenSearch, but I got a lot of \\\\ on Fluentd logs. So, I added some parse configurations as environment variables:

          - name: FLUENT_CONTAINER_TAIL_PARSER_TYPE
            value: "cri"
          - name: FLUENT_CONTAINER_TAIL_PARSER_TIME_FORMAT
            value: "%Y-%m-%dT%H:%M:%S.%N%:z"

Now everything is working fine. Thanks again!

[marcio-pessoa]

I just submitted a pull request with a working OpenSearch example, because the final definition file is really different of ElasticSearch example file.