Document and integration test OpenSearch support

[spencergilbert]

The existing elasticsearch sink should support OpenSearch today, we should however integration test it to maintain this support as well as better documenting it.

[benjaminhuo]

@spencergilbert you mean currently vector supports adding OpenSearch configuration to Elasticsearch sink to send logs to OpenSearch? And yes, vector do lack of docs to sending logs to OpenSearch

[spencergilbert]

Correct, Vector's elasticsearch sink should be compatible with OpenSearch - and as I understand it we do have users doing that today.

We are lacking tests to ensure that remains the case and documentation around what versions are supported and if any specific configuration needs to be used.

[benjaminhuo]

That sounds good.

I also noticed that there's an issue saying Elastic made its latest libraries not work with OpenSearch : https://opensearch.org/blog/community/2021/08/community-clients/

So it might be great if there is a dedicated OpenSearch sink

https://github.com/vectordotdev/vector/issues/11738

[spencergilbert]

We don't use Elastic's libraries and construct the payloads ourselves - so the single source should be fine until they drift further apart.

[benjaminhuo]

We don't use Elastic's libraries and construct the payloads ourselves - so the single source should be fine until they drift further apart.

That's great!

[benjaminhuo]

@spencergilbert I got the following error when sending logs to OpenSearch v2.6.0:

vector is deployed in agent mode following the method in https://github.com/vectordotdev/helm-charts/tree/develop/charts/vector

helm repo add vector https://helm.vector.dev
helm repo update
helm install -n vector -f values.yaml vector vector/vector

error:

root@benjamin-lab:~/vector# kubectl -n vector logs ks-vector-cp9zs -f
2023-05-10T06:31:47.680831Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-05-10T06:31:47.685117Z  INFO vector::config::watcher: Creating configuration file watcher.
2023-05-10T06:31:47.686085Z  INFO vector::config::watcher: Watching configuration files.
2023-05-10T06:31:47.686390Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2023-05-10T06:31:47.693439Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="benjamin-lab"
2023-05-10T06:31:47.712322Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2023-05-10T06:31:47.773932Z  WARN http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-05-10T06:31:47.777630Z  WARN vector::sinks::elasticsearch::common: Failed to determine Elasticsearch version from `/_cluster/state/version`. Please fix the reported error or set an API version explicitly via `api_version`. assumed_version=8 error=Failed to get Elasticsearch API version: Failed to make HTTP(S) request: connection closed before message completed
2023-05-10T06:31:47.811141Z  INFO vector::topology::running: Running healthchecks.
2023-05-10T06:31:47.811762Z  INFO vector: Vector has started. debug="false" version="0.29.1" arch="x86_64" revision="74ae15e 2023-04-20 14:50:42.739094536"
2023-05-10T06:31:47.817480Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-05-10T06:31:47.818404Z  WARN http: vector::internal_events::http_client: Internal log [HTTP error.] is being rate limited.
2023-05-10T06:31:47.818691Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: connection closed before message completed component_kind="sink" component_type="elasticsearch" component_id=opensearch component_name=opensearch
2023-05-10T06:31:47.818215Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground
2023-05-10T06:32:49.319390Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kubesphere-monitoring-system_notification-manager-operator-85d67fdc46-pc8w6_4f7abd6f-004d-4e09-9fd1-7c12f3dc9164/kube-rbac-proxy/1.log file_position=498

config:

customConfig:
  data_dir: /vector-data-dir
  # Vector's API for introspection
  api:
    enabled: true
    address: "0.0.0.0:8686"
    playground: true

  # Read Kubernetes logs from files
  sources:
    kube_logs:
      type: "kubernetes_logs"
      pod_annotation_fields:
        container_id: ".kubernetes.docker_id"
        container_image_id: ""
        pod_annotations: ""
        pod_ip: ""
        pod_ips: ""
        pod_labels: ""
        pod_owner: ""
        pod_uid: ""
        pod_namespace: ".kubernetes.namespace_name"
        pod_node_name: ".kubernetes.node_name"
      namespace_annotation_fields:
        namespace_labels: ""
      node_annotation_fields:
        node_labels: ""
  # Transform Kube logs and remove unused fields
  transforms:
    kube_logs_remapped:
      type: "remap"
      inputs:
        - kube_logs
      source: |-
        .log = .message
        .time = .timestamp
        del(.file)
        del(.message)
        del(.timestamp_end)
        del(.stream)
        del(.source_type)
  # Forward logs to stdout 
  sinks:
#    stdout:
#      type: console
#      inputs:
#        - kube_logs_remapped
#      target: stdout
#      encoding:
#        codec: json
    opensearch:
      type: elasticsearch
      inputs:
        - kube_logs_remapped
      auth:
        strategy: basic
        user: admin
        password: admin  
      api_version: auto
      compression: none
      # This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to set this option since Elasticsearch has removed it.
      # doc_type: _doc
      endpoints:
        - http://opensearch-cluster-data.<namespace>.svc:9200
      mode: bulk
      bulk: 
        index: "k8s-logging-%Y.%m.%d"

After changing the api_version from auto to v8 or v7, still got below errors:

root@benjamin-lab:~/vector# kubectl -n vector logs ks-vector-qwg2m -f
2023-05-10T07:04:20.840957Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=trace,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-05-10T07:04:20.844872Z  INFO vector::config::watcher: Creating configuration file watcher.
2023-05-10T07:04:20.846328Z  INFO vector::config::watcher: Watching configuration files.
2023-05-10T07:04:20.846574Z  INFO vector::app: Loading configs. paths=["/etc/vector"]
2023-05-10T07:04:20.856001Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Obtained Kubernetes Node name to collect logs for (self). self_node_name="benjamin-lab"
2023-05-10T07:04:20.873952Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}: vector::sources::kubernetes_logs: Excluding matching files. exclude_paths=["**/*.gz", "**/*.tmp"]
2023-05-10T07:04:20.926348Z  INFO vector::topology::running: Running healthchecks.
2023-05-10T07:04:20.927123Z  INFO vector: Vector has started. debug="false" version="0.29.1" arch="x86_64" revision="74ae15e 2023-04-20 14:50:42.739094536"
2023-05-10T07:04:20.937496Z  INFO vector::internal_events::api: API server running. address=0.0.0.0:8686 playground=http://0.0.0.0:8686/playground
2023-05-10T07:04:20.938250Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: file_source::checkpointer: Loaded checkpoint data.
2023-05-10T07:04:20.956226Z  WARN http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-05-10T07:04:20.956766Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Failed to make HTTP(S) request: connection closed before message completed component_kind="sink" component_type="elasticsearch" component_id=opensearch component_name=opensearch
2023-05-10T07:05:22.440838Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kubesphere-system_minio-7879c5dd65-jgv8s_21b492b2-fa71-4a8c-9a10-d16f92bfae10/minio/1.log file_position=1023
...
2023-05-10T07:05:22.497413Z  INFO source{component_kind="source" component_id=kube_logs component_type=kubernetes_logs component_name=kube_logs}:file_server: vector::internal_events::file::source: Resuming to watch file. file=/var/log/pods/kubesphere-logging-system_opensearch-logging-curator-elasticsearch-curator-28060860-hrtl7_aaa43c9a-c5cc-4898-aa42-936cf41d29dd/elasticsearch-curator/0.log file_position=16079
2023-05-10T07:05:22.883870Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=1}:http: vector::internal_events::http_client: HTTP error. error=connection closed before message completed error_type="request_failed" stage="processing" internal_log_rate_limit=true
2023-05-10T07:05:22.884067Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=1}: vector::sinks::util::retries: Retrying after error. error=Failed to make HTTP(S) request: connection closed before message completed internal_log_rate_limit=true
2023-05-10T07:05:23.185165Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=2}:http: vector::internal_events::http_client: Internal log [HTTP error.] is being rate limited.
2023-05-10T07:05:23.185424Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=2}: vector::sinks::util::retries: Internal log [Retrying after error.] is being rate limited.
2023-05-10T07:05:24.187024Z  WARN sink{component_kind="sink" component_id=opensearch component_type=elasticsearch component_name=opensearch}:request{request_id=2}: vector::sinks::util::service::health: Endpoint is unhealthy. endpoint=http://opensearch-cluster-data.kubesphere-logging-system.svc:9200

[spencergilbert]

@benjaminhuo please open an issue or discussion if you're having issues - this issue is to track adding docs and tests for OpenSearch. Additionally, those logs suggest nothing to do with ElasticSearch or OpenSearch, merely that there are networking issues between the client and server.

[benjaminhuo]

@benjaminhuo please open an issue or discussion if you're having issues - this issue is to track adding docs and tests for OpenSearch. Additionally, those logs suggest nothing to do with ElasticSearch or OpenSearch, merely that there are networking issues between the client and server.

Sure, I'll double check my opensearch setup and will open an issue if it's indeed a problem. Thanks again

[benjaminhuo]

Problem solved by skipping tls and adding https prefix:

    opensearch:
      type: elasticsearch
      inputs:
        - kube_logs_remapped
      auth:
        strategy: basic
        user: admin
        password: admin
      tls:
        verify_certificate: false
      api_version: auto
      compression: none
      # This is only relevant for Elasticsearch <= 6.X. If you are using >= 7.0 you do not need to set this option since Elasticsearch has removed it.
      # doc_type: _doc
      endpoints:
        - https://<opensearch-cluster>:9200
      mode: bulk
      bulk: 
        index: "ks-whizard-logging-%Y.%m.%d"

Thanks!