Elasicsearch autodetect version doesn't support OpenSearch versions

[arve0]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Seems like vector to 0.30.0 broke elasticsearch version detection for OpenSearch endpoints. Our upgrade was from 0.29.1 to 0.30.0. Endpoint is running OpenSearch 2.5.0.

Error log is:

2023-06-14T08:04:31.859502Z ERROR sink{component_kind="sink" component_id=logs_domstol_no component_type=elasticsearch component_name=logs_domstol_no}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"}],\"type\":\"illegal_argument_exception\",\"reason\":\"Action/metadata line [1] contains an unknown parameter [_type]\"},\"status\":400}" internal_log_rate_limit=true

Configuration

We are deploying vector with helm chart version v0.22.0 (vector 0.30.0) or v0.21.1 (vector 0.29.1). Only image version changes was detected on upgrade:

-          image: "timberio/vector:0.30.0-distroless-libc"
+          image: "timberio/vector:0.29.1-distroless-libc"

Settings to vector are:

  logs_domstol_no:
    api_version: auto
    auth:
      password: $LOG_COLLECTOR_PASSWORD
      strategy: basic
      user: log-collector
    bulk:
      action: create
    compression: none
    data_stream:
      dataset: '{{ kubernetes.pod_labels.logs_domstol_no }}'
      namespace: '{{ kubernetes.pod_namespace }}'
      type: logs
    endpoints:
    - https://opensearch:9200
    inputs:
    - ecs_parsing
    - eventrouter_parsing
    - filter._unmatched
    mode: data_stream
    tls:
      verify_certificate: false
      verify_hostname: false
    type: elasticsearch

Version

vector 0.30.0 (aarch64-unknown-linux-gnu 38c3f0b 2023-05-22 17:38:48.655488673)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[arve0]

Workaround is specifying elasticsearch version, which works:

api_version: v8

[spencergilbert]

@arve0 - can you check your logs to see if this line is emitted?

"Unexpected response from Elasticsearch endpoint `/`. Consider setting `api_version` option."

[arve0]

I do not have the deployment running, but I do have log saved to OpenShift logging. No matches when searching for

Unexpected response from Elasticsearch endpoint

I do find matches for

Healthcheck passed

[dsmith3197]

@arve0 Thank you for following up.

I found the root issue here. Vector omits the _type parameter if version >= 7 (code ref). This logic assumes that the version number is an ElasticSearch version number and does not correctly handle OpenSearch version numbers (latest today is 2.5). To fix this, we need to detect both the version number and backend type (ES or OS).

For now, you can continue to use the workaround until a fix is implemented.

[balonik]

This is still present in 0.39.0, just the message changed to {"timestamp":"2024-07-10T13:57:36.040431Z","level":"WARN","message":"Failed to determine Elasticsearch API version. Please fix the reported error or set an API version explicitly via `api_version`.","assumed_version":"8","error":"EOF while parsing a value at line 1 column 0","target":"vector::sinks::elasticsearch::common","span":{"component_id":"opensearch","component_kind":"sink","component_type":"elasticsearch","name":"sink"},"spans":[{"component_id":"opensearch","component_kind":"sink","component_type":"elasticsearch","name":"sink"}]}

Any ETA when this will be resolved by removing the automatic detection and how it will affect any future differences between ES and OpenSearch?

[jszwedko]

This is still present in 0.39.0, just the message changed to {"timestamp":"2024-07-10T13:57:36.040431Z","level":"WARN","message":"Failed to determine Elasticsearch API version. Please fix the reported error or set an API version explicitly via `api_version`.","assumed_version":"8","error":"EOF while parsing a value at line 1 column 0","target":"vector::sinks::elasticsearch::common","span":{"component_id":"opensearch","component_kind":"sink","component_type":"elasticsearch","name":"sink"},"spans":[{"component_id":"opensearch","component_kind":"sink","component_type":"elasticsearch","name":"sink"}]}

Any ETA when this will be resolved by removing the automatic detection and how it will affect any future differences between ES and OpenSearch?

You can disable the automatic detection by setting api_version.