Data is discarded when elasticsearch index is in read-only state, which is not expected

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

2024-02-02T05:51:53.962140Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="error type: cluster_block_exception, reason: index [k8s-test-2024.02.02] blocked by: [FORBIDDEN/5/index read-only (api)];" internal_log_rate_limit=true 2024-02-02T05:51:53.962202Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=None request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true 2024-02-02T05:51:53.962230Z ERROR sink{component_kind="sink" component_id=my_sink_id component_type=elasticsearch}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=1 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

Configuration

sinks:
  my_sink_id:
    type: elasticsearch
    inputs:
    - my_transform
    auth:
      strategy: "basic"
      user: "user"
      password: "password"
    bulk:
      index: "k8s-test-%Y.%m.%d"
    endpoints: ["https://es-http.default.svc:9200"]
    tls:
      verify_certificate: false 
    healthcheck:
      enabled: true

Version

vector 0.35.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

vectordotdev / vector