opensearch-project / data-prepper

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
259 stars 191 forks source link

[BUG] [Docs] OpenSearch sink documented permissions insufficient #4194

Open arichtman-srt opened 7 months ago

arichtman-srt commented 7 months ago

Describe the bug

When configuring an OpenSearch sink with a user, the permissions documented result in failure to write data to OpenSearch

To Reproduce

Configure pipeline sink:

entry-pipeline:
  buffer:
    bounded_blocking:
      batch_size: 160
      buffer_size: 10240
  delay: "100"
  sink:
  - pipeline:
      name: raw-pipeline
raw-pipeline:
  buffer:
    bounded_blocking:
      batch_size: 160
      buffer_size: 10240
  processor:
  - otel_traces:
      trace_flush_interval: 1
  sink:
  - opensearch:
      hosts:
      - https://vpc-opensearch-dev-nevn7hqedhhppf7rw4c674cbne.eu-west-1.es.amazonaws.com
      index_type: trace-analytics-raw
      password: REDACTED
      username: observability
  - stdout: null
  source:
    pipeline:
      name: entry-pipeline

Configure role:

image

Map role to user:

image

Expected behavior

Data prepper starts without error, listens on the appropriate port, and writes to OpenSearch

Screenshots

As above.

log gist

Environment (please complete the following information):

Additional context

When I map the same user to all_access role, Data Prepper behaves as expected, and is able to set up the pipeline, listens on the configured port, and successfully writes to OpenSearch. If I add index permissions of indices_all on pattern *, it still doesn't work. So it's maybe cluster level index permissions? I've set JAVA_OPTS=-Dlog4j2.debug=true so hopefully the logs show something useful, but not the failing API call that I can see.

graytaylor0 commented 3 months ago

This may be a good starting point to update the documentation with the permissions needed with FGAC domains

image