search

opensearch-project / data-prepper

Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
253 stars 185 forks source link

[BUG] OpenSearch Serverless Sink does not create index on startup #4097

Open graytaylor0 opened 7 months ago

graytaylor0 commented 7 months ago

Describe the bug When using an OpenSearch Serverless collection in a pipeline in the opensearch sink, the index configured is not created until a document is sent to the pipeline and is indexed into the collection.

To Reproduce Steps to reproduce the behavior:

  1. Create a pipeline with an opensearch serverless collection as a sink
  2. Observe the index configured in the sink is not created when starting Data Prepper
  3. Send an Event to the pipeline
  4. Observe the document in the collection in the specified index

Expected behavior The index should be created on startup like it is for non-serverless opensearch sinks

Screenshots If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

Additional context Add any other context about the problem here.

oeyh commented 7 months ago

I haven't tested it but my understanding is that when using opensearch serverless collection as sink, index_type defaults to management_disabled, which is probably the reason pipeline doesn't create index or index template when sink is initialized. Set index_type to custom would probably make it work.

dlvenable commented 6 months ago

@graytaylor0 , Yes, this should work. I think in Data Prepper 3.0 we could fix these defaults. But, this default was chosen because we first added support for Serverless without any support for index management. So changing the default could change existing behavior.