Open graytaylor0 opened 7 months ago
I haven't tested it but my understanding is that when using opensearch serverless collection as sink, index_type defaults to management_disabled
, which is probably the reason pipeline doesn't create index or index template when sink is initialized. Set index_type
to custom
would probably make it work.
@graytaylor0 , Yes, this should work. I think in Data Prepper 3.0 we could fix these defaults. But, this default was chosen because we first added support for Serverless without any support for index management. So changing the default could change existing behavior.
Describe the bug When using an OpenSearch Serverless collection in a pipeline in the
opensearch
sink, the index configured is not created until a document is sent to the pipeline and is indexed into the collection.To Reproduce Steps to reproduce the behavior:
Expected behavior The index should be created on startup like it is for non-serverless opensearch sinks
Screenshots If applicable, add screenshots to help explain your problem.
Environment (please complete the following information):
Additional context Add any other context about the problem here.