search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
9.74k stars 1.8k forks source link

Changing op_type in ingest pipeline in case of _bulk operation #2856

Open lockwooddev opened 2 years ago

lockwooddev commented 2 years ago

Problem I tried using an ingest pipeline to write to a data stream, but the client writing to the pipeline is using the bulk API without create as op_type. This would require a code change to the writing client.

Proposed solution It would be nice to have roughly the following processor in my ingest pipeline:

        {
            "set": {
                "field": "op_type",
                "value": "create"
            }
        }

Alternative Alternatively I could not use a data stream or change the client (which I don't want to do in this case)

I hope the case I present is clear. I would love some feedback on this strange case I have. Thanks in advance!

mvillafuertem commented 3 months ago

Hi @lockwooddev @ryanbogan , any update about this? it seems that I have this problem

org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - operation = Index, status = 400, error = only write ops with an op_type of create are allowed in data streams
gaobinlong commented 3 months ago

Sounds like a valid use case, now we supports modifying part of the metadata fields: https://github.com/opensearch-project/OpenSearch/blob/087355f0ee676064ea409ed68090b33e568ea941/server/src/main/java/org/opensearch/ingest/IngestDocument.java#L865, we can also support modifying the op_type parameter during the execution of ingest pipeline, will dive deep into this.