Bulk API documentation does not indicate that pipelines are only supported update operations using the `upsert` action and no other update actions #114811
(Optional, string) ID of the pipeline to use to preprocess incoming documents. If the index has a default ingest pipeline specified, then setting the value to _none disables the default ingest pipeline for this request. If a final pipeline is configured it will always run, regardless of the value of this parameter.
However, it doesn't indicate that for update operations, this parameter is only supported for upsert actions and no other update actions. Moreover, the final pipeline will not run for update operations.
I believe this is because Elasticsearch doesn't have access to the complete document for the other update actions and, as such, it can't reliably execute any ingest pipelines because they may rely on the other fields for that document.
Please document this behavior: searching for pipeline on the Bulk API page doesn't yield any related information.
Create a second document with the bulk API + pipeline:
POST _bulk?pipeline=bulk-test-pipeline
{"create":{"_index":"bulk-test","_id":"doc_2"}}
{"field_c": "value-c-1"}
Results: Expected. This document contains an existing field that is unrelated to the ingest pipeline. There is currently no value for field_a or field_b
Update the second document with bulk API + pipeline:
POST _bulk?pipeline=bulk-test-pipeline
{"update":{"_index":"bulk-test","_id":"doc_2"}}
{"doc": {"field_a": "value-a-1"}}
Results: Unexpected. On document update, the pipeline parameter does not seem to be honoured. The value from field_a is not copied to field_b. This test shows that the issue is not related to the "Overwrite" flag on the Set processor in the pipeline.
Elasticsearch Version
All versions since ingest pipelines were introduced
Installed Plugins
No response
Java Version
bundled
OS Version
N/A
Problem Description
The Bulk API documentation mentions a
pipeline
query parameter:However, it doesn't indicate that for update operations, this parameter is only supported for
upsert
actions and no other update actions. Moreover, the final pipeline will not run for update operations.I believe this is because Elasticsearch doesn't have access to the complete document for the other update actions and, as such, it can't reliably execute any ingest pipelines because they may rely on the other fields for that document.
Please document this behavior: searching for
pipeline
on the Bulk API page doesn't yield any related information.Related:
Steps to Reproduce
Results: Expected. On document creation, the pipeline ran as expected and the value from
field_a
is correctly copied tofield_b
Results: Unexpected. On document update, the pipeline parameter does not seem to be honoured. The new value from
field_a
is not copied tofield_b
:Results: Expected. This document contains an existing field that is unrelated to the ingest pipeline. There is currently no value for
field_a
orfield_b
Results: Unexpected. On document update, the pipeline parameter does not seem to be honoured. The value from
field_a
is not copied tofield_b
. This test shows that the issue is not related to the "Overwrite" flag on the Set processor in the pipeline.Logs (if relevant)
No response