[DOC] Documentation of batch ingestion feature

[chishui]

What do you want to do?

• [x] Request a change to existing documentation
• [x] Add new documentation
• [ ] Report a technical problem with the documentation
• [ ] Other

Tell us about your request. Provide a summary of the request and all versions that are affected. We added this batch ingestion feature, user can now use it to accelerate their ingestion time if the ingest pipeline has ML processors text_embedding and sparse_encoding which connects to external ML servers.

The documentation changes would include a new parameter in _bulk API, a new paragraph in ingest processor to explain the batch enabled processors, and a new page to introduce this feature.

What other resources are available? Provide links to related issues, POCs, steps for testing, etc.

• OpenSearch core: https://github.com/opensearch-project/OpenSearch/issues/12457
• neural-search plugin: https://github.com/opensearch-project/neural-search/issues/743

[hdhalter]

Thanks, @chishui! Is this being released in 2.15?

[chishui]

Thanks, @chishui! Is this being released in 2.15?

Yes, sorry that I forgot to mention the version

[hdhalter]

Hi @chishui , what is the ETA for this doc PR? The first release candidate is today. Thanks!

[peterzhuamazon]

Hi @chishui could you please provide a PR for this since it is labeled for 2.15 and we are approaching the release date?

Thanks.

[chishui]

Since there has been an active discussion on deprecating the parameter we introduced (https://github.com/opensearch-project/OpenSearch/issues/14283), whether we should announce the feature is still pending decision.

[peterzhuamazon]

Hi @chishui please give a decision on this next Monday/Tuesday. As we need to stay on the timeline.

If you have concerns including in 2.15.0 I would suggest removing the 2.15.0 label until resurface. That way, this would not be a blocker for 2.15.0 anymore.

Thanks.

[hdhalter]

Hi @chishui, I removed the 2.15 label. If something changes, we'll be happy to help get the content updated.

[chishui]

PR is raised, can we add the 2.15 label back? Thanks