[ML] Enable chunking for ingestion pipelines

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine

https://www.elastic.co/products/elasticsearch

Other

1.33k stars 24.87k forks source link

[ML] Enable chunking for ingestion pipelines #117192

Open dan-rubinstein opened 1 day ago

dan-rubinstein commented 1 day ago

Description

There is currently no option in ingestion pipelines to use chunked inferences. This story is to implement the ability to do chunked inference within ingestion pipelines. The team has not yet aligned on whether we want to enable chunking in the existing ingestion pipeline processor or in a new processor? This will require some investigation to understand what the complexity/UX implications of either option are.

elasticsearchmachine commented 10 hours ago

Pinging @elastic/ml-core (Team:ML)