This is a meta issue capturing all of the details for the plugin's initial release targeted for 2.17.
Overview
Background & problem statement
Over the last several OpenSearch releases, there have been many enhancements made in ingest and search pipelines to support more flexible ML-related use cases, allowing customers to build exactly what they need. But with flexibility, comes complication and difficulty in configuration. It can be hard for users to see how their data flows and is transformed through these pipelines, as well as uncovering where certain problems may lie.
Solution
The [Flow Framework] plugin lets users build out custom search applications using ingest and search pipelines in a user-friendly UI. Users can iteratively build and test these pipelines, and visually see how data is transformed at each step. They can test out different processors, test out different configurations within individual processors, and finally, run concrete ingest and search queries to see how it performs. When users are satisfied with their solution, they can export their produced configuration via a Flow Framework workflow template. This template can then be exported and imported into different clusters, letting users scale and deploy their solution across their workloads/fleets.
Under the hood, this plugin uses the Flow Framework plugin to maintain configuration, created resources, and automatic provisioning and deprovisioning of such resources as users iteratively build out and test their applications.
Next steps
In the future, we would like to expand the plugin's functionality in several different ways. These include:
More processor support: onboarding and integrating with more existing ingest / search request / search pipeline processors can increase the usability and flexibility of the plugin
Add drag-and-drop support: rather than a form-based, guardrailed approach to configuration, the plugin could utilize the (currently read-only) ReactFlow workspace, into a customizable, drag-and-drop experience for building out their flows.
Advanced / DAG-based flows: currently, the flows are tied to ingest and search pipelines, which only allow a single flow of data transformations. Allowing users to build DAG-based flows and parallel executions could greatly enhance the flexibility provided to users.
Checklist
For the 2.17 release, these are the following remaining issues to be completed:
This is a meta issue capturing all of the details for the plugin's initial release targeted for 2.17.
Overview
Background & problem statement
Over the last several OpenSearch releases, there have been many enhancements made in ingest and search pipelines to support more flexible ML-related use cases, allowing customers to build exactly what they need. But with flexibility, comes complication and difficulty in configuration. It can be hard for users to see how their data flows and is transformed through these pipelines, as well as uncovering where certain problems may lie.
Solution
The [Flow Framework] plugin lets users build out custom search applications using ingest and search pipelines in a user-friendly UI. Users can iteratively build and test these pipelines, and visually see how data is transformed at each step. They can test out different processors, test out different configurations within individual processors, and finally, run concrete ingest and search queries to see how it performs. When users are satisfied with their solution, they can export their produced configuration via a Flow Framework workflow template. This template can then be exported and imported into different clusters, letting users scale and deploy their solution across their workloads/fleets.
Under the hood, this plugin uses the Flow Framework plugin to maintain configuration, created resources, and automatic provisioning and deprovisioning of such resources as users iteratively build out and test their applications.
Next steps
In the future, we would like to expand the plugin's functionality in several different ways. These include:
Checklist
For the 2.17 release, these are the following remaining issues to be completed: