This PR begins the process of supporting the ML inference processor form inputs to support an advanced transformation flow. The scope of this PR is to set up the base form changes to the input map / output map inputs, the base transformation modals for configuring advanced input / output flows, and partial support of input transformations in the context of ingest. More specifically:
finished removing UI fields from the workflow config (help text / help link) for ML inference processors and embed them directly in the new MLProcessorInputs component
changing the ML processor inputs to using a specialized MLProcessorInputs component instead of the generic ConfigFieldList. This is required, since this particular processor we need to maintain more specific and complex state unique to this processor type. We still keep a parent/generic component with a switch to select the appropriate processor UX component based on the processor type. The main idea is switching from the components being flexible on a per-field basis ---> per-processor-type basis. This allows easy onboarding of new processor types we will eventually add.
introduces the InputTransformModal with partial implementation. Specifically, the ability to dynamically fetch the input schema. Future PRs will integrate with JSONPath and the ML interfaces to determine the output schema.
introduces the stubbed transform modals for configuring complex outputs
onboarding bulk API and updating the default document ingestion to be an array instead of an individual doc. This expands the capability and flexibility for users when trying to test/ingest with a set of documents
onboarding _simulate ingest pipeline API for executing proposed ingest pipeline configurations when users are configuring advanced inputs. For example, given some pipeline that looks like Processor A -> Processor B -> Processor C, suppose a user wants to configure Processor D at the end. They go into the advanced view to configure an input transformation. To fetch the transformed document data up toProcessor D, this PR adds logic to 1/ create a temporary pipeline configuration containing Processor A -> Processor B -> Processor C, collect the currently-configured documents in the form, run _simulate, and display the transformed doc results. See demo for a visual explanation
updating/adding global interfaces, including for the expected input/output of the simulate pipeline API, as these can get complicated and error-prone, and to make the code cleaner & more readable
Demo video, showing the advanced flow. Expected input to each processor is shown. For the first/initial processor, the expected input is simply the list of documents. For the second processor, the expected input is a transformed version of the document, containing the embeddings values. Internally, this creates an ingest pipeline containing all of the preceding processors and executes _simulate against the provided documents. _Note a single inputmap transform is made to convert the hello field to the model's expected input field.
Additional note: there is no defined UX on this, subject to change. But core functionality will remain the same.
Issues Resolved
Makes progress on #23
Check List
[x] Commits are signed per the DCO using --signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.
Description
This PR begins the process of supporting the ML inference processor form inputs to support an advanced transformation flow. The scope of this PR is to set up the base form changes to the input map / output map inputs, the base transformation modals for configuring advanced input / output flows, and partial support of input transformations in the context of ingest. More specifically:
MLProcessorInputs
componentMLProcessorInputs
component instead of the genericConfigFieldList
. This is required, since this particular processor we need to maintain more specific and complex state unique to this processor type. We still keep a parent/generic component with aswitch
to select the appropriate processor UX component based on the processor type. The main idea is switching from the components being flexible on a per-field basis ---> per-processor-type basis. This allows easy onboarding of new processor types we will eventually add.InputTransformModal
with partial implementation. Specifically, the ability to dynamically fetch the input schema. Future PRs will integrate with JSONPath and the ML interfaces to determine the output schema.bulk
API and updating the default document ingestion to be an array instead of an individual doc. This expands the capability and flexibility for users when trying to test/ingest with a set of documents_simulate
ingest pipeline API for executing proposed ingest pipeline configurations when users are configuring advanced inputs. For example, given some pipeline that looks likeProcessor A -> Processor B -> Processor C
, suppose a user wants to configureProcessor D
at the end. They go into the advanced view to configure an input transformation. To fetch the transformed document data up toProcessor D
, this PR adds logic to 1/ create a temporary pipeline configuration containingProcessor A -> Processor B -> Processor C
, collect the currently-configured documents in the form, run _simulate, and display the transformed doc results. See demo for a visual explanationDemo video, showing the advanced flow. Expected input to each processor is shown. For the first/initial processor, the expected input is simply the list of documents. For the second processor, the expected input is a transformed version of the document, containing the embeddings values. Internally, this creates an ingest pipeline containing all of the preceding processors and executes
_simulate
against the provided documents. _Note a single inputmap transform is made to convert thehello
field to the model's expectedinput
field.demo
Additional note: there is no defined UX on this, subject to change. But core functionality will remain the same.
Issues Resolved
Makes progress on #23
Check List
--signoff
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.