search

run-llama / llama_index

LlamaIndex is a data framework for your LLM applications
https://docs.llamaindex.ai
MIT License
36.57k stars 5.23k forks source link

[Question]: Ingestion Pipelines and Workflows? #15401

Closed brycecf closed 2 months ago

brycecf commented 2 months ago

Question Validation

Question

Is the IngestionPipeline impacted by workflows at all or is it strictly QueryPipeline?

dosubot[bot] commented 2 months ago

The IngestionPipeline and QueryPipeline in the LlamaIndex framework share some common components, such as Document, VectorStoreIndex, and LLM, which could indicate an overlap in workflow impact. However, workflows are primarily defined and implemented within the QueryPipeline context, using the Workflow class and related components. There is no explicit indication that workflows directly impact the IngestionPipeline [1][2].

To continue talking to Dosu, mention @dosu.

logan-markewich commented 2 months ago

@brycecf actually, despite the name, the ingestion pipeline is neither a query pipeline or a workflow.

Probably at some point, we could consider making it a workflow so that it's more dynamic, but a little unsure what that will look like just yet. Maybe just something like IngestionPipeline(workflow=workflow, ...) rather than specifying transformations?