[Feature Request]: Knowledge Graph Index generation parallelized (multiple LLM calls and embeddings generation at the same time)

[robertobalestri]

Feature Description

Current Problem: We need to parallelize the process of creating the index of a knowledge graph. Currently, the code sequentially processes each call to the language model (LLM) to build the triplets. The current implementation is as follows:

knowledge_graph_index = KnowledgeGraphIndex.from_documents(
    documents,
    storage_context=storage_context,
    max_triplets_per_chunk=3,
    show_progress=True,
    include_embeddings=True,
)

Asked Solution: We ask to be able to split the process to enable parallel processing of document chunks. For example. If i have a document of 1000 characters, i should be able to split it in chunks of 100 with Sentence Splitter, and then parallelize the process of calculating embeddings and creating triplets for these 10 chunks.

Reason

Efficiency: The parallel processing of document chunks significantly speeds up the index creation process.

Value of Feature

The primary challenge I face with using a knowledge graph is the slow speed of building the index. Currently, the process is sequential, which significantly hampers efficiency.

[logan-markewich]

@robertobalestri have you seen the PropertyGraphIndex? It's newer, more customizable, and most importantly, use async concurrency for LLM calls. Much faster

[logan-markewich]

(Imo the KnowledgeGraphIndex is basically in feature freeze, due to the newer PropertyGraphIndex)