Run Fast-GraphRAG locally

[Z-ZHHH]

Thanks for sharing the work. How to Run Fast-GraphRAG Locally with FastChat/Ollama Models

[liukidar]

Hello. Ollama supports the OpenAI Chat Completions API (https://ollama.com/blog/openai-compatibility). We have created (in the repo, I have't updated the pypi package yet) a wrap around it that can be used as following:

from fast_graphrag._llm import OllamaAIEmbeddingService, OllamaAILLMService

grag = GraphRAG(
    working_dir=working_dir,
    domain=DOMAIN,
    example_queries="\n".join(QUERIES),
    entity_types=ENTITY_TYPES,
    config=GraphRAG.Config(
        llm_service=OllamaAILLMService(model="your-llm-model"),
        embedding_service=OllamaAIEmbeddingService(model="your-embedding-model"),
    ),
)

As default base_url I put the one used in the link above, but you can change it by passing a custom one to each Service. Please note that I'm not 100% sure about the Ollama support for embeddings (according to here it should work), so I'd love to have some feedback on this.

[Z-ZHHH]

Thank you for your response. I'd like to clarify the setup process for Fast-GraphRAG.

To run GraphRAG with Fastchat, the setup process involves two main steps:

1. Initialize the environment by running: python -m graphrag.index --init --root ./ragtest

2. Modify the setting.yaml file in ./ragtest to configure:

   • The api_base
   • The model_name

After these configurations, GraphRAG can be run with my local model.

Could you confirm if Fast-GraphRAG follows this same setup procedure, or are there additional/different steps required?

[liukidar]

I'm not very familiar with FastChat, but according to here (https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md), they support the OpenAI API format so it should be compatible. In particular, please try the following configuration:

from fast_graphrag._llm import OllamaAIEmbeddingService, OllamaAILLMService

grag = GraphRAG(
    working_dir=working_dir,
    domain=DOMAIN,
    example_queries="\n".join(QUERIES),
    entity_types=ENTITY_TYPES,
    config=GraphRAG.Config(
        llm_service=OllamaAILLMService(model="your-llm-model", base_url="api_base"),
        embedding_service=OllamaAIEmbeddingService(model="your-embedding-model", base_url="api_base"),
    ),
)

where, if my understanding is correct, base_url corresponds to the api entry point which for you is api_base and model is your model_name. Let me know if this works, otherwise I'm happy to get into the details.

[yang605222]

I want use other LLM,not gpt-4o,how can I do?

[liukidar]

I want use other LLM,not gpt-4o,how can I do?

Hello! Do you want to use other open ai LLMs, or local ones?

[yang605222]

Thanks for your reply. Yeah,I want use use other open ai LLMs.Such as client = OpenAI( api_key="sk-xxx", base_url="https://api.deepseek.com", ) And how should I modify the code?

[YeKelvin]

Error during information extraction from document: Instructor does not support multiple tool calls, use List[Model] instead

https://python.useinstructor.com/concepts/parallel/?h=parallel#understanding-parallel-function-calling

[Dormiveglia-elf]

Thanks for your reply. Yeah,I want use use other open ai LLMs.Such as client = OpenAI( api_key="sk-xxx", base_url="https://api.deepseek.com", ) And how should I modify the code?

Same question

[liukidar]

Thanks for your reply. Yeah,I want use use other open ai LLMs.Such as client = OpenAI( api_key="sk-xxx", base_url="https://api.deepseek.com", ) And how should I modify the code?

Hello, deepseek APIs seem to be compatible with OpenAI APIs. Please check examples/custom_llm.py tp see how to configure GraphRAG to use your service.

[liukidar]

Error during information extraction from document: Instructor does not support multiple tool calls, use List[Model] instead

https://python.useinstructor.com/concepts/parallel/?h=parallel#understanding-parallel-function-calling

Hey, thanks for highlighting this. I'm unsure what's the cause beyond this error message. Could you please open a new issue and maybe provide a minimum working example that generates this issue?