Closed samalanubhab closed 1 month ago
Consolidating Ollama-related issues: https://github.com/microsoft/graphrag/issues/657
这是我的最终配置。VSCode 崩溃后,当我再次启动它时,摘要报告不知何故开始工作。 这是我迄今为止工作的最终完整配置:
encoding_model: cl100k_base skip_workflows: [] llm: api_key: ${GRAPHRAG_API_KEY} type: openai_chat # or azure_openai_chat model: llama3 model_supports_json: true # recommended if this is available for your model. max_tokens: 2000 # request_timeout: 180.0 api_base: http://localhost:11434/v1 # api_version: 2024-02-15-preview # organization: <organization_id> # deployment_name: <azure_model_deployment_name> # tokens_per_minute: 150_000 # set a leaky bucket throttle # requests_per_minute: 10_000 # set a leaky bucket throttle max_retries: 1 # max_retry_wait: 10.0 # sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times concurrent_requests: 1 # the number of parallel inflight requests that may be made parallelization: stagger: 0.3 # num_threads: 50 # the number of threads to use for parallel processing async_mode: threaded # or asyncio embeddings: ## parallelization: override the global parallelization settings for embeddings async_mode: threaded # or asyncio llm: api_key: ${GRAPHRAG_API_KEY} type: openai_embedding # or azure_openai_embedding model: text-embedding-3-small api_base: https://api.openai.com/v1 # api_version: 2024-02-15-preview # organization: <organization_id> # deployment_name: <azure_model_deployment_name> # tokens_per_minute: 150_000 # set a leaky bucket throttle # requests_per_minute: 10_000 # set a leaky bucket throttle max_retries: 1 # max_retry_wait: 10.0 # sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times concurrent_requests: 1 # the number of parallel inflight requests that may be made batch_size: 1 # the number of documents to send in a single request batch_max_tokens: 8191 # the maximum number of tokens to send in a single request # target: required # or optional chunks: size: 300 overlap: 100 group_by_columns: [id] # by default, we don't allow chunks to cross documents input: type: file # or blob file_type: text # or csv base_dir: "input" file_encoding: utf-8 file_pattern: ".*\\.txt$" cache: type: file # or blob base_dir: "cache" # connection_string: <azure_blob_storage_connection_string> # container_name: <azure_blob_storage_container_name> storage: type: file # or blob base_dir: "output/${timestamp}/artifacts" # connection_string: <azure_blob_storage_connection_string> # container_name: <azure_blob_storage_container_name> reporting: type: file # or console, blob base_dir: "output/${timestamp}/reports" # connection_string: <azure_blob_storage_connection_string> # container_name: <azure_blob_storage_container_name> entity_extraction: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task prompt: "prompts/entity_extraction.txt" entity_types: [organization,person,geo,event] max_gleanings: 0 summarize_descriptions: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task prompt: "prompts/summarize_descriptions.txt" max_length: 500 claim_extraction: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task # enabled: true prompt: "prompts/claim_extraction.txt" description: "Any claims or facts that could be relevant to information discovery." max_gleanings: 0 community_report: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task prompt: "prompts/community_report.txt" max_length: 2000 max_input_length: 7000 cluster_graph: max_cluster_size: 10 embed_graph: enabled: false # if true, will generate node2vec embeddings for nodes # num_walks: 10 # walk_length: 40 # window_size: 2 # iterations: 3 # random_seed: 597832 umap: enabled: false # if true, will generate UMAP embeddings for nodes snapshots: graphml: false raw_entities: false top_level_nodes: false local_search: # text_unit_prop: 0.5 # community_prop: 0.1 # conversation_history_max_turns: 5 # top_k_mapped_entities: 10 # top_k_relationships: 10 # max_tokens: 12000 global_search: # max_tokens: 12000 # data_max_tokens: 12000 # map_max_tokens: 1000 # reduce_max_tokens: 2000 # concurrency: 32
本质上我通过 ollama 在本地使用 llama3 作为实体,并使用 openai 嵌入(便宜得多),直到我们找到使用 ollama 的解决方案。
我使用你的设置和默认文本,并且不更改任何其他内容,但仍然
❌ create_final_entities None ⠹ GraphRAG Indexer ├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00 ├── create_base_text_units ├── create_base_extracted_entities ├── create_summarized_entities ├── create_base_entity_graph └── create_final_entities ❌ Errors occurred during the pipeline run, see logs for more details.
您需要先启动 ollama curl -fsSL https://ollama.com/install.sh | sh ollama pull llama3 ollama serve
我按照你的步骤操作,仍然得到同样的错误,显示调用 llm 的错误
remove cache
我将其配置为:
embeddings: ## parallelization: override the global parallelization settings for embeddings async_mode: threaded # or asyncio llm: api_key: ${GRAPHRAG_API_KEY} type: openai_embedding # or azure_openai_embedding model: nomic-embed-text api_base: http://localhost:11434/v1 # api_version: 2024-02-15-preview # organization: <organization_id> # deployment_name: <azure_model_deployment_name> # tokens_per_minute: 150_000 # set a leaky bucket throttle # requests_per_minute: 10_000 # set a leaky bucket throttle # max_retries: 10 # max_retry_wait: 10.0 # sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times concurrent_requests: 1 # the number of parallel inflight requests that may be made # batch_size: 16 # the number of documents to send in a single request # batch_max_tokens: 8191 # the maximum number of tokens to send in a single request # target: required # or optional
改用 api v1 👍 14:55:29,949 graphrag.index.verbs.text.embed.strategies.openai INFO 使用 1 个批次通过 9 个片段嵌入 9 个输入。 max_batch_size=16,max_tokens=8191 14:55:31,373 httpx INFO HTTP 请求:POST http://127.0.0.1:11434/api/embeddings “HTTP/1.1 200 OK” 14:55:31,375 graphrag.index.reporting.file_workflow_callbacks INFO 调用 LLM 时出错,详细信息 = {'input':['“团队”:“团队被描绘成一群从被动观察者转变为任务中积极参与者的个人,展示了他们角色的动态变化。”','“华盛顿”:','“行动:杜尔塞”:','“亚历克斯”:“亚历克斯是试图与未知情报进行首次接触的团队的领导者,承认他们任务的重要性。”','“控制”:“控制是指管理或治理的能力,这是受到编写自己规则的情报的挑战。”', '“INTELLIGENCE”:“这里的情报是指能够编写自己的规则并学习交流的未知实体。”', '“FIRST CONTACT”:“第一次接触”:“第一次接触是人类与未知情报之间潜在的初次交流。”', '“SAM RIVERA”:', '“人类”的回应:']} 14:55:31,375 datashaper.workflow.workflow 错误在 create_final_entities 中执行动词“text_embed”时出错:“NoneType”对象不可迭代
至少我嵌入成功了,但格式似乎不对
How to solve this text_embed problem. I have the same problem. And then,the whole log problem is as follows
datashaper.workflow.workflow ERROR Error executing verb "text_embed" in create_final_entities: Error code: 400 - {'object': 'error', 'message': 'NETWORK ERROR DUE TO HIGH TRAFFIC. PLEASE REGENERATE OR REFRESH THIS PAGE.\n\n(The expanded size of the tensor (513) must match the existing size (512) at non-singleton dimension 1. Target sizes: [4, 513]. Tensor sizes: [1, 512])', 'code': 50001}
是否有人遇到过全局查询有效但本地查询无效的情况?
did you solve it?
Here is my final config. Somehow after VSCode crashed the summary reports started working when I started it again.
Here is my final full config that work so far:
encoding_model: cl100k_base skip_workflows: [] llm: api_key: ${GRAPHRAG_API_KEY} type: openai_chat # or azure_openai_chat model: llama3 model_supports_json: true # recommended if this is available for your model. max_tokens: 2000 # request_timeout: 180.0 api_base: http://localhost:11434/v1 # api_version: 2024-02-15-preview # organization: <organization_id> # deployment_name: <azure_model_deployment_name> # tokens_per_minute: 150_000 # set a leaky bucket throttle # requests_per_minute: 10_000 # set a leaky bucket throttle max_retries: 1 # max_retry_wait: 10.0 # sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times concurrent_requests: 1 # the number of parallel inflight requests that may be made parallelization: stagger: 0.3 # num_threads: 50 # the number of threads to use for parallel processing async_mode: threaded # or asyncio embeddings: ## parallelization: override the global parallelization settings for embeddings async_mode: threaded # or asyncio llm: api_key: ${GRAPHRAG_API_KEY} type: openai_embedding # or azure_openai_embedding model: text-embedding-3-small api_base: https://api.openai.com/v1 # api_version: 2024-02-15-preview # organization: <organization_id> # deployment_name: <azure_model_deployment_name> # tokens_per_minute: 150_000 # set a leaky bucket throttle # requests_per_minute: 10_000 # set a leaky bucket throttle max_retries: 1 # max_retry_wait: 10.0 # sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times concurrent_requests: 1 # the number of parallel inflight requests that may be made batch_size: 1 # the number of documents to send in a single request batch_max_tokens: 8191 # the maximum number of tokens to send in a single request # target: required # or optional chunks: size: 300 overlap: 100 group_by_columns: [id] # by default, we don't allow chunks to cross documents input: type: file # or blob file_type: text # or csv base_dir: "input" file_encoding: utf-8 file_pattern: ".*\\.txt$" cache: type: file # or blob base_dir: "cache" # connection_string: <azure_blob_storage_connection_string> # container_name: <azure_blob_storage_container_name> storage: type: file # or blob base_dir: "output/${timestamp}/artifacts" # connection_string: <azure_blob_storage_connection_string> # container_name: <azure_blob_storage_container_name> reporting: type: file # or console, blob base_dir: "output/${timestamp}/reports" # connection_string: <azure_blob_storage_connection_string> # container_name: <azure_blob_storage_container_name> entity_extraction: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task prompt: "prompts/entity_extraction.txt" entity_types: [organization,person,geo,event] max_gleanings: 0 summarize_descriptions: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task prompt: "prompts/summarize_descriptions.txt" max_length: 500 claim_extraction: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task # enabled: true prompt: "prompts/claim_extraction.txt" description: "Any claims or facts that could be relevant to information discovery." max_gleanings: 0 community_report: ## llm: override the global llm settings for this task ## parallelization: override the global parallelization settings for this task ## async_mode: override the global async_mode settings for this task prompt: "prompts/community_report.txt" max_length: 2000 max_input_length: 7000 cluster_graph: max_cluster_size: 10 embed_graph: enabled: false # if true, will generate node2vec embeddings for nodes # num_walks: 10 # walk_length: 40 # window_size: 2 # iterations: 3 # random_seed: 597832 umap: enabled: false # if true, will generate UMAP embeddings for nodes snapshots: graphml: false raw_entities: false top_level_nodes: false local_search: # text_unit_prop: 0.5 # community_prop: 0.1 # conversation_history_max_turns: 5 # top_k_mapped_entities: 10 # top_k_relationships: 10 # max_tokens: 12000 global_search: # max_tokens: 12000 # data_max_tokens: 12000 # map_max_tokens: 1000 # reduce_max_tokens: 2000 # concurrency: 32
Essentially I use llama3 localy via ollama for the entities and use openai embeddings (much cheaper) until we have a solution to use ollama.
@bmaltais hi,I don't understand what value should be set for your api-key in your example. Can you tell me more about it? Thanks.
If you want to use open-source models, I've put together a repository for deploying models from HuggingFace to local endpoints, having similar endpoints with compatible format as OpenAI API. Here’s the link to the repo: https://github.com/rushizirpe/open-llm-server
Also, I have created a Colab notebook (working for global as well as local search) for Graphrag: https://colab.research.google.com/drive/1uhFDnih1WKrSRQHisU-L6xw6coapgR51?usp=sharing
I use ollama as local LLM API provider , and set chat model and embedding model api_base the same both with :http://loclahost:11434/v1
.
global search works ok, but local search failed with this message:
(graphRAG) D:\Learn\GraphRAG>python -m graphrag.query --root ./newTest09 --method local "谁是叶文洁"
INFO: Reading settings from newTest09\settings.yaml
creating llm client with {'api_key': 'REDACTED,len=6', 'type': "openai_chat", 'model': 'qwen2', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1/', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': True, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 10, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25}
creating embedding llm client with {'api_key': 'REDACTED,len=6', 'type': "openai_embedding", 'model': 'nomic-embed-text', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1/', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': None, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 10, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25}
Error embedding chunk {'OpenAIEmbedding': "Error code: 400 - {'error': {'message': 'invalid input type', 'type': 'api_error', 'param': None, 'code': None}}"}
Traceback (most recent call last):
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\runpy.py", line 86, in _run_code
exec(code, run_globals)
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\__main__.py", line 75, in <module>
run_local_search(
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\cli.py", line 154, in run_local_search
result = search_engine.search(query=query)
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\structured_search\local_search\search.py", line 118, in search
context_text, context_records = self.context_builder.build_context(
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\structured_search\local_search\mixed_context.py", line 139, in build_context
selected_entities = map_query_to_entities(
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\context_builder\entity_extraction.py", line 55, in map_query_to_entities
search_results = text_embedding_vectorstore.similarity_search_by_text(
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\vector_stores\lancedb.py", line 118, in similarity_search_by_text
query_embedding = text_embedder(text)
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\context_builder\entity_extraction.py", line 57, in <lambda>
text_embedder=lambda t: text_embedder.embed(t),
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\llm\oai\embedding.py", line 96, in embed
chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens)
File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\numpy\lib\function_base.py", line 550, in average
raise ZeroDivisionError(
ZeroDivisionError: Weights sum to zero, can't be normalized
I doubted maybe the embedding process doesn't work correct, beacause it mentioned code:400 error with OpenAIEmbedding api. But the indexing process seems working fine:
⠋ GraphRAG Indexer
├── Loading Input (text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00
├── create_base_text_units
├── create_base_extracted_entities
├── create_summarized_entities
├── create_base_entity_graph
├── create_final_entities
├── create_final_nodes
├── create_final_communities
├── join_text_units_to_entity_ids
├── create_final_relationships
├── join_text_units_to_relationship_ids
├── create_final_community_reports
├── create_final_text_units
├── create_base_documents
└── create_final_documents
🚀 All workflows completed successfully.
As my understanding, indexing process also need do embedding , how come it doesn't work when local search ? Does anyone have the same issues with GraphRAG ?
I had the same problem and I was wondering if anyone had completely solved it?
$ ollama -v ollama version is 0.1.34
and my settings.yaml `encoding_model: cl100k_base skip_workflows: [] llm: api_key: ollama # ${GRAPHRAG_API_KEY} type: openai_chat # or azure_openai_chat model: llama3:70b model_supports_json: false # recommended if this is available for your model. api_base: http://10.110.0.25:11434/v1 ...
embeddings:
async_mode: threaded # or asyncio llm: api_key: ollama type: openai_embedding # or azure_openai_embedding model: nomic-embed-text:latest api_base: http://10.110.0.25:11434/api ...`
then run the command: `poetry run poe index --root .
...
❌ create_final_entities None ⠦ GraphRAG Indexer ├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00 ├── create_base_text_units ├── create_base_extracted_entities ├── create_summarized_entities ├── create_base_entity_graph └── create_final_entities ❌ Errors occurred during the pipeline run, see logs for more details.`
checked the error log:
httpx INFO HTTP Request: POST http://10.110.0.25:11434/api/embeddings "HTTP/1.1 200 OK" graphrag.index.reporting.file_workflow_callbacks INFO Error Invoking LLM details={....} 17:19:55,244 datashaper.workflow.workflow ERROR Error executing verb "text_embed" in create_final_entities: 'NoneType' object is not iterable
I had the same problem and I was wondering if anyone had completely solved it?
$ ollama -v ollama version is 0.1.34
and my settings.yaml `encoding_model: cl100k_base skip_workflows: [] llm: api_key: ollama # ${GRAPHRAG_API_KEY} type: openai_chat # or azure_openai_chat model: llama3:70b model_supports_json: false # recommended if this is available for your model. api_base: http://10.110.0.25:11434/v1 ...
embeddings:
parallelization: override the global parallelization settings for embeddings
async_mode: threaded # or asyncio llm: api_key: ollama type: openai_embedding # or azure_openai_embedding model: nomic-embed-text:latest api_base: http://10.110.0.25:11434/api ...`
then run the command: `poetry run poe index --root .
...
❌ create_final_entities None ⠦ GraphRAG Indexer ├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00 ├── create_base_text_units ├── create_base_extracted_entities ├── create_summarized_entities ├── create_base_entity_graph └── create_final_entities ❌ Errors occurred during the pipeline run, see logs for more details.`
checked the error log:
httpx INFO HTTP Request: POST http://10.110.0.25:11434/api/embeddings "HTTP/1.1 200 OK" graphrag.index.reporting.file_workflow_callbacks INFO Error Invoking LLM details={....} 17:19:55,244 datashaper.workflow.workflow ERROR Error executing verb "text_embed" in create_final_entities: 'NoneType' object is not iterable
https://github.com/microsoft/graphrag/issues/339#issuecomment-2250094743
ZeroDivisionError: Weights sum to zero, can't be normalized
The locally running embedding model in OLLAMA returns the weights in an incorrect format. OpenAI internally uses base64 encoded floats, whereas most other models return floats as numbers.
This is working: https://github.com/rushizirpe/open-llm-server I have created a Colab notebook (working for global as well as local search) for Graphrag: https://colab.research.google.com/drive/1uhFDnih1WKrSRQHisU-L6xw6coapgR51?usp=sharing
I had the same problem and I was wondering if anyone had completely solved it?
$ ollama -v ollama version is 0.1.34
and my settings.yaml `encoding_model: cl100k_base skip_workflows: [] llm: api_key: ollama # ${GRAPHRAG_API_KEY} type: openai_chat # or azure_openai_chat model: llama3:70b model_supports_json: false # recommended if this is available for your model. api_base: http://10.110.0.25:11434/v1 ... embeddings:parallelization: override the global parallelization settings for embeddings
async_mode: threaded # or asyncio llm: api_key: ollama type: openai_embedding # or azure_openai_embedding model: nomic-embed-text:latest api_base: http://10.110.0.25:11434/api ...
then run the command:
poetry run poe index --root . ... ❌ create_final_entities None ⠦ GraphRAG Indexer ├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00 ├── create_base_text_units ├── create_base_extracted_entities ├── create_summarized_entities ├── create_base_entity_graph └── create_final_entities ❌ Errors occurred during the pipeline run, see logs for more details.checked the error log:
httpx INFO HTTP Request: POST http://10.110.0.25:11434/api/embeddings "HTTP/1.1 200 OK" graphrag.index.reporting.file_workflow_callbacks INFO Error Invoking LLM details={....} 17:19:55,244 datashaper.workflow.workflow ERROR Error executing verb "text_embed" in create_final_entities: 'NoneType' object is not iterable`
I have read the comment above, where the embedding uses openai, but I hope it will be local. : -)
I had the same problem and I was wondering if anyone had completely solved it?
$ ollama -v ollama version is 0.1.34
and my settings.yaml `encoding_model: cl100k_base skip_workflows: [] llm: api_key: ollama # ${GRAPHRAG_API_KEY} type: openai_chat # or azure_openai_chat model: llama3:70b model_supports_json: false # recommended if this is available for your model. api_base: http://10.110.0.25:11434/v1 ... embeddings:parallelization: override the global parallelization settings for embeddings
async_mode: threaded # or asyncio llm: api_key: ollama type: openai_embedding # or azure_openai_embedding model: nomic-embed-text:latest api_base: http://10.110.0.25:11434/api ...
then run the command:
poetry run poe index --root . ... ❌ create_final_entities None ⠦ GraphRAG Indexer ├── Loading Input (InputFileType.text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00 ├── create_base_text_units ├── create_base_extracted_entities ├── create_summarized_entities ├── create_base_entity_graph └── create_final_entities ❌ Errors occurred during the pipeline run, see logs for more details.checked the error log:
httpx INFO HTTP Request: POST http://10.110.0.25:11434/api/embeddings "HTTP/1.1 200 OK" graphrag.index.reporting.file_workflow_callbacks INFO Error Invoking LLM details={....} 17:19:55,244 datashaper.workflow.workflow ERROR Error executing verb "text_embed" in create_final_entities: 'NoneType' object is not iterable`I have read the comment above, where the embedding uses openai, but I hope it will be local. : -)
if you take a look at notebook you'll find it uses nomic-ai/nomic-embed-text-v1.5 as mentioned in .yaml config (You can use any valid model which can be loaded from huggingface). You need GROQ API key only in terms of chat completion aif you don't have higher end GPU, and if you have it then you just need to replace API endpoint as http://localhost:1234/v1 and model name (From HF) you want to use.
ZeroDivisionError: Weights sum to zero, can't be normalized
The locally running embedding model in OLLAMA returns the weights in an incorrect format. OpenAI internally uses base64 encoded floats, whereas most other models return floats as numbers.
This is working: https://github.com/rushizirpe/open-llm-server I have created a Colab notebook (working for global as well as local search) for Graphrag: https://colab.research.google.com/drive/1uhFDnih1WKrSRQHisU-L6xw6coapgR51?usp=sharing
oh...are you actually replacing ollama with open llm server in this notebook?
ZeroDivisionError: Weights sum to zero, can't be normalized
The locally running embedding model in OLLAMA returns the weights in an incorrect format. OpenAI internally uses base64 encoded floats, whereas most other models return floats as numbers. This is working: https://github.com/rushizirpe/open-llm-server I have created a Colab notebook (working for global as well as local search) for Graphrag: https://colab.research.google.com/drive/1uhFDnih1WKrSRQHisU-L6xw6coapgR51?usp=sharing
oh...are you actually replacing ollama with open llm server in this notebook?
Yes, hope it helps!
I use ollama as local LLM API provider , and set chat model and embedding model api_base the same both with :
http://loclahost:11434/v1
. global search works ok, but local search failed with this message:(graphRAG) D:\Learn\GraphRAG>python -m graphrag.query --root ./newTest09 --method local "谁是叶文洁" INFO: Reading settings from newTest09\settings.yaml creating llm client with {'api_key': 'REDACTED,len=6', 'type': "openai_chat", 'model': 'qwen2', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1/', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': True, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 10, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25} creating embedding llm client with {'api_key': 'REDACTED,len=6', 'type': "openai_embedding", 'model': 'nomic-embed-text', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1/', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': None, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 10, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25} Error embedding chunk {'OpenAIEmbedding': "Error code: 400 - {'error': {'message': 'invalid input type', 'type': 'api_error', 'param': None, 'code': None}}"} Traceback (most recent call last): File "D:\ProgramData\miniconda3\envs\graphRAG\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "D:\ProgramData\miniconda3\envs\graphRAG\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\__main__.py", line 75, in <module> run_local_search( File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\cli.py", line 154, in run_local_search result = search_engine.search(query=query) File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\structured_search\local_search\search.py", line 118, in search context_text, context_records = self.context_builder.build_context( File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\structured_search\local_search\mixed_context.py", line 139, in build_context selected_entities = map_query_to_entities( File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\context_builder\entity_extraction.py", line 55, in map_query_to_entities search_results = text_embedding_vectorstore.similarity_search_by_text( File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\vector_stores\lancedb.py", line 118, in similarity_search_by_text query_embedding = text_embedder(text) File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\context_builder\entity_extraction.py", line 57, in <lambda> text_embedder=lambda t: text_embedder.embed(t), File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\graphrag\query\llm\oai\embedding.py", line 96, in embed chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens) File "D:\ProgramData\miniconda3\envs\graphRAG\lib\site-packages\numpy\lib\function_base.py", line 550, in average raise ZeroDivisionError( ZeroDivisionError: Weights sum to zero, can't be normalized
I doubted maybe the embedding process doesn't work correct, beacause it mentioned code:400 error with OpenAIEmbedding api. But the indexing process seems working fine:
⠋ GraphRAG Indexer ├── Loading Input (text) - 1 files loaded (0 filtered) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00 0:00:00 ├── create_base_text_units ├── create_base_extracted_entities ├── create_summarized_entities ├── create_base_entity_graph ├── create_final_entities ├── create_final_nodes ├── create_final_communities ├── join_text_units_to_entity_ids ├── create_final_relationships ├── join_text_units_to_relationship_ids ├── create_final_community_reports ├── create_final_text_units ├── create_base_documents └── create_final_documents 🚀 All workflows completed successfully.
As my understanding, indexing process also need do embedding , how come it doesn't work when local search ? Does anyone have the same issues with GraphRAG ?
I am experiencing the same issue here, from my understanding Ollama now has oai compatible api endpoint v1/embeddings
for embeddings, so indexing step works fine when using http://localhost:11434/v1
as api_base for both llm and embeddings. Global query also works. Local query may have its separate embedding logics that caused the issue, it generates lancedb files, digging into it now.
Note:
concurrent_requests
, chunks.size
and the specific ollama served model's num_ctx
have to be set properly to pass the indexing step. In my case:
In the beginning I used llama3.1
model, its modelfile (you can check using ollama show --modelfile llama3.1
or ollama show --parameters llama3.1
) does not specify num_ctx
, so by default when Ollama will serve the llama3.1 model with num_ctx = 2048
, in Ollama's console log where it loads the model you will find something like:
n_ctx
in the image is 4 x num_ctx
. Graphrag default chunks size is 1200, plus default prompts (inside promtps
folder after initializing the project) are lengthy, the final prompt sent to Ollama can easily surpass num_ctx
(you will find input truncated
in Ollama console logs for /v1/chat/completions
api calls). The result of this is llm will not follow the specified output format in returned response, you can check the generated files inside cache
folder to see if it follows the output format as specified in prompts
folder. In my case it does output any entity and caused this issue: https://github.com/microsoft/graphrag/issues/443#issuecomment-2248000519
The way to solve this is:
chunks.size
smaller, e.g 300. (However I still noticed input truncated
from Ollama at the final step of indexing create_community_report
, because prompts/community_report.txt
is too long).Or (I would prefer this way) increase the num_ctx
from Ollama side. However, we can only set/change num_ctx
options
in Ollama api endpoint /api/chat
(not oai endpoint /v1/chat/completions
)
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1",
"messages": [
{
"role": "user",
"content": "Tell me a random story:"
}
], "stream": false, "options": {"num_ctx": 4096}
}' | jq .
Modelfile
with num_ctx
parameter specified as suggested https://github.com/ollama/ollama/issues/5902 .I specified 4096 as num_ctx
in the new modelfile (instead of 8192, now I can see n_ctx
is 16384 in ollama console log when loading the model) and kept chunks.size
as 1200.
As for concurrent_requests
setting, the default value 25
will cause request timed out error during indexing, ollama console log looks fine but you can find request timed out errors in indexing-engine.log
and logs.json
file. how Ollama handle concurrent requests. I set llm.concurrent_requests: 1
and embeddings.llm.concurrent_requests: 4
.
With these settings, indexing and global query work smoothly. Local query still has issue.
local query issue is solved by https://github.com/microsoft/graphrag/issues/451#issuecomment-2220861232 and https://github.com/microsoft/graphrag/pull/568/files , but the query result seems not so good.
Update:
num_ctx
from 4096
to 10240
, local search result starts looking relevant and give references. chunks.size
will make prompts longer, in my case global search prompt surpasses num_ctx
and llm response does not return json structured response as required in the prompt. in this case increase chunks.size
or num_ctx
.regarding generating embeddings with ollama. I had this issue already in another project. Calling function embeddings() of ollama -> _client.py -> Client returns:
'POST',
'/api/embeddings',
json={
'model': model,
'prompt': prompt,
'options': options or {},
'keep_alive': keep_alive,
},
).json()
At this point '/api/embeddings/'
is not replaced with the specified api_base.
Replacing it manually with ollama's 'http://localhost:11434/api/embeddings/
at least makes it possible to generate embeddings with ollama
What I tried: I ran this on my local GPU and and tried replacing the api_base to a model served on ollama in settings.yaml file. model: llama3:latest api_base: http://localhost:11434/v1 #https://.openai.azure.com
Error: graphrag.index.reporting.file_workflow_callbacks INFO Error Invoking LLM details={'input': '\n-Goal-\nGiven a text document that is pot....}
Commands:
initialize
python -m graphrag.index --init --root .
index
python -m graphrag.index --root .
query
python -m graphrag.query --root . --method global "query"
query
python -m graphrag.query --root . --method local "query"
Does graphrag support other llm hosted server frameworks?