search

microsoft / graphrag

A modular graph-based Retrieval-Augmented Generation (RAG) system
https://microsoft.github.io/graphrag/
MIT License
16.22k stars 1.52k forks source link

[Bug]: <title>Error embedding chunk {'OpenAIEmbedding': "'NoneType' object is not iterable"} #528

Closed hongyispace closed 1 month ago

hongyispace commented 1 month ago

Describe the bug

I have finished the pipeline with llm and ebedding of ollama. However, when I tried to query: python -m graphrag.query --root pg18v8 --method local "describe the types of impact craters"

The error message is: INFO: Reading settings from pg18v8/settings.yaml creating llm client with {'api_key': 'REDACTED,len=56', 'type': "openai_chat", 'model': 'llama3:70b-instruct-q5_K_M', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': True, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 1, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 1} creating embedding llm client with {'api_key': 'REDACTED,len=56', 'type': "openai_embedding", 'model': 'nomic-embed-text', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/api', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': None, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 3, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25} Error embedding chunk {'OpenAIEmbedding': "'NoneType' object is not iterable"} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/main.py", line 75, in run_local_search( File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/cli.py", line 154, in run_local_search result = search_engine.search(query=query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/structured_search/local_search/search.py", line 118, in search context_text, context_records = self.context_builder.build_context( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/structured_search/local_search/mixed_context.py", line 139, in build_context selected_entities = map_query_to_entities( ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/context_builder/entity_extraction.py", line 55, in map_query_to_entities search_results = text_embedding_vectorstore.similarity_search_by_text( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/vector_stores/lancedb.py", line 118, in similarity_search_by_text query_embedding = text_embedder(text) ^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/context_builder/entity_extraction.py", line 57, in text_embedder=lambda t: text_embedder.embed(t), ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/llm/oai/embedding.py", line 96, in embed chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/numpy/lib/function_base.py", line 550, in average raise ZeroDivisionError( ZeroDivisionError: Weights sum to zero, can't be normalized

The environmental variables are: export GRAPHRAG_LLM_MODEL="llama3:70b-instruct" export GRAPHRAG_EMBEDDING_MODEL="nomic-embed-text" export GRAPHRAG_LLM_API_BASE="http://localhost:11434/v1" export GRAPHRAG_EMBEDDING_API_BASE="http://localhost:11434/api"

Any suggestion is appreciated!

Steps to reproduce

No response

Expected Behavior

No response

GraphRAG Config Used

encoding_model: cl100k_base skip_workflows: [] llm: api_key: ${GRAPHRAG_API_KEY} type: openai_chat # or azure_openai_chat model: llama3:70b-instruct-q5_K_M #可以换成其他模型 model_supports_json: true # recommended if this is available for your model.

max_tokens: 4000

request_timeout: 180.0

api_base: http://localhost:11434/v1

api_version: 2024-02-15-preview

organization:

deployment_name:

tokens_per_minute: 150_000 # set a leaky bucket throttle

requests_per_minute: 10_000 # set a leaky bucket throttle

max_retries: 1

max_retry_wait: 10.0

sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times

concurrent_requests: 1 # the number of parallel inflight requests that may be made

parallelization: stagger: 0.3

num_threads: 50 # the number of threads to use for parallel processing

async_mode: threaded # or asyncio

embeddings:

parallelization: override the global parallelization settings for embeddings

async_mode: threaded # or asyncio llm: api_key: ${GRAPHRAG_API_KEY} type: openai_embedding # or azure_openai_embedding model: nomic-embed-text #号称现在最厉害的embedding模型 api_base: http://localhost:11434/api

api_version: 2024-02-15-preview

# organization: <organization_id>
# deployment_name: <azure_model_deployment_name>
# tokens_per_minute: 150_000 # set a leaky bucket throttle
# requests_per_minute: 10_000 # set a leaky bucket throttle
max_retries: 3
# max_retry_wait: 10.0
# sleep_on_rate_limit_recommendation: true # whether to sleep when azure suggests wait-times
#concurrent_requests: 1 # the number of parallel inflight requests that may be made
batch_size: 4 # the number of documents to send in a single request
batch_max_tokens: 2048 # the maximum number of tokens to send in a single request
# target: required # or optional

chunks: size: 4000 overlap: 500 group_by_columns: [id] # by default, we don't allow chunks to cross documents

input: type: file # or blob file_type: text # or csv base_dir: "input" file_encoding: utf-8 file_pattern: ".*\.txt$"

cache: type: file # or blob base_dir: "cache"

connection_string:

container_name:

storage: type: file # or blob base_dir: "output/${timestamp}/artifacts"

connection_string:

container_name:

reporting: type: file # or console, blob base_dir: "output/${timestamp}/reports"

connection_string:

container_name:

entity_extraction:

llm: override the global llm settings for this task

parallelization: override the global parallelization settings for this task

async_mode: override the global async_mode settings for this task

prompt: "prompts/entity_extraction.txt" entity_types: [organization,person,geo,event] max_gleanings: 0

summarize_descriptions:

llm: override the global llm settings for this task

parallelization: override the global parallelization settings for this task

async_mode: override the global async_mode settings for this task

prompt: "prompts/summarize_descriptions.txt" max_length: 500

claim_extraction:

llm: override the global llm settings for this task

parallelization: override the global parallelization settings for this task

async_mode: override the global async_mode settings for this task

enabled: true

prompt: "prompts/claim_extraction.txt" description: "Any claims or facts that could be relevant to information discovery." max_gleanings: 0

community_report:

llm: override the global llm settings for this task

parallelization: override the global parallelization settings for this task

async_mode: override the global async_mode settings for this task

prompt: "prompts/community_report.txt" max_length: 2000 max_input_length: 8000

cluster_graph: max_cluster_size: 5

embed_graph: enabled: false # if true, will generate node2vec embeddings for nodes

num_walks: 10

walk_length: 40

window_size: 2

iterations: 3

random_seed: 597832

umap: enabled: false # if true, will generate UMAP embeddings for nodes

snapshots: graphml: true raw_entities: false top_level_nodes: false

local_search:

text_unit_prop: 0.5

community_prop: 0.1

conversation_history_max_turns: 5

top_k_mapped_entities: 10

top_k_relationships: 10

max_tokens: 12000

global_search:

max_tokens: 12000

data_max_tokens: 12000

map_max_tokens: 1000

reduce_max_tokens: 2000

concurrency: 32

Logs and screenshots

No response

Additional Information

s106916 commented 1 month ago

this is a temp hacked solution for ollama https://github.com/s106916/graphrag

hongyispace commented 1 month ago

this is a temp hacked solution for ollama https://github.com/s106916/graphrag

Thank you for replaying. I succeed in finishing the pipeline, but I am looking for solution for the bug when query with local models.

dengkeshun commented 1 month ago

Hi @hongyispace Did you find a solution to fix the bug when local query with local model?

dengkeshun commented 1 month ago

Hi I have hacked it via changing the code in graphrag/query/llm/oai/openai.python "chunck_text" method: (remove token encode, because the local llm just can embed text instead of encoded ones)

def chunk_text( text: str, max_tokens: int, token_encoder: tiktoken.Encoding | None = None ): """Chunk text by token length."""

if token_encoder is None:

#     token_encoder = tiktoken.get_encoding("cl100k_base")
# tokens = token_encoder.encode(text)  # type: ignore
# chunk_iterator = batched(iter(tokens), max_tokens)
chunk_iterator = batched(iter(text), max_tokens)
yield from chunk_iterator
hongyispace commented 1 month ago

Hi I have hacked it via changing the code in graphrag/query/llm/oai/openai.python "chunck_text" method: (remove token encode, because the local llm just can embed text instead of encoded ones)

def chunk_text( text: str, max_tokens: int, token_encoder: tiktoken.Encoding | None = None ): """Chunk text by token length.""" # if token_encoder is None: # token_encoder = tiktoken.get_encoding("cl100k_base") # tokens = token_encoder.encode(text) # type: ignore # chunk_iterator = batched(iter(tokens), max_tokens) chunk_iterator = batched(iter(text), max_tokens) yield from chunk_iterator

I tried to change "raphrag/query/llm/text_utils.py" wihch contain "chunk_text". New error is as follows: INFO: Reading settings from pg18v8/settings.yaml creating llm client with {'api_key': 'REDACTED,len=56', 'type': "openai_chat", 'model': 'llama3:70b-instruct-q5_K_M', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/v1', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': True, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 1, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 1} creating embedding llm client with {'api_key': 'REDACTED,len=56', 'type': "openai_embedding", 'model': 'nomic-embed-text', 'max_tokens': 4000, 'request_timeout': 180.0, 'api_base': 'http://localhost:11434/api', 'api_version': None, 'organization': None, 'proxy': None, 'cognitive_services_endpoint': None, 'deployment_name': None, 'model_supports_json': None, 'tokens_per_minute': 0, 'requests_per_minute': 0, 'max_retries': 3, 'max_retry_wait': 10.0, 'sleep_on_rate_limit_recommendation': True, 'concurrent_requests': 25} Error embedding chunk {'OpenAIEmbedding': "'NoneType' object is not iterable"} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/main.py", line 75, in run_local_search( File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/cli.py", line 154, in run_local_search result = search_engine.search(query=query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/structured_search/local_search/search.py", line 118, in search context_text, context_records = self.context_builder.build_context( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/structured_search/local_search/mixed_context.py", line 139, in build_context selected_entities = map_query_to_entities( ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/context_builder/entity_extraction.py", line 55, in map_query_to_entities search_results = text_embedding_vectorstore.similarity_search_by_text( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/vector_stores/lancedb.py", line 118, in similarity_search_by_text query_embedding = text_embedder(text) ^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/context_builder/entity_extraction.py", line 57, in text_embedder=lambda t: text_embedder.embed(t), ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/graphrag/query/llm/oai/embedding.py", line 96, in embed chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/grag/lib/python3.12/site-packages/numpy/lib/function_base.py", line 550, in average raise ZeroDivisionError( ZeroDivisionError: Weights sum to zero, can't be normalized

karthik-codex commented 1 month ago

The local search with embeddings from Ollama now works. You can read full guide here: https://medium.com/@karthik.codex/microsofts-graphrag-autogen-ollama-chainlit-fully-local-free-multi-agent-rag-superbot-61ad3759f06f Here is the link to the repo: https://github.com/karthik-codex/autogen_graphRAG

natoverse commented 1 month ago

Consolidating alternate model issues here: https://github.com/microsoft/graphrag/issues/657