Closed KylinMountain closed 1 month ago
@AlonsoGuevara Look into the prompt of MAP_SYSTEM_PROMPT of global query, I think we should optimize it, the example is not clearly, which may cause it won't generate JSON reply in stable. I will try to pull a request to do this.
I initially had problems with the models outputting more than just JSON, un-handy descriptors or irrelevant information, such as "Here is the JSON you requested: etc..". Modified the json scrubbing in llm/openai/_json.py to shrink the string into the first and last brackets found. Likely error-prone on larger datasets and/or models other than what I'm using, but it works well enough for what I need at the moment:
def clean_up_json(json_str: str) -> str:
"""Clean up json string."""
json_str = (
json_str.replace("\\n", "")
.replace("\n", "")
.replace("\r", "")
.replace('"[{', "[{")
.replace('}]"', "}]")
.replace("\\", "")
.strip()
)
# Constrain string to brackets regardless of content
open_brackets = __find_all(json_str, "{")
close_brackets = __find_all(json_str, "}")
json_str = json_str[min(open_brackets):max(close_brackets)+1]
# Remove JSON Markdown Frame
if json_str.startswith("```json"):
json_str = json_str[len("```json") :]
if json_str.endswith("```"):
json_str = json_str[: len(json_str) - len("```")]
return json_str
# https://stackoverflow.com/a/4665027
def __find_all(string, substring):
start = 0
while True:
start = string.find(substring, start)
if start == -1: return
yield start
start += len(substring)
I'm hitting this bug.
Update:
meta-llama/Llama-3-8b-chat-hf : causes this error.
Qwen/Qwen2-72B-Instruct : empty response, no error reported.
Should I open another issue?
I was able to use Claude 3 haiku with my proxy(https://github.com/jaigouk/claude-proxy-api) on k3s with 3 replica and graphrag/query/structured_search/global_search/search.py
needs to be improved as @Ech0riginal said.
I tested my setup and I was able to get the response with global search.
https://github.com/microsoft/graphrag/pull/545
My settings.yaml is
llm:
api_key: ${CLAUDE_PROXY_API_KEY}
type: openai_chat
model_supports_json: true
model: "claude-3-haiku-20240307"
api_base: "http://192.168.8.213:30012/v1"
# max_tokens: 10000 # Adjusted based on Claude 3 Haiku's typical context window
request_timeout: 30
tokens_per_minute: 100000
requests_per_minute: 1000
max_retry_wait: 5
temperature: 0.1
embeddings:
async_mode: threaded
llm:
api_key: ${EMBEDDING_API_KEY}
type: openai_embedding
model: "BAAI/bge-m3"
api_base: "http://localhost:7997"
I am using https://github.com/michaelfeil/infinity for embeddings. with 8k text tokens, the indexing time takes about 1 min. without model_supports_json: true
it takes about 3 min.
Consolidating alternate model issues here: #657
Describe the bug
When I try to search using global query, it report json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0).
And the response missing a paragraph which is not generated with JSON formatted.
Steps to reproduce
using gemma 9b as llm model as confined as the following.
Expected Behavior
It should be able to query successfully and not miss any content which is not formatted by JSON.
GraphRAG Config Used
Logs and screenshots
Additional Information