could not load the model

[hui-max]

I use llama.cpp to load the llama-3-8B locally, I test the model with llama.cpp project, It can run well. But when I run the src/setup_hipporag.sh file, I got those result, I don't know why? I monitor the running process, the usage of GPU is 0%

[yhshu]

Hi, I would suggest printing llama3's output here to see if it produces the proper output. It would be more appropriate for us to find the problem if you could provide the full commands being executed here. In addition, we have not tested the quantization to Q4. I'm not sure Q4 will be sufficient for our task requirement.

[hui-max]

Thanks for your reply, I could not print the llama3's output, becaues the code stopped at this line of code, at file src/openie_with_retrieval_option_parallel.py my executed commands is below:

data=sample
retriever_name=/HOME/scw6dlp/run/HippoRAG/contriever
extraction_model=/HOME/scw6dlp/run/HippoRAG/llama.cpp/models/Lexi-Llama-3-8B-Uncensored_Q4_K_M.gguf  # LLM using LangChain, e.g., 'gpt-3.5-turbo' or 'meta-llama/Llama-3-8b-chat-hf'
available_gpus=0,1
syn_thresh=0.8
llm_api=llama.cpp # e.g., 'openai', 'together'
extraction_type=ner

# Running Open Information Extraction
python src/openie_with_retrieval_option_parallel.py --dataset $data --llm $llm_api --model_name $extraction_model --run_ner --num_passages all # NER and OpenIE for passages
python src/named_entity_extraction_parallel.py --dataset $data --llm $llm_api --model_name $extraction_model  # NER for queries

# Creating Contriever Graph
python src/create_graph.py --dataset $data --model_name $retriever_name --extraction_model $extraction_model --threshold $syn_thresh --extraction_type $extraction_type --cosine_sim_edges

# Getting Nearest Neighbor Files
CUDA_VISIBLE_DEVICES=$available_gpus python src/RetrievalModule.py --retriever_name $retriever_name --string_filename output/query_to_kb.tsv
CUDA_VISIBLE_DEVICES=$available_gpus python src/RetrievalModule.py --retriever_name $retriever_name --string_filename output/kb_to_kb.tsv
CUDA_VISIBLE_DEVICES=$available_gpus python src/RetrievalModule.py --retriever_name $retriever_name --string_filename output/rel_kb_to_kb.tsv

python src/create_graph.py --dataset $data --model_name $retriever_name --extraction_model $extraction_model --threshold $syn_thresh --create_graph --extraction_type $extraction_type --cosine_sim_edges

[yhshu]

I think that's the problem. We need to find out why the loop is not executing, is it because triple_json is empty? If so, maybe the previous step before this script has already gone wrong.

[hui-max]

Apparently, the args is the triple_json, the args is below, it's not empty. num_processes is 10, so the execue code is outputs = pool.map(extract_openie_from_triples, args)

args=[[(0, {'title': 'Thomas C. Sudhof', 'text': "Introduction\nThomas Christian Sudhof (German pronunciation: ['to:mas 'zy:t,ho:f] i; born December 22, 1955), ForMemRS, is a German-American biochemist known for his study of synaptic transmission. Currently, he is a professor in the school of medicine in the department of molecular and cellular physiology, and by courtesy in neurology, and in psychiatry and behavioral sciences at Stanford University.", 'idx': 0, 'passage': "Thomas C. Sudhof\nIntroduction\nThomas Christian Sudhof (German pronunciation: ['to:mas 'zy:t,ho:f] i; born December 22, 1955), ForMemRS, is a German-American biochemist known for his study of synaptic transmission. Currently, he is a professor in the school of medicine in the department of molecular and cellular physiology, and by courtesy in neurology, and in psychiatry and behavioral sciences at Stanford University."})], [(1, {'title': 'David Eagleman', 'text': 'Introduction\nDavid Eagleman (born April 25, 1971) is an American neuroscientist, author, and science communicator. He teaches neuroscience at Stanford University[1] and is CEO and co-founder of Neosensory, a company that develops devices for sensory substitution. [2] He also directs the non-profit Center for Science and Law, which seeks to align the legal system with modern neuroscience[3] and is Chief Science Officer and co-founder of BrainCheck, a digital cognitive health platform used in medical practices and health systems.', 'idx': 1, 'passage': 'David Eagleman\nIntroduction\nDavid Eagleman (born April 25, 1971) is an American neuroscientist, author, and science communicator. He teaches neuroscience at Stanford University[1] and is CEO and co-founder of Neosensory, a company that develops devices for sensory substitution. [2] He also directs the non-profit Center for Science and Law, which seeks to align the legal system with modern neuroscience[3] and is Chief Science Officer and co-founder of BrainCheck, a digital cognitive health platform used in medical practices and health systems.'})], [(2, {'title': 'Karl Deisseroth', 'text': 'Introduction\nKarl Alexander Deisseroth (born November 18, 1971) is an American scientist. He is the D.H. Chen Foundation Professor of Bioengineering and of psychiatry and behavioral sciences at Stanford University. He is known for creating and developing the technologies of hydrogel-tissue chemistry (e.g., CLARITY, STARmap) and optogenetics, and for applying integrated optical and genetic strategies to study normal neural circuit function, as well as dysfunction in neurological and psychiatric disease.', 'idx': 2, 'passage': 'Karl Deisseroth\nIntroduction\nKarl Alexander Deisseroth (born November 18, 1971) is an American scientist. He is the D.H. Chen Foundation Professor of Bioengineering and of psychiatry and behavioral sciences at Stanford University. He is known for creating and developing the technologies of hydrogel-tissue chemistry (e.g., CLARITY, STARmap) and optogenetics, and for applying integrated optical and genetic strategies to study normal neural circuit function, as well as dysfunction in neurological and psychiatric disease.'})], [], [], [], [], [], [], []]