I am trying to use a LLM model that can use GPU , model works fine with CPU but when with gpu it gives following error
Error-Traceback (most recent call last):
File "D:\Artificial Intelligence\summarization.py", line 6, in
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-Chat-GGUF", gpu_layers=50,temperature=0.4,max_new_tokens=4096,context_length=4096)
File "D:\Artificial Intelligence\AI_env\lib\site-packages\ctransformers\hub.py", line 175, in from_pretrained
llm = LLM(
File "D:\Artificial Intelligence\AI_env\lib\site-packages\ctransformers\llm.py", line 246, in init
self._lib = load_library(lib, gpu=config.gpu_layers > 0)
File "D:\Artificial Intelligence\AI_env\lib\site-packages\ctransformers\llm.py", line 126, in load_library
lib = CDLL(path)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\ctypes__init.py", line 374, in init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed
My code - from ctransformers import AutoModelForCausalLM
import time
print('program started')
start_time = time.time()
print('loading model')
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-Chat-GGUF", gpu_layers=50,temperature=0.4,max_new_tokens=4096,context_length=4096)
prompt=""" text
"""
prompts=f"""[INST] <>
text
<>
{prompt}[/INST]
"""
print('promt passed')
print(llm(prompt=prompts))
end_time = time.time()
I am trying to use a LLM model that can use GPU , model works fine with CPU but when with gpu it gives following error Error-Traceback (most recent call last): File "D:\Artificial Intelligence\summarization.py", line 6, in
llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-Chat-GGUF", gpu_layers=50,temperature=0.4,max_new_tokens=4096,context_length=4096)
File "D:\Artificial Intelligence\AI_env\lib\site-packages\ctransformers\hub.py", line 175, in from_pretrained
llm = LLM(
File "D:\Artificial Intelligence\AI_env\lib\site-packages\ctransformers\llm.py", line 246, in init
self._lib = load_library(lib, gpu=config.gpu_layers > 0)
File "D:\Artificial Intelligence\AI_env\lib\site-packages\ctransformers\llm.py", line 126, in load_library
lib = CDLL(path)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python310\lib\ctypes__init.py", line 374, in init__
self._handle = _dlopen(self._name, mode)
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed
My code - from ctransformers import AutoModelForCausalLM import time print('program started') start_time = time.time() print('loading model') llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7B-Chat-GGUF", gpu_layers=50,temperature=0.4,max_new_tokens=4096,context_length=4096) prompt=""" text """ prompts=f"""[INST] <>
text
< >
{prompt}[/INST]
"""
print('promt passed')
print(llm(prompt=prompts))
end_time = time.time()
Calculate the elapsed time
elapsed_time = end_time - start_time
Print the elapsed time
print(f"Program execution time: {elapsed_time} seconds")
thank you