Open ParetoOptimalDev opened 11 months ago
This piece:
File "/nix/store/ypar8406iyb6r22n755ygvfbplwjs050-textgen-patchedSrc/modules/llamacpp_model.py", line 91, in from_pretrained
result.model = Llama(**params)
TypeError: Llama.__init__() got an unexpected keyword argument 'n_threads_batch'
is odd because I see n_threads_batch
in that file:
params = {
# ...
'n_threads': shared.args.threads or None,
'n_threads_batch': shared.args.threads_batch or None,
# ...
}
This piece:
File "/nix/store/ypar8406iyb6r22n755ygvfbplwjs050-textgen-patchedSrc/modules/llamacpp_model.py", line 91, in from_pretrained
result.model = Llama(**params)
TypeError: Llama.__init__() got an unexpected keyword argument 'n_threads_batch'
is odd because I see n_threads_batch
in that file:
params = {
# ...
'n_threads': shared.args.threads or None,
'n_threads_batch': shared.args.threads_batch or None,
# ...
}
I also have this installed on a mac computer without Nix and know that with whatever those version are, this model loads fine.
I think models are just not loading for me period. Is there some known working model I should be able to use? I tried:
TheBloke/orca_mini_v3_7B-GGUF
It downloaded fine but I get this error when trying to load with
llama.cpp
loader: