Open BlueprintCoding opened 3 months ago
I also tested if this fix adversely effects Windows installs of Text Web UI. I found no issues in my testing with the change.
I also run into this problem whenever I try to use "logit_bias" field in OpenAI API call. In my case it gives error for both llama.cpp and ExLlamav2 loaders. I also tried to run GGUF model on CPU, but got the same error. So I think the problem is more general than just GGUF models on GPU.
Describe the bug
I recently ran into an issue while using the OogaBooga Text Generation Web UI with some GGUF models. The models loaded fine into Text Gen Web UI, but when I tried to use the OpenAI API extension for text completions via SillyTavern, I got an error saying the LlamaCppModel object had no attribute device. After some troubleshooting, I figured out how to fix it. This may be hyperspecific to my hardware and software version but I did not find any other information about this on the web so I figured I'd make a post about it.
I was able to fix this error by modifying the "LogitsBiasProcessor" function in the /extensions/openai/completions.py Changing line 36:
to the following:
This fix checks if the shared.model object has a device attribute before attempting to use it. If the device attribute exists, the code creates a tensor with the specified values on the given device (e.g., a GPU). If the device attribute does not exist, it defaults to creating the tensor without specifying a device, which typically means it will be created on the CPU. This ensures compatibility with models that do not have the device attribute, preventing the AttributeError and allowing the code to run smoothly.
Is there an existing issue for this?
Reproduction
Screenshot
Text Web UI Model Settings
Logs
System Info