search

intel-analytics / text-generation-webui

A Gradio Web UI for running local LLM on Intel GPU (e.g., local PC with iGPU, discrete GPU such as Arc, Flex and Max) using IPEX-LLM.
GNU Affero General Public License v3.0
14 stars 8 forks source link

Fix load_in_4bit and load_in_low_bit not taking effect in UI #28

Closed hkvision closed 5 months ago

hkvision commented 5 months ago

Currently with variable names ipex_llm_load_in_4bit or ipex_llm_load_in_low_bit, when modifying their values in the UI (unchecking load_in_4bit or change values for low-bit), the modified values won't take effect in the actual code. Seems in this way the values can't be modified after the webui is launched (either with the argument value or the default value if not specified).

Changing the variable name resolves the problem.

Using the same argument --load-in-4bit as bitsandbytes.