search

oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.
GNU Affero General Public License v3.0
40.58k stars 5.31k forks source link

Request to Update Exllamav2 Module to version 0.1.8 #6266

Closed GrennKren closed 3 months ago

GrennKren commented 3 months ago

Description

I would like to request an update to the Exllamav2 module to version 0.1.8. This version includes some various bug fixes like https://github.com/turboderp/exllamav2/issues/566

Additional Context For reference, the changes in version 0.1.8 are as follows:

St33lMouse commented 3 months ago

I'd like to second that motion.

jepjoo commented 3 months ago

It's in dev branch.

https://github.com/oobabooga/exllamav2/releases/download/v0.1.8/exllamav2-0.1.8+cu121.torch2.2.2-cp311-cp311-win_amd64.whl; platform_system == "Windows" and python_version == "3.11" https://github.com/oobabooga/exllamav2/releases/download/v0.1.8/exllamav2-0.1.8+cu121.torch2.2.2-cp310-cp310-win_amd64.whl; platform_system == "Windows" and python_version == "3.10" https://github.com/oobabooga/exllamav2/releases/download/v0.1.8/exllamav2-0.1.8+cu121.torch2.2.2-cp311-cp311-linux_x86_64.whl; platform_system == "Linux" and platform_machine == "x86_64" and python_version == "3.11" https://github.com/oobabooga/exllamav2/releases/download/v0.1.8/exllamav2-0.1.8+cu121.torch2.2.2-cp310-cp310-linux_x86_64.whl; platform_system == "Linux" and platform_machine == "x86_64" and python_version == "3.10" https://github.com/oobabooga/exllamav2/releases/download/v0.1.8/exllamav2-0.1.8-py3-none-any.whl; platform_system == "Linux" and platform_machine != "x86_64"

oobabooga commented 3 months ago

That's correct, today I have compiled it for PyTorch 2.2.2 and updated the requirements.txt in the dev branch. Llama-3.1 EXL2 is functional and I have benchmarked it. I'm still testing other unrelated things but tomorrow I should release a new version.