search

johnsmith0031 / alpaca_lora_4bit

MIT License
534 stars 84 forks source link

Suggestion: make bitsandbytes optional #85

Open mcmonkey4eva opened 1 year ago

mcmonkey4eva commented 1 year ago

If I'm not mistaken, bitsandbytes is only used for this one line: https://github.com/johnsmith0031/alpaca_lora_4bit/blob/main/monkeypatch/peft_tuners_lora_monkey_patch.py#L125

Meaning, you can move the import to within that if loaded_in_8bit, and thus make bitsandbytes only required when 8bit models are loaded.

I suggest this because on the text-gen-webui Discord I've been hearing so many users complain about the difficulty of getting bitsandbytes installed on Windows, and that's been renewed by monkey-patch requiring it... but it should be simple to remove the explicit dependency, if I'm not missing anything.

johnsmith0031 commented 1 year ago

Yes I think I'll remove it from the monkey patch. Thanks for the advice