-
RUNNING
```
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
model_path = '/data/bloomz_7b1'
quant_path = '/data/bloomz_7b1_4bit'
quant_config = { "zero_point": True, …
-
### Branch/Tag/Commit
main
### Docker Image Version
nvcr.io/nvidia/pytorch:22.09-py3
### GPU name
V100-32G
### CUDA Driver
11.0
### Reproduced Steps
steps 1: pull images w…
-
I run this script
`deepspeed --num_gpus 1 bloom-inference-scripts/bloom-ds-inference.py --name bigscience/bloomz-7b1 --batch_size 8`
and it gets stuck just like in the picture.
Log:
```
(ba…
-
Add more examples for different models like Bloomz, Dolly, GPT etc.
-
It seems that bloom is not support for quantization right now
```
Traceback (most recent call last):
File "/home/work/vllm-main/scripts/w8a8v2.py", line 40, in
oneshot(
File "/home/bml/.…
-
Hi!
I'm trying to quantize a 3B Bloom Model (https://huggingface.co/bigscience/bloom-3b). But it seems like it's missing the alibi tensor when performing the forward pass of the model.
Could y…
-
Hi guys,
First of all, great video and funny project !
There are 3 ideas that could be nice to implement:
1) Model selector, to be able to easily download and switch between models.
2) Bloom Pet…
-
[bloomz-7b1-mt](https://huggingface.co/bigscience/bloomz-7b1-mt)进行微调),不知道参考具体哪个github资源，麻烦能提供一下吗？
-
I use this repo to finetune bloomz-7b1-mt with alpaca data (50k conversation) and the results are terrible. It takes 8 hours to train with the same arguments as in how you finetune the llama. What cou…
-
Hey hey,
We are working hard to help you unlock the truest potential of open-source LLMs. In order for us to build better and cater to the majority of hardware we need your help to run benchmarks w…