Closed sigmareaver closed 1 year ago
Not sure what I did differently, but it started suggesting qweight now...
AttributeError: 'QuantLinear' object has no attribute 'weight'. Did you mean: 'qweight'?
My apologies. It seems a requirement was somehow not installed, or overwritten with your transformers-t5 repo.
i7-13700k 128GB RAM RTX 4090
Python = 3.9.10 Transformers = 4.30.0.dev0 PyTorch = 2.0.1 Model = Google/flan-ul2
Quantization command:
I also needed to edit
t5_sequential()
to run on 24GB of VRAM, but I don't think this should affect the model? The following snippet shows the extent of my changes, except forimport gc
at the top of the file.Otherwise my 4090 runs out of memory when trying to load
model.encoder.final_layer_norm = model.encoder.final_layer_norm.to(dev)
to the GPU.Benchmark command (also applies to t5_inference.py):
Yields the following error:
Edit: added snippet showing code modifications, and edited quantization command to show PYTORCH_CUDA_ALLOC_CONF environment variable.