Open FurkanGozukara opened 2 weeks ago
Hi @FurkanGozukara, thanks for the report ! I've opened this draft PR for you to try ! This should have been solved with bnb>=0.43.0 with 4-bit models but it wasn't upstreamed to transformers. Let me know how it goes ! https://github.com/huggingface/transformers/pull/33122
Hi @FurkanGozukara, thanks for the report ! I've opened this draft PR for you to try ! This should have been solved with bnb>0.43.0 with 4-bit models but it wasn't upstreamed to transformers. Let me know how it goes ! #33122
awesome i will test later
i made a different implementation and made it work with 4bit :D it was really hard
I am developing an very advanced multi-GPU batch captioning APP
The below code works when I dont use quantization_config=quantization_config because i am able to set
.to(device)
but when
quantization_config=quantization_config
is used it doesn't allow me to set.to(device)
Any ideas ?
When
quantization_config=quantization_config
is set the error i got isYou shouldn't move a model that is dispatched using accelerate hooks.
Error processing image R:\Joy_Caption_v1\outputs\1_3x_Ultimate_Fidelity_Standard_Texture_2_Creativity_0.png on GPU 0:
.tois not supported for
4-bitor
8-bitbitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct
dtype.
If I remove
.to
part then the multi GPU part fails :/The entire code is below it is not very big 460 lines I want to be able to run 8 different captionioning on 8 different GPUs
Who can help?
@SunMarc @Narsil