search

mistralai / mistral-inference

Official inference library for Mistral models
https://mistral.ai/
Apache License 2.0
9.77k stars 874 forks source link

Fix device error when using cuda device other than cuda:0 #216

Open cornzz opened 3 months ago

cornzz commented 3 months ago

Fixes https://github.com/mistralai/mistral-inference/issues/215

Attention bias was being created on cuda:0 regardless of the selected cuda device as the correct device was not being passed to from_seqlens() in BufferCache.get_input_metadata()