search

randaller / llama-chat

Chat with Meta's LLaMA models at home made easy
GNU General Public License v3.0
834 stars 118 forks source link

how can I use cuda tensor #11

Closed cha0s-repo closed 1 year ago

cha0s-repo commented 1 year ago

Hi randaller, I tried to use cuda tensor type by uncommenting this line "torch.set_default_tensor_type(torch.cuda.HalfTensor)" in example. But got error like this: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument index in method wrapper__index_select)

My question is how can I use cuda type within your example? Thanks in advance. Kyle

randaller commented 1 year ago

@cha0s-repo fp16 tensors is not supported by pytorch on CPU.