Hello!when I use 16 A10(16 23g) to inference llama2-70b, it appears error:
I ask many people to solve this problem,but failed.
I know 8 gpu can work it! But I need to increase the prompt of llama2, the 8 GPU is not enough!
Do you have some ideas, thanks!
I have 16 gpus in one machine.![image](https://github.com/nicknochnack/Llama2RAG/assets/74184102/64feb252-9405-42ef-bf77-80f77023f0e6)
Hello!when I use 16 A10(16 23g) to inference llama2-70b, it appears error:![image](https://github.com/nicknochnack/Llama2RAG/assets/74184102/131700ec-c4bf-4053-a1f0-97e117c1dd45)
I ask many people to solve this problem,but failed. I know 8 gpu can work it! But I need to increase the prompt of llama2, the 8 GPU is not enough! Do you have some ideas, thanks!