Alpaca 7B and 13B on Linux

[RIAZAHAMMED]

I have a server that has Linux and I installed Dalai on it. I made many modifications in appearance and tried. Even with 70GB RAM this thing became very slow. But I cannot say it is not working... See the attached image you will know what is wrong with it. It was a simple question... When will summer start? The Response was very funny... at leat good for a laugh after 3 hours of processing.

It may need GPU with better processing power. Or I may have to install cmd only server software. Thinking about all the options... Here this will give you an idea about the computer...

Any suggestions will be appreciated.

[mirek190]

3 hours?

With i9 9990 ( 16 threads CPU ) model 7B will be process such amount of text 1-2 min ... I tried also 30B and such amount text would be process around 15 min max.

I have no idea why is so slow for you.

[RIAZAHAMMED]

3 hours?

With i9 9990 ( 16 threads CPU ) model 7B will be process such amount of text 1-2 min ... I tried also 30B and such amount text would be process around 15 min max.

I have no idea why is so slow for you.

I think Graphics is the problem... I have Zorin ... so I may have to just run command only server to avoid the graphics altogether.

[mirek190]

but dalai uses cpu not gpu.

[RIAZAHAMMED]

but dalai uses cpu not gpu.

That's what I thought too.. I used a GPU machine and it runs faster on it.. I removed all the dims and reinstalled it and it got a bit better... I am not an expert with servers or hardware stuff..

see what it did for me in 12 mins

Not bad isn't it? It is not Shakespeare but it can do stuff.. hehehehe

[mirek190]

That's still extremely slow for 24 threads CPU .... Such amount of text with 7b model should be produced within 20 second ..my CPU is 16 threads.maybe this Xeon is very slow compare to i9 9900....

[RIAZAHAMMED]

That's still extremely slow for 24 threads CPU .... Such amount of text with 7b model should be produced within 20 second ..my CPU is 16 threads.maybe this Xeon is very slow compare to i9 9900....

This thing I old... it was sitting in my storage. So I pulled it out and started it to see how this goes. Yeah, you are right probably need to try a different server

[VanHallein]

Very related to the issue I reported here in the forum, please see here: #323. This is the exact problem I am having when using alpaca.7B. It also happens with the larger models. The system, being in that shape, is pretty useless.

[RiccaDS]

That's still extremely slow for 24 threads CPU .... Such amount of text with 7b model should be produced within 20 second ..my CPU is 16 threads.maybe this Xeon is very slow compare to i9 9900....

This thing I old... it was sitting in my storage. So I pulled it out and started it to see how this goes. Yeah, you are right probably need to try a different server

Did you find a solution?

[ADANCE-STACK]

buenas como hago para conseguir (llama versión 65b). (https://huggingface.co/boboto/LLaMA-65B-HF/tree/main) o el torrent. porque estoy rastreando hace rato para instalar y probar encontré los pesos pero me faltan archivos de instalación, si alguno me puede ayudar le agradezco , aunque sea por mje.privado. slot1_@hotmail.com .

[RIAZAHAMMED]

That's still extremely slow for 24 threads CPU .... Such amount of text with 7b model should be produced within 20 second ..my CPU is 16 threads.maybe this Xeon is very slow compare to i9 9900....

This thing I old... it was sitting in my storage. So I pulled it out and started it to see how this goes. Yeah, you are right probably need to try a different server

Did you find a solution?

I am out of town through the end of the month so I am sorry I cannot work on this now... but once I am back in town I will get back to this... I am going to do a full performance tuning of the server and try again...

[RIAZAHAMMED]

buenas como hago para conseguir (llama versión 65b). (https://huggingface.co/boboto/LLaMA-65B-HF/tree/main) o el torrent. porque estoy rastreando hace rato para instalar y probar encontré los pesos pero me faltan archivos de instalación, si alguno me puede ayudar le agradezco , aunque sea por mje.privado. slot1_@hotmail.com .

As I am not familiar with your language I tried translating this... from my understanding, I think you need to follow the instructions in getting dalai installed and then install the models you want.

[titolindj]

I think Dalai doesn't like Xeon's I have the exact same problem...
A decent Build with more than plenty of RAM and an 8 core Xeon, nothing fancy but plenty more power than the average laptop or desktop, and still the output takes forever, even with the simplest questions.