test

[tkone2018]

hello, when i ask question,need always load the model when AI gives a word, is it ok？

[randaller]

Hi @tkone2018, model weights are loaded in RAM once at startup, but then for each token weights are passing from RAM to GPU one by one.