Open linuxmagic-mp opened 11 months ago
llm(...)
doesn't return until the entire text is generated whereas llm.generate(...)
sends tokens one-by-one as they get generated.
Is it exiting without error and without printing anything? Try using stream=True
:
for text in llm(prompt, stream=True):
print(text)
Just could use some feedback on debugging with ctransformers, have a strange case where things are generally working, but occasionally I don't get output... using /models/WizardLM-Uncensored-Falcon-40b/ggml-model-falcon-40b-wizardlm-qt_k5.bin (GGML)
works always..
Sometimes there is NO output.
Scratching my head on how to debug this?