Open zoubaihan opened 1 year ago
Hi @zoubaihan, you can specify parameters such as top_p
and max_tokens
when calling the StreamModel
instance obtained using the load_model
function. However, we haven't implemented a streaming version for all parameters in HF Transformers yet, so parameters like do_sample
are currently not supported.
Here's the full list of supported params: https://github.com/hyperonym/basaran#completions
OK, thank you, I hope one day it could support all parameters of model.generate()
!
Hello, I want use my customize parameters when model.generate(), like this:
but if I use basaran, the code is like this:
It seems no place I can set parameters like
do_sample
,max_length
,top_p
, ..., just like I use model.generate() directly. So that I can not set those parameters by myself. How to solve this problem?