Closed tkone2018 closed 5 months ago
还有就是请教下vllm推理是不是不支持ntk上下文扩展技术
你max_new_token
设置的太大,参数不对。而且NTK不是无限扩展,比如原来支持4k长度,NTK可扩展到8-12k。
@iMountTai 好的,谢谢
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your consideration.
Closing the issue, since no updates observed. Feel free to re-open if you need any further assistance.
提交前必须检查以下项目
问题类型
模型推理
基础模型
Chinese-Alpaca-2 (7B/13B)
操作系统
Linux
详细描述问题
generation_config = GenerationConfig( temperature=0.2, top_k=40, top_p=0.9, do_sample=True, num_beams=1, repetition_penalty=1.1, max_new_tokens=30000 ) 推理文件是inference_hf.py,采用ntk扩展,想问下是不是输入还是限制在4096了,请问下ntk是可以扩展输入的上下文吗,还是说我使用ntk后,还需要修改哪些部分才能扩展输入的上下文token数,谢谢
依赖情况(代码类问题务必提供)
运行日志或截图