search

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
https://nvidia.github.io/TensorRT-LLM
Apache License 2.0
8.61k stars 978 forks source link

Deepseek moe support #1277

Open akhoroshev opened 8 months ago

akhoroshev commented 8 months ago

Hi!

Does trtllm support https://huggingface.co/deepseek-ai/deepseek-moe-16b-base ? Do you have any plans to support?

akhoroshev commented 5 months ago

I implemented this arch several times ago in trtllm

https://github.com/NVIDIA/TensorRT-LLM/pull/1758

trebladev commented 4 months ago

hi， Thanks for your implementation that I can run deepseek-moe with trt-llm But when I try to run it with 2x4090, got a attributeError: ‘PluginNode’ object has no attribute ‘_collect_strategies'/ Have you ever seem this? and do you know the solution?

akhoroshev commented 4 months ago

Have you ever seem this? and do you know the solution?

No, I have not encountered such an error