why Dit does not support pp_size > 1

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

https://nvidia.github.io/TensorRT-LLM

Apache License 2.0

8.74k stars 1k forks source link

why Dit does not support pp_size > 1 #2427

Open algorithmconquer opened 2 weeks ago

algorithmconquer commented 2 weeks ago

In convert_checkpoint.py, the code is 'assert args.pp_size == 1, "PP is not supported yet."', why dit does not support pp_size > 1?

ChunhuanLin commented 2 weeks ago

We thought DiT requeires lower latency rather than higher throughput. TP/CP can possibly reduce the model inference latency, however PP is not help with latency. That's the reason why we didn't support PP for DiT now.