tensorrt-llm: 0.9 -> 0.10, triton: 2.42.0 -> 2.44.0

replicate / cog-triton

A cog implementation of Nvidia's Triton server

Apache License 2.0

11 stars 0 forks source link

Open yorickvP opened 1 month ago

yorickvP commented 1 month ago

Open questions:

[x] Should we keep detokenizing in predict.py instead of postprocessing?
[x] It looks like the required config changed. Notably, enable_trt_overlap disappeared and a whole bunch of cache options got added.
[x] Yorick: do some testing.

technillogue commented 4 weeks ago

enable_trt_overlap is set to false in a lot of places, we will probably need to change that

we should review the new configuration options as well