Open arnocandel opened 1 year ago
Is this possible yet?
waiting
If you are looking for a fast inference docker engine, I'd recommend hugging faces text generation inference server instead. It's continuous batching makes it far superior to nvidia triton.
We provide documentation related to this here: https://github.com/h2oai/h2ogpt/blob/main/docs/README_InferenceServers.md#docker-install
@pseudotensor Thanks, I'll try to test what you said
https://github.com/h2oai/h2ogpt/blob/main/docs/TRITON.md
do same for Falcon 7B, then Falcon 40B