search

dusty-nv / jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
MIT License
2.36k stars 481 forks source link

Can we have vllm jetson images? #635

Closed PredyDaddy closed 2 months ago

PredyDaddy commented 2 months ago

Hello Dusty, Can I know if we will have the vllm jetson images?

johnnynunez commented 2 months ago

Super https://x.com/_akhaliq/status/1836232050443006020?s=46

PredyDaddy commented 2 months ago

Super https://x.com/_akhaliq/status/1836232050443006020?s=46

looks great! Can this vlm do open word object detection task?

dusty-nv commented 2 months ago

@PredyDaddy this is not vLLM (like the HF inference serving engine), that is not supported on aarch64+iGPU.

The NVLM looks promising, great paper!, but is a 72B model not super applicable to deploying onboard edge devices. For that reason I don't plan special quantization support for it or dedicated container for Jetson, but you may be able to use it through HF Transformers when the weights come out (although without quantization, even AGX Orin 64GB would not be able to load)

PredyDaddy commented 2 months ago

Thanks~