search

dusty-nv / jetson-inference

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
https://developer.nvidia.com/embedded/twodaystoademo
MIT License
7.86k stars 2.98k forks source link

Want to run a Local LLM on Nvidia Jetson AGX Orin over GPU #1877

Open MausamJain opened 4 months ago

MausamJain commented 4 months ago

I am looking to run a local LLM (Large Language Model) on an Nvidia Jetson AGX Orin over the GPU CUDA Cores . Could anyone provide guidance or share resources on how to achieve this?

I was able to run a local LLM (.gguf model) over the CPU but unable to utilize the GPU.

Thank you in advance for your help!