search

NVIDIA-AI-IOT / jetson-copilot

A reference application for a local AI assistant with LLM and RAG
Apache License 2.0
65 stars 9 forks source link

Does it work with Jetson Xavier NX? #2

Open shivamordanny opened 2 weeks ago

shivamordanny commented 2 weeks ago

What all add ons I would need to make it work with Xavier NX?

tokk-nv commented 2 weeks ago

Hi shivamordanny, There is JetPack 5 version of the Docker container. You can edit this line on the launch script for now (replace jetrag:r36.3.0 with jetrag:r35.4.1) and give a try, after you make sure you are on JetPack 5. https://github.com/NVIDIA-AI-IOT/jetson-copilot/blob/main/launch_jetson_copilot.sh#L40

tokk-nv commented 2 weeks ago

Hi shivamordanny, I just updated the launch scripts, so you don't need to manually edit them. Please go ahead and try. A quick test on my Jetson Xavier NX running JetPack 5.1.1 shows that Jetson Copilot runs, however, the Ollama seemed to run without GPU acceleration so it was running very slow. I will look into this issue separately.

tokk-nv commented 2 weeks ago

So, Ollama server in the jetrag container was configured alright. It was just the tight memory space of Xavier NX allowed only the portion of Llama3 model loaded on the GPU memory, effectively making it spend a long time on CPU.

Here are some of the workaround you can try.

wyzeguyz44 commented 3 days ago

Thank you Tokk for doing this I was going to ask you the same questions for the Xavier. Dustin successfully backported the Ai studio tool to r35.4 for the Xavier and I will be testing next week. I definitely was hoping to do the same for copilot on my Xavier