Can you please add Gemma-2 support, I can not run my gemma2 model

dusty-nv / jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T

MIT License

2.37k stars 482 forks source link

Can you please add Gemma-2 support, I can not run my gemma2 model #589

Open dilerbatu opened 3 months ago

dusty-nv commented 3 months ago

Hi @dilerbatu, what is your version of JetPack-L4T? dustynv/ollama:r36.2.0 for JetPack 6 was rebuilt with gemma2 - https://hub.docker.com/r/dustynv/ollama/tags

dilerbatu commented 3 months ago

Hi, the version of JetPack-L4T is 35.5.0. The main thing is I want to run this model inside of nanoLLM which seems faster than other LLM containers.

dusty-nv commented 3 months ago

@dilerbatu yes NanoLLM uses MLC (also supports AWQ and HF backends). Have not tested MLC performance for gemma2. I've been meaning to update the MLC build for Jetson, it requires patching, debugging API changes, and testing a bunch of models. Hope to have time for it at some point, but you can find my previous patches from the last build here: https://github.com/dusty-nv/jetson-containers/blob/master/packages/llm/mlc/patches/3403a4e.diff