search

dusty-nv / jetson-containers

Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
MIT License
1.88k stars 416 forks source link

MLC-LLM version deprecated? #552

Open oglok opened 3 weeks ago

oglok commented 3 weeks ago

Hello @dusty-nv !

I'm having a hard time trying to update a demo that I have based on your mlc-llm container. I moved to JP6 and the server I wrote to use a quantized version of Vicuna seems to be broken for some lib incompatibility.

I opened a bug to the mlc-ai team, but I'm a bit stuck at the moment: https://github.com/mlc-ai/mlc-llm/issues/2512

how did you get those wheels? did you build them yourself? I can't install newer versions from here: https://mlc.ai/wheels as it seems they are not compatible (of course, the cuda ones are built for x86!!)

Do you have any clue about what's going on?

Thank you so much in advance!

dusty-nv commented 3 weeks ago

Hi @oglok, yes jetson-containers builds those wheels, see my reply in the other thread here: https://github.com/mlc-ai/mlc-llm/issues/2512#issuecomment-2152427360