Lava demo fails to run on AGX Orin with "AssertionError: Model type llava not supported."

[sina-sixwheel]

I am attempting to run the Lava demo on AGX Orin but quantaztion step seems to fail with "AssertionError: Model type llava not supported. Is there a dependency issue that I can fix with my setup? It is pulling the container. `"

Commit sha:

nvidia@nvidia:/external/jetson-containers$ git log
commit 250292c6f8e4c571d6127a3b645d1815d9fd9863 (HEAD -> master, origin/master, origin/HEAD)
Author: Dustin Franklin <dustinf@nvidia.com>
Date:   Wed Jan 31 22:22:34 2024 -0500

Full logs here Command ./run.sh -e SSL_KEY=/data/key.pem -e SSL_CERT=/data/cert.pem $(./autotag local_llm) python3 -m local_llm.agents.video_query --api=mlc --verbose --model liuhaotian/llava-v1.5-7b --max-new-tokens 32 --video-input /dev/video5 --video-output webrtc://@:8554/output --prompt "How many fingers am I holding up?"

Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/usr/local/lib/python3.8/dist-packages/mlc_llm/build.py", line 47, in <module> main() File "/usr/local/lib/python3.8/dist-packages/mlc_llm/build.py", line 41, in main parsed_args = core._parse_args(parsed_args) # pylint: disable=protected-access File "/usr/local/lib/python3.8/dist-packages/mlc_llm/core.py", line 444, in _parse_args parsed = _setup_model_path(parsed) File "/usr/local/lib/python3.8/dist-packages/mlc_llm/core.py", line 494, in _setup_model_path validate_config(args.model_path) File "/usr/local/lib/python3.8/dist-packages/mlc_llm/core.py", line 537, in validate_config assert ( AssertionError: Model type llava not supported.

[dusty-nv]

@sina-sixwheel hmm that model type: llava should automatically have been changed by the runtime. Try doing this manually:

[!WARNING]
Patch the model's config.json that was downloaded under data/models/huggingface/models--liuhaotian--llava-v1.5-13b/snapshots/*

• modify "model_type": "llava",
• to "model_type": "llama",
  

Then re-run the command above - the quantization tools will then treat it like a Llama model (which it is)

[snejati86]

@dusty-nv that did it, still having issues with webrtc crashing but separate issue I will investigate thanks!

[dusty-nv]

OK gotcha - if having WebRTC problems, you should be able to attach display directly to Jetson and view the video through the ubuntu UI with --video-output display://0 or for debugging WebRTC, sometimes viewing the browser developer console (Ctrl+Shift+I on Chrome) can have more error messages.