GoogleCloudPlatform / localllm

Apache License 2.0
1.51k stars 113 forks source link

Not able to run llama model 3.2 vision #30

Open jkp007 opened 2 days ago

jkp007 commented 2 days ago

Command - local-llm run meta-llama/Llama-3.2-11B-Vision 8000 --verbose

Issues - Not able to download and run the LLM model

Traceback (most recent call last):
  File "/home/ubuntu/.local/bin/uvicorn", line 8, in <module>
    sys.exit(main())
  File "/usr/lib/python3/dist-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3/dist-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3/dist-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/uvicorn/main.py", line 410, in main
    run(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/uvicorn/main.py", line 577, in run
    server.run()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/uvicorn/server.py", line 65, in run
    return asyncio.run(self.serve(sockets=sockets))
  File "/usr/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/uvicorn/server.py", line 69, in serve
    await self._serve(sockets)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/uvicorn/server.py", line 76, in _serve
    config.load()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/uvicorn/config.py", line 440, in load
    self.loaded_app = self.loaded_app()
  File "/home/ubuntu/.local/lib/python3.10/site-packages/llama_cpp/server/app.py", line 150, in create_app
    set_llama_proxy(model_settings=model_settings)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/llama_cpp/server/app.py", line 70, in set_llama_proxy
    _llama_proxy = LlamaProxy(models=model_settings)
  File "/home/ubuntu/.local/lib/python3.10/site-packages/llama_cpp/server/model.py", line 31, in __init__
    self._current_model = self.load_llama_from_model_settings(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/llama_cpp/server/model.py", line 236, in load_llama_from_model_settings
    _model = create_fn(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/llama_cpp/llama.py", line 369, in __init__
    internals.LlamaModel(
  File "/home/ubuntu/.local/lib/python3.10/site-packages/llama_cpp/_internals.py", line 56, in __init__
    raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: /home/ubuntu/.cache/huggingface/hub/models--meta-llama--Llama-3.2-11B-Vision/refs/main
Error starting LLM, run with --verbose for more
jkp007 commented 2 days ago

@bobcatfish Can you please check on this?