search

instructlab / instructlab

InstructLab Command-Line Interface. Use this to chat with a model and execute the InstructLab workflow to train a model using custom taxonomy data.
https://instructlab.ai
Apache License 2.0
616 stars 221 forks source link

Traceback when model not found on disk #1378

Open mairin opened 3 weeks ago

mairin commented 3 weeks ago

Describe the bug

I have models on my system that I'd already downloaded in /srv/models.... I didn't realize there were three model paths in the config.yaml to start so I got a traceback when i tried ilab serve and it couldn't find the model in the default model path.

Instead of tracing back, ideally we'd catch the error condition of the model not existing on disk and provide the user a nice human readable error message, something like:

We couldn't find the model to serve on disk. config.yaml indicates it should be located at models/merlinite-7b-lab-Q4_K_M.gguf - can you check to make sure this is correct? Update config.yaml with the correct location or run ilab download to get a model to use.

^ something like this. @kelbrown20 prolly has better ideas on phrasing.

Here's the traceback:

(venv) duffy@pandamonium:/srv/instructlab$ ilab serve
You are using an aliased command, this will be deprecated in a future release. Please consider using `ilab model serve` instead
INFO 2024-06-16 20:43:33,831 serve.py:51: serve Using model 'models/merlinite-7b-lab-Q4_K_M.gguf' with -1 gpu-layers and 4096 max context size.
ERROR 2024-06-16 20:43:33,852 llama_cpp:0: flush Traceback (most recent call last):
  File "/srv/instructlab/venv/lib64/python3.12/site-packages/instructlab/server.py", line 177, in server
    app = create_app(settings=settings)
ERROR 2024-06-16 20:43:33,852 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,852 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/llama_cpp/server/app.py", line 147, in create_app
    set_llama_proxy(model_settings=model_settings)
ERROR 2024-06-16 20:43:33,852 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/llama_cpp/server/app.py", line 68, in set_llama_proxy
    _llama_proxy = LlamaProxy(models=model_settings)
ERROR 2024-06-16 20:43:33,852 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,852 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/llama_cpp/server/model.py", line 31, in __init__
    self._current_model = self.load_llama_from_model_settings(
ERROR 2024-06-16 20:43:33,852 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/llama_cpp/server/model.py", line 217, in load_llama_from_model_settings
    _model = create_fn(
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush ^^^^^^^^^^
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/llama_cpp/llama.py", line 336, in __init__
    raise ValueError(f"Model path does not exist: {model_path}")
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush ValueError: Model path does not exist: models/merlinite-7b-lab-Q4_K_M.gguf
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush 
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush The above exception was the direct cause of the following exception:
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush Traceback (most recent call last):
  File "/srv/instructlab/venv/lib64/python3.12/site-packages/instructlab/model/serve.py", line 58, in serve
    server(
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/instructlab/server.py", line 190, in server
    raise ServerException(f"failed creating the server application: {exc}") from exc
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush instructlab.server.ServerException: failed creating the server application: Model path does not exist: models/merlinite-7b-lab-Q4_K_M.gguf
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush 
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush During handling of the above exception, another exception occurred:
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush Traceback (most recent call last):
  File "/srv/instructlab/venv/bin/ilab", line 8, in <module>
    sys.exit(ilab())
ERROR 2024-06-16 20:43:33,853 llama_cpp:0: flush ^^^^^^
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush ^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,854 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/instructlab/model/serve.py", line 69, in serve
    click.secho(f"Error creating server: {exc}", fg="red")
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/termui.py", line 634, in secho
    return echo(message, file=file, nl=nl, err=err, color=color)
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/utils.py", line 318, in echo
    file.write(out)  # type: ignore
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush ^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush File "/srv/instructlab/venv/lib64/python3.12/site-packages/click/_compat.py", line 99, in __getattr__
    return getattr(self._stream, name)
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush ^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 2024-06-16 20:43:33,855 llama_cpp:0: flush AttributeError: 'str' object has no attribute 'write'
(venv) duffy@pandamonium:/srv/instructlab$

Device Info (please complete the following information):

drpaneas commented 3 weeks ago

Hi there @mairin 👋 please have a look at the output I have provided at the PR and provide your review, if you have some time for it.

mairin commented 3 weeks ago

@drpaneas of course I have time - that was a record time for me to experience from filing a bug to having a fix 😁 thank you so much for the PR! I took a look and left some review comments. My main point was we support more than *.gguf format so I think that specificity can be dropped. @russellb 's comment on the logger I think is worth revisiting as well. Other than that this looks great.

Do you have some time to make the requested changes? 😃?

drpaneas commented 3 weeks ago

ready for review again ;)

drpaneas commented 2 weeks ago

Hi again, I am not working anymore on this, as per discussion it requires refactoring of input validation and debugging information (verbosity levels) in the whole project in general.

chudegao commented 5 days ago

I think this issue is already solved by other commits.

(chudg1) [root@llm lab]# ilab serve --model-path models/abc.gguf --backend llama-cpp
You are using an aliased command, this will be deprecated in a future release. Please consider using `ilab model serve` instead
Failed to determine backend: Failed to determine whether the model is a GGUF format: [Errno 2] No such file or directory: 'models/abc.gguf'
(chudg1) [root@llm lab]# ls models/abc.gguf
ls: cat not access 'models/abc.gguf': No such file or directory
(chudg1) [root@llm lab]#