search

bentoml / OpenLLM

Run any open-source LLMs, such as Llama, Mistral, as OpenAI compatible API endpoint in the cloud.
https://bentoml.com
Apache License 2.0
10.16k stars 642 forks source link

bug: openllm server crash on startup #507

Closed Clivern closed 1 year ago

Clivern commented 1 year ago

Describe the bug

When i try to run openllm, it crashed with the following exception. I followed the few steps explained in README . Any idea what i might be missing?

$ pip install openllm
$ openllm start opt --model-id facebook/opt-2.7b
Downloading (…)lve/main/config.json: 100%|███████████████████████████████████████████████████| 653/653 [00:00<00:00, 4.46MB/s]
Downloading (…)okenizer_config.json: 100%|███████████████████████████████████████████████████| 685/685 [00:00<00:00, 5.66MB/s]
Downloading (…)olve/main/vocab.json: 100%|█████████████████████████████████████████████████| 899k/899k [00:00<00:00, 38.4MB/s]
Downloading (…)olve/main/merges.txt: 100%|█████████████████████████████████████████████████| 456k/456k [00:00<00:00, 26.3MB/s]
Downloading (…)cial_tokens_map.json: 100%|███████████████████████████████████████████████████| 441/441 [00:00<00:00, 2.93MB/s]
Downloading (…)neration_config.json: 100%|████████████████████████████████████████████████████| 137/137 [00:00<00:00, 704kB/s]
Downloading pytorch_model.bin: 100%|██████████████████████████████████████████████████████| 2.63G/2.63G [00:06<00:00, 414MB/s]
Fetching 7 files: 100%|█████████████████████████████████████████████████████████████████████████| 7/7 [00:06<00:00,  1.03it/s]
Building Bento for 'opt'
Traceback (most recent call last):
  File "/usr/local/bin/openllm", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 192, in wrapper
    return_value = func(*args, **attrs)
  File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 174, in wrapper
    return f(*args, **attrs)
  File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 591, in build_command
    raise err from None
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 578, in build_command
    bento = bundle.create_bento(
  File "/usr/local/lib/python3.10/dist-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/usr/local/lib/python3.10/dist-packages/openllm/bundle/_package.py", line 247, in create_bento
    bento = bentoml.Bento.create(build_config=build_config, version=bento_tag.version, build_ctx=llm_fs.getsyspath('/'))
  File "/usr/local/lib/python3.10/dist-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/bento/bento.py", line 187, in create
    svc = import_service(
  File "/usr/local/lib/python3.10/dist-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/service/loader.py", line 137, in import_service
    module = importlib.import_module(module_name, package=working_dir)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tmp/tmpfobpe3nyllm_opt/generated_opt_service.py", line 35, in <module>
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 1131, in Runner
    runner = infer_auto_class(backend).create_runner(model_name, llm_config=llm_config, ensure_available=ensure_available, **attrs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/models/auto/factory.py", line 71, in create_runner
    return cls.for_model(model, model_id=model_id, **attrs).to_runner(**runner_attrs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 883, in to_runner
    return llm_runner_class(self)(llm_runnable_class(self, embeddings_sig, generate_sig, generate_iterator_sig),
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 1328, in llm_runner_class
    return types.new_class(self.__class__.__name__ + 'Runner', (bentoml.Runner,),
  File "/usr/lib/python3.10/types.py", line 74, in new_class
    exec_body(ns)
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 1348, in <lambda>
    'prompt_template': self._prompt_template.to_string() if self._prompt_template else self.config.default_prompt_template,
  File "/usr/local/lib/python3.10/dist-packages/openllm_core/_configuration.py", line 1281, in __getattribute__
    return _object_getattribute.__get__(self)(item)
AttributeError: 'OPTConfig' object has no attribute 'default_prompt_template'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 76, in apport_excepthook
    binary = os.path.realpath(os.path.join(os.getcwd(), sys.argv[0]))
FileNotFoundError: [Errno 2] No such file or directory

Original exception was:
Traceback (most recent call last):
  File "/usr/local/bin/openllm", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.10/dist-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 192, in wrapper
    return_value = func(*args, **attrs)
  File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 174, in wrapper
    return f(*args, **attrs)
  File "/usr/local/lib/python3.10/dist-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 591, in build_command
    raise err from None
  File "/usr/local/lib/python3.10/dist-packages/openllm/cli/entrypoint.py", line 578, in build_command
    bento = bundle.create_bento(
  File "/usr/local/lib/python3.10/dist-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/usr/local/lib/python3.10/dist-packages/openllm/bundle/_package.py", line 247, in create_bento
    bento = bentoml.Bento.create(build_config=build_config, version=bento_tag.version, build_ctx=llm_fs.getsyspath('/'))
  File "/usr/local/lib/python3.10/dist-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/bento/bento.py", line 187, in create
    svc = import_service(
  File "/usr/local/lib/python3.10/dist-packages/simple_di/__init__.py", line 139, in _
    return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/service/loader.py", line 137, in import_service
    module = importlib.import_module(module_name, package=working_dir)
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/tmp/tmpfobpe3nyllm_opt/generated_opt_service.py", line 35, in <module>
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 1131, in Runner
    runner = infer_auto_class(backend).create_runner(model_name, llm_config=llm_config, ensure_available=ensure_available, **attrs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/models/auto/factory.py", line 71, in create_runner
    return cls.for_model(model, model_id=model_id, **attrs).to_runner(**runner_attrs)
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 883, in to_runner
    return llm_runner_class(self)(llm_runnable_class(self, embeddings_sig, generate_sig, generate_iterator_sig),
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 1328, in llm_runner_class
    return types.new_class(self.__class__.__name__ + 'Runner', (bentoml.Runner,),
  File "/usr/lib/python3.10/types.py", line 74, in new_class
    exec_body(ns)
  File "/usr/local/lib/python3.10/dist-packages/openllm/_llm.py", line 1348, in <lambda>
    'prompt_template': self._prompt_template.to_string() if self._prompt_template else self.config.default_prompt_template,
  File "/usr/local/lib/python3.10/dist-packages/openllm_core/_configuration.py", line 1281, in __getattribute__
    return _object_getattribute.__get__(self)(item)
AttributeError: 'OPTConfig' object has no attribute 'default_prompt_template'

To reproduce

No response

Logs

No response

Environment

Python 3.10

System information (Optional)

Python 3.10 OS: Ubuntu 22.04 8 CPU RAM 30 GB Accelerators: Quadro M4000 (8 GB GPU memory)

aarnphm commented 1 year ago

I will release a patch promptly.

aarnphm commented 1 year ago

Can you try again with 0.3.8?

Clivern commented 1 year ago

Can you try again with 0.3.8?

Server starts but still error raised during API call. I guess because of GenerationInput in the following lines

https://github.com/bentoml/OpenLLM/blob/main/openllm-python/src/openllm/_service.py#L41 https://github.com/bentoml/OpenLLM/blob/main/openllm-python/src/openllm/_service.py#L54

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/bentoml/_internal/server/http_app.py", line 341, in api_func
    output = await api.func(*args)
  File "/usr/local/lib/python3.10/dist-packages/openllm/_service.py", line 54, in generate_stream_v1
    qa_inputs = openllm.GenerationInput.from_llm_config(llm_config)(**input_dict)
  File "/usr/local/lib/python3.10/dist-packages/openllm_core/utils/lazy.py", line 155, in __getattr__
    raise AttributeError(f'module {self.__name__} has no attribute {name}')
AttributeError: module openllm has no attribute GenerationInput
aarnphm commented 1 year ago

Ah this is an oversight on my end. 0.3.9 will come out promptly

aarnphm commented 1 year ago

Please try again with 0.3.9

Clivern commented 1 year ago

Yes, it works fine 🔥 , Thanks @aarnphm!