[Bug]: --use_queue fails when run for the first time

Mikkolehtimaki commented 9 months ago

What happened?

Starting litellm server with --use_queue for the first time in a clean environment fails (no variable redis). Trying again, the server starts normally.

Relevant log output

No response

Twitter / LinkedIn details

No response

Mikkolehtimaki commented 9 months ago

There is also this in the server logs when starting the queue:

Traceback (most recent call last):
  File "/.../.pyenv/versions/3.11.0/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/.../.pyenv/versions/3.11.0/lib/python3.11/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/.../.pyenv/versions/3.11.0/envs/yokotai-llm-server/lib/python3.11/site-packages/litellm/proxy/queue/celery_worker.py", line 5, in run_worker
    os.chdir(cwd)
FileNotFoundError: [Errno 2] No such file or directory: '<workdir-redacted>/queue'

krrishdholakia commented 9 months ago

@Mikkolehtimaki which version of litellm are you using?

looks like this folder is missing

Mikkolehtimaki commented 9 months ago

I'm on 1.7.5

aseifert commented 9 months ago

Same(?) problem. First I'm getting

FileNotFoundError: [Errno 2] No such file or directory: '<cwd>/queue'

when I manually create this dir I'm getting

INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
Usage: celery [OPTIONS] COMMAND [ARGS]...
Try 'celery --help' for help.

Error: Invalid value for '-A' / '--app':
Unable to load celery application.
While trying to load the module celery_app.celery_app the following error occurred:
Traceback (most recent call last):
  File "<env>/lib/python3.12/site-packages/celery/bin/celery.py", line 58, in convert
    return find_app(value)
           ^^^^^^^^^^^^^^^
  File "<env>/lib/python3.12/site-packages/celery/app/utils.py", line 383, in find_app
    sym = symbol_by_name(app, imp=imp)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<env>/lib/python3.12/site-packages/kombu/utils/imports.py", line 59, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<env>/lib/python3.12/site-packages/celery/utils/imports.py", line 109, in import_from_cwd
    return imp(module, package=package)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<env>/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1318, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'celery_app'

not sure what to do at this point.

krrishdholakia commented 9 months ago

I believe it's looking for this - https://github.com/BerriAI/litellm/blob/7856d65d44a509b450ac8fadd8f2b60da46d5c1e/litellm/proxy/queue/celery_app.py#L36

Not sure why it's not being found.

I also think we found the celery approach to overload the redis cache, @aseifert. Will probably need to tackle this in a v2.

Maybe have it work with /chat/completions -> and have the endpoint add it to a queue that's being worked on by a fastapi background task.

That way it's still compatible with a normal openai chat completions call.

ishaan-jaff commented 3 months ago

closing as not planned. We support Request Prioritization now: https://docs.litellm.ai/docs/scheduler

BerriAI / litellm