integration with litellm

[thistleknot]

How can I use this with say a litellm provider? I see litellm in the code. Can't I just point to text-generation-webui's openai endpoint?

import os
from litellm import completion

## set ENV variables
os.environ["OPENAI_API_KEY"] = "anything" #key is not used for proxy

messages = [{ "content": "Hello, how are you?","role": "user"}]

response = completion(
    model="command-nightly", 
    messages=[{ "content": "Hello, how are you?","role": "user"}],
    api_base="http://192.168.3.17:5000/v1",
    custom_llm_provider="openai" # litellm will use the openai.ChatCompletion to make the request

)
print(response)

ModelResponse(id='chatcmpl-1719636857145463296', choices=[Choices(finish_reason='stop', index=0, message=Message(content="Hello! I'm just an AI, so I don't have feelings, but I'm here and ready to assist you with any questions or problems you might have. How can I help you today?", role='assistant'))], created=1719636857, model='mistral', object='chat.completion', system_fingerprint=None, usage=Usage(completion_tokens=44, prompt_tokens=14, total_tokens=58))

[emrgnt-cmplxty]

You should be able to do so by executing this command:

python -m r2r.examples.quickstart rag \
  --query="What contributions did Aristotle make to biology?" \
  --config_name=local_ollama \
  --rag_generation_config='{"model": "command-nightly"}'

[thistleknot]

What about ingesting documents?

On Fri, Jun 28, 2024, 10:45 PM emrgnt-cmplxty @.***> wrote:

You should be able to do so by executing this command:

python -m r2r.examples.quickstart rag \ --query="What contributions did Aristotle make to biology?" \ --config_name=local_ollama \ --rag_generation_config='{"model": "command-nightly"}'

— Reply to this email directly, view it on GitHub https://github.com/SciPhi-AI/R2R/issues/573#issuecomment-2197993852, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHKKOUEDNO3L3SXTGXRX6TZJZCWHAVCNFSM6AAAAABKC4H2H2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCOJXHE4TGOBVGI . You are receiving this because you authored the thread.Message ID: @.***>

[thistleknot]

(textgen) [root@pve0 R2R]# python -m r2r.examples.quickstart rag \
>   --query="What contributions did Aristotle make to biology?" \
>   --config_name=local_ollama \
>   --rag_generation_config='{"model": "command-nightly"}'

2024-06-29 07:22:31,905 - INFO - r2r.base.providers.vector_db_provider - Initializing VectorDBProvider with config extra_fields={} provider='pgvector'.
Traceback (most recent call last):
  File "/root/miniconda3/envs/textgen/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/root/miniconda3/envs/textgen/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/data/R2R/r2r/examples/quickstart.py", line 600, in <module>
    fire.Fire(R2RQuickstart)
  File "/root/miniconda3/envs/textgen/lib/python3.10/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/root/miniconda3/envs/textgen/lib/python3.10/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/root/miniconda3/envs/textgen/lib/python3.10/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/data/R2R/r2r/examples/quickstart.py", line 71, in __init__
    self.app = R2R(config=config)
  File "/data/R2R/r2r/main/r2r.py", line 37, in __init__
    built = builder.build()
  File "/data/R2R/r2r/main/assembly/builder.py", line 170, in build
    providers = provider_factory(self.config).create_providers(
  File "/data/R2R/r2r/main/assembly/factory.py", line 177, in create_providers
    or self.create_vector_db_provider(
  File "/data/R2R/r2r/main/assembly/factory.py", line 53, in create_vector_db_provider
    vector_db_provider.initialize_collection(
  File "/data/R2R/r2r/providers/vector_dbs/pgvector/pgvector_db.py", line 97, in initialize_collection
    self.collection = self.vx.get_or_create_collection(
  File "/data/R2R/r2r/vecs/client.py", line 177, in get_or_create_collection
    return collection._create_if_not_exists()
  File "/data/R2R/r2r/vecs/collection.py", line 280, in _create_if_not_exists
    raise MismatchedDimension(
r2r.vecs.exc.MismatchedDimension: Dimensions reported by adapter, dimension, and existing collection do not match

I think I'm going to wait until the documentation is more mature with proper use cases documented.

I'm going to rely on pgvector and langchain in the interim. (It took me less time to set up my own homebrewed version than trying to figure out how to utilize this with text-generation-webui api endpoint). Hopefully someone will write a medium.com article from start to finish.