Most ergonomic way for `client=custom_client` to dispatch between **default** or custom clients?

[gwpl]

Situation:

• for some models (examples: ollama_ex.py , vllm_ex.py ), we need to specify client= in decorator
• what if we would like to provide there client=my_client_dispacher, so my_client_dispacher for some values of model would return default client , and for other , one of few custom clients?
• that way I could allow e.g. user to select model via dotenv (.env) or other way?
  • relaying on default client for most models, and for some (matching patterns), to route them to custom_defined_clients... (e.g. local or shared ollama server etc)

[alex-dixon]

@gwpl

To clarify, you don't need to specify client= for ollama models if you use this code:

import ell

ell.init(verbose=True, store='./logdir')
ell.models.ollama.register(base_url="http://localhost:11434/v1")

# in terminal run ollama list to see available models
@ell.simple(model="llama3.1:latest", temperature=0.1)
def write_a_story():
    return "write me a story"

Everything else should use the default OpenAI client instance.

If you're looking to parameterize the model kwarg that is used for a particular LMP (like write_a_story), you could define a variable for the model kwarg. I believe you can also override this parameter when you invoke it with write_a_story(model="llama3.2"),

[gwpl]

Sure! (btw. do ell.config.register_model("foo",...) overwirte earlier resolution for "foo" https://docs.ell.so/core_concepts/models_and_api_clients.html )

Here I was thinking about more sophisticated setups, that ofc I can make by making ollama api compatible proxy that would have resolution logic, but I thought that maybe I could register own dispatcher (e.g. ell.config.register_providers_dispatcher( my_dispatcher(fallback=ell.config.get_current_providers_dispatcher())) )

[alex-dixon]

do ell.config.register_model("foo",...) overwirte earlier resolution for "foo" https://docs.ell.so/core_concepts/models_and_api_clients.html

They should yes.

If you look at the provider and model files you can do a lot more custom stuff, maybe along the lines of what you’re describing.

I’m having a hard time seeing the end goal you want to achieve. If you could paint more of a picture for me there I may be able to help with a particular implementation that achieves it.

[gwpl]

Setting custom dispacher, like I mentioned:

ell.config.register_providers_dispatcher( my_dispatcher(fallback=ell.config.get_current_providers_dispatcher())) )

could be good when playing on small scale with setup utilizing either load balancing, failover or just resolution of unregistered models.

However for more production setup, very likely it should use suggested by you, e.g. Ollama or just ell.config.via_proxy(...) to make everything regardless of model go via proxy and do all the magic inside the proxy.