Selecting Models Directly

[dbmanifest]

I'm sure it's not an issue and I'm just doing it wrong, but I can seem to figure out how to select a specific model directly. Right now im using the clear_models method and then the add_model method passing in the model name as a string as an arg, I know for a fact the model I am trying to select is available and running on the Horde because Im making a request to confirm that prior to trying to add it. It has to be an issue with what I'm doing, I presume.

I'll close this when I figure it out if I do.

[dbmanifest]

So I figured out what it is, in order to set a model you need the stats object from the API.

I propose setting another method on the client called "get_model" which will accept a model name and return the model stats object

I also propose a clear_all_but method that also accepts a model name and clears all but the incoming name

The utility of this is to allow a user to decide which model specifically to use after evaluating the available models.

If the plan is approved, I can work on a PR in the next few days.

[rahuldshetty]

Hi @dbmanifest , You're right on the part to use the _setmodel method with the model name to select the desired model for inference.

Here is a quick example, which I will add in the docs in the next release.

from horde_client import HordeClient, TextGenParams, ModelType

client = HordeClient(
  # To access public KoboldAI service
  insecure=True
)

# Prompt
prompt =  """### Instruction: 
Tell me a knock knock joke.

### Response:
"""

for model in client.list_models(
    type=ModelType.text
):
        print(model)

# Set model for inference
client.set_model('Gryphe/MythoMax-L2-13b')

# Latest model will be referred
client.text_gen(prompt)

# Reset the model
client.clear_model()

client.set_model('tgi-fp16-8k/Gryphe/MythoMax-L2-13b')

client.text_gen(prompt)

You don't need any stats while setting the model. You might need it if you want to select the best model based on performance. I've built the _setmodel and _clearmodel based on that assumption by checking out the functionality in https://lite.koboldai.net/.

Do let me know if you think I've missed anything.