search

meta-llama / llama-stack-client-python

Python SDK for Llama Stack
Apache License 2.0
77 stars 19 forks source link

[CLI] llama-stack-client CLI for querying server distro #3

Closed yanxi0830 closed 1 month ago

yanxi0830 commented 1 month ago

CLI Reference

llama-stack-client

$ llama-stack-client -h

usage: llama-stack-client [-h] {models,memory_banks,shields} ...

Welcome to the LlamaStackClient CLI

options:
  -h, --help            show this help message and exit

subcommands:
  {models,memory_banks,shields}

llama-stack-client configure

$ llama-stack-client configure
> Enter the host name of the Llama Stack distribution server: localhost
> Enter the port number of the Llama Stack distribution server: 5000
Done! You can now use the Llama Stack Client CLI with endpoint http://localhost:5000

llama-stack-client models list

llama-stack-client models list

image

llama-stack-client models get

llama-stack-client models get Llama3.1-8B-Instruct

image

$ llama-stack-client models get Random-Model 

Model RandomModel is not found at distribution endpoint host:port. Please ensure endpoint is serving specified model.

llama-stack-client memory_banks list

llama-stack-client memory_banks list

image

llama-stack-client shields list

llama-stack-client shields list

image

raghotham commented 1 month ago

I should be able to just configure the client cli to point to a host by providing an API key if needed.

$ llama-stack-client configure host: ... port: ... api_key: ...

After that, I should be able to do:

$ llama-stack-client <models|memory_banks|shields> <list|get>

without having to specify host/port each time.

yanxi0830 commented 1 month ago

Requesting changes mainly for Raghu's comment on doing a configuration first and then reusing those values for contacting the distribution server.

A small nit -- can the cli directory be placed until src/llama_stack_client instead of inside src/llama_stack_client/lib/

For purely additive helper or utility code, consider adding it to the package's lib directory. Stainless will never modify the contents of these directories.

I put it to lib for now to be compatible w/ Stainless's custom code suggestion, and think may us save some further trouble when syncing. We can do further refactor when API is more stable.