search

meta-llama / llama-stack-client-python

Python SDK for Llama Stack
Apache License 2.0
77 stars 21 forks source link

Resource create #26

Closed dineshyv closed 2 weeks ago

dineshyv commented 2 weeks ago 
❯llama-stack-client models register meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo
Successfully registered model meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo
❯ llama-stack-client models list
+----------------------+--------------------+---------------------------------------------+------------+
| identifier           | provider_id        | provider_resource_id                        | metadata   |
+======================+====================+=============================================+============+
| meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo| remote::together-0 | meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo | {}         |
+----------------------+--------------------+---------------------------------------------+------------+
| meta-llama/Llama-3.2-3B-Instruct-Turbo  | remote::together-0 | meta-llama/Llama-3.2-3B-Instruct-Turbo      | {}         |
+----------------------+--------------------+---------------------------------------------+------------+
❯ llama-stack-client models register dineshyv-model --provider-model-id=meta-llama/Meta-Llama-Guard-3-8B
Successfully registered model dineshyv-model
❯ llama-stack-client models list
+----------------------------------------------+--------------------+----------------------------------------------+------------+
| identifier                                   | provider_id        | provider_resource_id                         | metadata   |
+==============================================+====================+==============================================+============+
+----------------------------------------------+--------------------+----------------------------------------------+------------+
| dineshyv-model                               | remote::together-0 | meta-llama/Meta-Llama-Guard-3-8B             | {}         |
+----------------------------------------------+--------------------+----------------------------------------------+------------+

llama-stack-client shields register --shield-id llama-guard --provider-shield-id Llama-Guard-3-8B
identifier: llama-guard
params: {}
provider_id: inline::llama-guard-0
provider_resource_id: Llama-Guard-3-8B
type: shield

❯ llama-stack-client shields list
+--------------+----------+-----------------------+------------------------+--------+
| identifier   | params   | provider_id           | provider_resource_id   | type   |
+==============+==========+=======================+========================+========+
| llama-guard  | {}       | inline::llama-guard-0 | Llama-Guard-3-8B       | shield |
+--------------+----------+-----------------------+------------------------+--------+

 llama-stack-client datasets register \
  --dataset-id "alpaca" \
  --url "https://huggingface.co/datasets/tatsu-lab/alpaca" \
  --schema '{"instruction": {"type": "string"}, "input": {"type": "string"}, "output": {"type": "string"}}' --provider-id huggingface-0
❯ llama-stack-client datasets list
+--------------+---------------+---------------------------------------------------------------------------------+---------+
| identifier   | provider_id   | metadata                                                                        | type    |
+==============+===============+=================================================================================+=========+
| mmlu         | huggingface-0 | {'path': 'llama-stack/evals', 'name': 'evals__mmlu__details', 'split': 'train'} | dataset |
+--------------+---------------+---------------------------------------------------------------------------------+---------+
| alpaca       | huggingface-0 | {}                                                                              | dataset |
+--------------+---------------+---------------------------------------------------------------------------------+---------+

 llama-stack-client eval_tasks register \
    --eval-task-id "my-eval-task" \
    --dataset-id "alpaca" \
    --scoring-functions "accuracy" \
    --scoring-functions "f1_score" \
    --metadata '{"description": "My evaluation task", "version": "1.0"}'

llama-stack-client eval_tasks list
+--------------+---------------------+---------------------------------------------------------+------------------+------------------------+------------------------------------------------+-----------+
| dataset_id   | identifier          | metadata                                                | provider_id      | provider_resource_id   | scoring_functions                              | type      |
+==============+=====================+=========================================================+==================+========================+================================================+===========+
| mmlu         | meta-reference-mmlu | {}                                                      | meta-reference-0 | meta-reference-mmlu    | ['basic::regex_parser_multiple_choice_answer'] | eval_task |
+--------------+---------------------+---------------------------------------------------------+------------------+------------------------+------------------------------------------------+-----------+
| alpaca       | my-eval-task        | {'description': 'My evaluation task', 'version': '1.0'} | meta-reference-0 | my-eval-task           | ['accuracy', 'f1_score']                       | eval_task |
+--------------+---------------------+---------------------------------------------------------+------------------+------------------------+------------------------------------------------+-----------+

Failure cases:

llama-stack-client models register dineshyv-model --provider-model-id=Llama-Guard-78979
Failed to register model: Error code: 400 - {'detail': 'Invalid value: Unknown model: `Llama-Guard-78979`'}