search

meta-llama / llama-stack

Model components of the Llama Stack APIs
MIT License
3.56k stars 497 forks source link

`llama stack run my-local-stack` causes pydantic error #174

Open frieda-huang opened 1 week ago

frieda-huang commented 1 week ago

I'm using remote::ollama and Meta-Llama3.1-8B-Instruct. llama stack build my-local-stack and llama stack configure my-local stack ran without problem, but encountered pydantic error upon running llama stack run my-local-stack. Any ideas? I'm running it on Mac M2 using conda since docker doesn't work.

llama stack run my-local-stack              2 err | stack py | at 18:39:31
Traceback (most recent call last):
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 364, in <module>
    fire.Fire(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 286, in main
    config = StackRunConfig(**yaml.safe_load(fp))
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 7 validation errors for StackRunConfig
api_providers.agents.GenericProviderConfig.provider_type
  Field required [type=missing, input_value={'provider_id': 'meta-ref...er', 'password': None}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
api_providers.agents.PlaceholderProviderConfig.providers
  Field required [type=missing, input_value={'provider_id': 'meta-ref...er', 'password': None}}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
api_providers.telemetry.GenericProviderConfig.provider_type
  Field required [type=missing, input_value={'provider_id': 'meta-reference', 'config': {}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
api_providers.telemetry.PlaceholderProviderConfig.providers
  Field required [type=missing, input_value={'provider_id': 'meta-reference', 'config': {}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
routing_table.inference.0.provider_type
  Field required [type=missing, input_value={'provider_id': 'remote::...a-Llama3.1-8B-Instruct'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
routing_table.safety.0.provider_type
  Field required [type=missing, input_value={'provider_id': 'meta-ref...d', 'jailbreak_shield']}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
routing_table.memory.0.provider_type
  Field required [type=missing, input_value={'provider_id': 'meta-ref...'routing_key': 'vector'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
Error occurred in script at line: 42
ashwinb commented 1 week ago

This is likely because we changed provider_id to provider_type and you are working from our recently updated code.

If you go to ~/.llama/builds/conda/my-local-stack-run.yaml and edit it to replace provider_id with provider_type -- these errors should go away. Let me know if they don't.

JeremyBickel commented 1 week ago

I've got this error and changed the yaml like you said, but it didn't work.

Traceback (most recent call last): File "/home/jeremy/.conda/envs/llamastack-local/bin/llama", line 8, in sys.exit(main()) File "/home/jeremy/.conda/envs/llamastack-local/lib/python3.10/site-packages/llama_stack/cli/llama.py", line 44, in main parser.run(args) File "/home/jeremy/.conda/envs/llamastack-local/lib/python3.10/site-packages/llama_stack/cli/llama.py", line 38, in run args.func(args) File "/home/jeremy/.conda/envs/llamastack-local/lib/python3.10/site-packages/llama_stack/cli/stack/run.py", line 79, in _run_stack_run_cmd config = StackRunConfig(**yaml.safe_load(f)) File "/home/jeremy/.conda/envs/llamastack-local/lib/python3.10/site-packages/pydantic/main.py", line 212, in init validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self) pydantic_core._pydantic_core.ValidationError: 7 validation errors for StackRunConfig api_providers.agents.GenericProviderConfig.provider_id Field required [type=missing, input_value={'provider_type': 'meta-r...a/runtime/kvstore.db'}}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing api_providers.agents.PlaceholderProviderConfig.providers Field required [type=missing, input_value={'provider_type': 'meta-r...a/runtime/kvstore.db'}}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing api_providers.telemetry.GenericProviderConfig.provider_id Field required [type=missing, input_value={'provider_type': 'meta-reference', 'config': {}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing api_providers.telemetry.PlaceholderProviderConfig.providers Field required [type=missing, input_value={'provider_type': 'meta-reference', 'config': {}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing routing_table.inference.0.provider_id Field required [type=missing, input_value={'provider_type': 'meta-r....2-11B-Vision-Instruct'}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing routing_table.memory.0.provider_id

frieda-huang commented 1 week ago

This is likely because we changed provider_id to provider_type and you are working from our recently updated code.

If you go to ~/.llama/builds/conda/my-local-stack-run.yaml and edit it to replace provider_id with provider_type -- these errors should go away. Let me know if they don't.

It looks like my-local-stack-run.yaml already replaced provider_id with provider_type. This is my current file:

version: v1
built_at: '2024-10-02T21:59:35.960179'
image_name: my-local-stack
docker_image: null
conda_env: my-local-stack
apis_to_serve:
- models
- shields
- agents
- memory
- memory_banks
- safety
- inference
api_providers:
  inference:
    providers:
    - remote::ollama
  safety:
    providers:
    - meta-reference
  agents:
    provider_type: meta-reference
    config:
      persistence_store:
        namespace: null
        type: postgres
        host: localhost
        port: 5432
        db: llamastack
        user: llamastack-user
        password: null
  memory:
    providers:
    - meta-reference
  telemetry:
    provider_type: meta-reference
    config: {}
routing_table:
  inference:
  - provider_type: remote::ollama
    config:
      host: localhost
      port: 5000
    routing_key: Meta-Llama3.1-8B-Instruct
  safety:
  - provider_type: meta-reference
    config:
      llama_guard_shield:
        model: Llama-Guard-3-1B
        excluded_categories: []

I also noticed that if I included Prompt-Guard-86M in the previous step of llama stack configure my-local-stack, it led to the error of Value error, Invalid model: Prompt-Guard-86M. Must be one of [] [type=value_error, input_value='Prompt-Guard-86M', input_type=str]. As I further investigate, I found that CoreModelId has no enum prompt_guard_86m and safey_models() also doesn't include the Model for prompt_guard_86m.

ashwinb commented 1 week ago

@frieda-huang Looking into this. I know the issue about Prompt-Guard but the one about validation errors is stumping me since I cannot reproduce it yet.

Can you tell me what version of the llama-stack package you are using?

frieda-huang commented 1 week ago

@frieda-huang Looking into this. I know the issue about Prompt-Guard but the one about validation errors is stumping me since I cannot reproduce it yet.

Can you tell me what version of the llama-stack package you are using?

I'm using the latest version. I think the error was stem from this portion of the code:

class PromptGuardShieldConfig(BaseModel):
    model: str = "Prompt-Guard-86M"

    @validator("model")
    @classmethod
    def validate_model(cls, model: str) -> str:
        permitted_models = [
            m.descriptor()
            for m in safety_models()
            if m.core_model_id == CoreModelId.prompt_guard_86m
        ]
        if model not in permitted_models:
            raise ValueError(
                f"Invalid model: {model}. Must be one of {permitted_models}"
            )
        return model

This is the original error message.

ERROR

llama stack configure my-local-stack
Could not find my-local-stack. Trying conda build name instead...
Configuring API `inference`...
=== Configuring provider `remote::ollama` for API inference...
Enter value for host (default: localhost) (required):
Enter value for port (required): 5000
> Please enter the supported model your provider has for inference: Meta-Llama3.
1-8B-Instruct

Configuring API `safety`...
=== Configuring provider `meta-reference` for API safety...
Do you want to configure llama_guard_shield? (y/n): y
Entering sub-configuration for llama_guard_shield:
Enter value for model (default: Llama-Guard-3-1B) (required):
Enter value for excluded_categories (default: []) (required):
Enter value for disable_input_check (default: False) (required):
Enter value for disable_output_check (default: False) (required):
Do you want to configure prompt_guard_shield? (y/n): y
Entering sub-configuration for prompt_guard_shield:
Enter value for model (default: Prompt-Guard-86M) (required):
Traceback (most recent call last):
  File "/opt/anaconda3/envs/stack/bin/llama", line 8, in <module>
    sys.exit(main())
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/cli/llama.py", line 44, in main
    parser.run(args)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/cli/llama.py", line 38, in run
    args.func(args)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/cli/stack/configure.py", line 87, in _run_stack_configure_cmd
    self._configure_llama_distribution(build_config, args.output_dir)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/cli/stack/configure.py", line 154, in _configure_llama_distribution
    config = configure_api_providers(config, build_config.distribution_spec)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/distribution/configure.py", line 95, in configure_api_providers
    cfg = prompt_for_config(config_type, existing)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/distribution/utils/prompt_for_config.py", line 201, in prompt_for_config
    config_data[field_name] = prompt_for_config(nested_type, existing_value)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/distribution/utils/prompt_for_config.py", line 316, in prompt_for_config
    return config_type(**config_data)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/pydantic/_internal/_decorators_v1.py", line 103, in wrapper2
    return val2(value)
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/providers/impls/meta_reference/safety/config.py", line 56, in validate_model
    permitted_models = [
  File "/opt/anaconda3/envs/stack/lib/python3.10/site-packages/llama_stack/providers/impls/meta_reference/safety/config.py", line 59, in <listcomp>
    if m.core_model_id == CoreModelId.prompt_guard_86m
  File "/opt/anaconda3/envs/stack/lib/python3.10/enum.py", line 437, in __getattr__
    raise AttributeError(name) from None
AttributeError: prompt_guard_86m

After adding some code to the llama-models, llama stack configure works!

I added Model to sku_list.py and prompt_guard_86m = "Prompt-Guard-86M" to datatypes.py

        Model(
            core_model_id=CoreModelId.prompt_guard_86m,
            is_default_variant=True,
            description="Llama Prompt Guard 86M system safety model",
            huggingface_repo="meta-llama/Prompt-Guard-86M",
            arch_args={
                "dim": 768,
                "n_layers": 12,
                "n_heads": 12,
                "vocab_size": PROMPT_GUARD_86M_VOCAB_SIZE,
                "norm_eps": 1e-07,
                "use_scaled_rope": False,
            },
            pth_file_count=1,
        )
ashwinb commented 1 week ago

I fixed this error now with https://github.com/meta-llama/llama-stack/commit/19ce6bf009a80dbc5ae269532b944e3579764fbd -- please pull again. However, this doesn't solve the validation error with provider_type vs provider_id you are seeing.

ashwinb commented 1 week ago

@frieda-huang @JeremyBickel I believe the error you are seeing with provider_id / provider_type is because of incompatibility between the published package and the repository code. What you are missing is something that's not quite well documented yet. When performing llama stack build you should do it like this:

cd llama-stack/
LLAMA_STACK_DIR=$PWD llama stack build <other arguments>

This will make sure it will use your current llama-stack checkout as the source for your build and will pick up your changes.

frieda-huang commented 1 week ago

@frieda-huang @JeremyBickel I believe the error you are seeing with provider_id / provider_type is because of incompatibility between the published package and the repository code. What you are missing is something that's not quite well documented yet. When performing llama stack build you should do it like this:

cd llama-stack/
LLAMA_STACK_DIR=$PWD llama stack build <other arguments>

This will make sure it will use your current llama-stack checkout as the source for your build and will pick up your changes.

Thank you for the quick response! Am I doing this correctly? 

LLAMA_STACK_DIR=$PWD llama stack build                                                                       1 err | took 18s | stack py | at 00:06:40
> Enter a name for your Llama Stack (e.g. my-local-stack): my-local-stack
> Enter the image type you want your Llama Stack to be built as (docker or conda): conda

 Llama Stack is composed of several APIs working together. Let's configure the providers (implementations) you want to use for these APIs.
> Enter provider for the inference API: (default=meta-reference): remote::ollama
> Enter provider for the safety API: (default=meta-reference): meta-reference
> Enter provider for the agents API: (default=meta-reference): meta-reference
> Enter provider for the memory API: (default=meta-reference): meta-reference
> Enter provider for the telemetry API: (default=meta-reference): meta-reference
Traceback (most recent call last):
  File "/opt/anaconda3/envs/stack/bin/llama", line 8, in <module>
    sys.exit(main())
  File "/Users/friedahuang/local/llama-stack/llama_stack/cli/llama.py", line 44, in main
    parser.run(args)
  File "/Users/friedahuang/local/llama-stack/llama_stack/cli/llama.py", line 38, in run
    args.func(args)
  File "/Users/friedahuang/local/llama-stack/llama_stack/cli/stack/build.py", line 257, in _run_stack_build_command
    providers_for_api = all_providers[api]
KeyError: <Api.inspect: 'inspect'>
ashwinb commented 1 week ago

Sorry, pull again please. https://github.com/meta-llama/llama-stack/commit/988a9cada3e7ea296611e20facdd2990f9512b2a

frieda-huang commented 1 week ago

Sorry, pull again please. 988a9ca

llama stack build works, but llama stack run my-local-stack fails. Looks like it's something related to Ollama. I'm currently using version 0.3.6.

llama stack run my-local-stack                                                                                          ok | stack py | at 00:20:40
Resolved 8 providers in topological order
  Api.models: routing_table
  Api.inference: router
  Api.shields: routing_table
  Api.safety: router
  Api.memory_banks: routing_table
  Api.memory: router
  Api.agents: meta-reference
  Api.telemetry: meta-reference

Initializing Ollama, checking connectivity to server...
Traceback (most recent call last):
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 351, in <module>
    fire.Fire(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 288, in main
    impls, specs = asyncio.run(resolve_impls_with_routing(config))
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 104, in resolve_impls_with_routing
    impl = await instantiate_provider(spec, deps, configs[api])
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 174, in instantiate_provider
    impl = await instantiate_provider(
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 192, in instantiate_provider
    impl = await fn(*args)
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/adapters/inference/ollama/__init__.py", line 14, in get_adapter_impl
    await impl.initialize()
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/adapters/inference/ollama/ollama.py", line 48, in initialize
    await self.client.ps()
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/ollama/_client.py", line 906, in ps
    response = await self._request('GET', '/api/ps')
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/ollama/_client.py", line 488, in _request
    raise ResponseError(e.response.text, e.response.status_code) from None
ollama._types.ResponseError
Error occurred in script at line: 42
cheesecake100201 commented 1 week ago

Sorry, pull again please. 988a9ca

llama stack build works, but llama stack run my-local-stack fails. Looks like it's something related to Ollama. I'm currently using version 0.3.6.

llama stack run my-local-stack                                                                                          ok | stack py | at 00:20:40
Resolved 8 providers in topological order
  Api.models: routing_table
  Api.inference: router
  Api.shields: routing_table
  Api.safety: router
  Api.memory_banks: routing_table
  Api.memory: router
  Api.agents: meta-reference
  Api.telemetry: meta-reference

Initializing Ollama, checking connectivity to server...
Traceback (most recent call last):
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 351, in <module>
    fire.Fire(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 288, in main
    impls, specs = asyncio.run(resolve_impls_with_routing(config))
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 104, in resolve_impls_with_routing
    impl = await instantiate_provider(spec, deps, configs[api])
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 174, in instantiate_provider
    impl = await instantiate_provider(
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 192, in instantiate_provider
    impl = await fn(*args)
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/adapters/inference/ollama/__init__.py", line 14, in get_adapter_impl
    await impl.initialize()
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/adapters/inference/ollama/ollama.py", line 48, in initialize
    await self.client.ps()
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/ollama/_client.py", line 906, in ps
    response = await self._request('GET', '/api/ps')
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/ollama/_client.py", line 488, in _request
    raise ResponseError(e.response.text, e.response.status_code) from None
ollama._types.ResponseError
Error occurred in script at line: 42

I am using the together inference and used the command told by @ashwinb. Still getting the same error of mismatch between provider_type and provider_id after editing the yaml file and running llama stack run .

JeremyBickel commented 1 week ago

@cheesecake100201 I think you're on to something there. I removed a normal pip install and all the conda envs I had built up and pulled again fresh last night, and the Pydantic problem didn't show up.

Double-checking, I did 'rm ~/.local/bin/llama', 'pip uninstall llama-stack', 'git pull', 'pip install -e .', and 'llama stack build --list-templates'. It gave the error:

Traceback (most recent call last): File "/home/jeremy/.local/bin/llama", line 8, in sys.exit(main()) ^^^^^^ File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/llama.py", line 44, in main parser.run(args) File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/llama.py", line 38, in run args.func(args) File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/stack/build.py", line 182, in _run_stack_build_command self._run_template_list_cmd(args) File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/stack/build.py", line 160, in _run_template_list_cmd for spec in available_templates_specs(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/stack/build.py", line 27, in available_templates_specs build_config = BuildConfig(**yaml.safe_load(f)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jeremy/.local/lib/python3.12/site-packages/pydantic/main.py", line 212, in init validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ pydantic_core._pydantic_core.ValidationError: 2 validation errors for BuildConfig name Field required [type=missing, input_value={'built_at': '2024-09-30T...uting_key': 'vector'}]}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing distribution_spec Field required [type=missing, input_value={'built_at': '2024-09-30T...uting_key': 'vector'}]}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing

Is that a known case or a new bug from the recent fix (or something else)?

EDIT: I skipped the conda step. I seem to have completely removed conda from my system last night.. With everything back in order, though, the error is similar:

Traceback (most recent call last): File "/home/jeremy/.conda/envs/stack/bin/llama", line 8, in sys.exit(main()) File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/llama.py", line 44, in main parser.run(args) File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/llama.py", line 38, in run args.func(args) File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/stack/build.py", line 182, in _run_stack_build_command self._run_template_list_cmd(args) File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/stack/build.py", line 160, in _run_template_list_cmd for spec in available_templates_specs(): File "/run/media/jeremy/4TB/Backups/AAA_llm/AI/Systems/llama-stack/llama-stack/llama_stack/cli/stack/build.py", line 27, in available_templates_specs build_config = BuildConfig(**yaml.safe_load(f)) File "/home/jeremy/.conda/envs/stack/lib/python3.10/site-packages/pydantic/main.py", line 212, in init validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self) pydantic_core._pydantic_core.ValidationError: 2 validation errors for BuildConfig name Field required [type=missing, input_value={'built_at': '2024-09-30T...uting_key': 'vector'}]}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing distribution_spec Field required [type=missing, input_value={'built_at': '2024-09-30T...uting_key': 'vector'}]}}, input_type=dict] For further information visit https://errors.pydantic.dev/2.9/v/missing

frieda-huang commented 1 week ago

19ce6bf

Sorry, pull again please. 988a9ca

llama stack build works, but llama stack run my-local-stack fails. Looks like it's something related to Ollama. I'm currently using version 0.3.6.

llama stack run my-local-stack                                                                                          ok | stack py | at 00:20:40
Resolved 8 providers in topological order
  Api.models: routing_table
  Api.inference: router
  Api.shields: routing_table
  Api.safety: router
  Api.memory_banks: routing_table
  Api.memory: router
  Api.agents: meta-reference
  Api.telemetry: meta-reference

Initializing Ollama, checking connectivity to server...
Traceback (most recent call last):
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 351, in <module>
    fire.Fire(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 288, in main
    impls, specs = asyncio.run(resolve_impls_with_routing(config))
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 104, in resolve_impls_with_routing
    impl = await instantiate_provider(spec, deps, configs[api])
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 174, in instantiate_provider
    impl = await instantiate_provider(
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 192, in instantiate_provider
    impl = await fn(*args)
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/adapters/inference/ollama/__init__.py", line 14, in get_adapter_impl
    await impl.initialize()
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/adapters/inference/ollama/ollama.py", line 48, in initialize
    await self.client.ps()
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/ollama/_client.py", line 906, in ps
    response = await self._request('GET', '/api/ps')
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/ollama/_client.py", line 488, in _request
    raise ResponseError(e.response.text, e.response.status_code) from None
ollama._types.ResponseError
Error occurred in script at line: 42

Update: Turns out I needed to update the port number inside .llama/builds/conda/my-local-stack-run.yaml and specify routing_key to be "Llama3.1-8B-Instruct" for Ollama to work.

Before

routing_table:
  inference:
  - provider_type: remote::ollama
    config:
      host: localhost
      port: 5000
    routing_key: Meta-Llama3.1-8B-Instruct

After

routing_table:
  inference:
  - provider_type: remote::ollama
    config:
      host: localhost
      port: 11434
    routing_key: Llama3.1-8B-Instruct

However, it's still not working due to AssertionError: Could not resolve model Prompt-Guard-86M. In the llama_stack/providers/impls/meta_reference/safety/safety.py, it requires the Prompt-Guard-86M to not be None

def resolve_and_get_path(model_name: str) -> str:
    model = resolve_model(model_name)
    assert model is not None, f"Could not resolve model {model_name}"
    model_dir = model_local_dir(model.descriptor())
    return model_dir
llama stack run my-local-stack --port 11434
Resolved 8 providers in topological order
  Api.models: routing_table
  Api.inference: router
  Api.shields: routing_table
  Api.safety: router
  Api.memory_banks: routing_table
  Api.memory: router
  Api.agents: meta-reference
  Api.telemetry: meta-reference

Initializing Ollama, checking connectivity to server...
Traceback (most recent call last):
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 351, in <module>
    fire.Fire(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/server/server.py", line 288, in main
    impls, specs = asyncio.run(resolve_impls_with_routing(config))
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/anaconda3/envs/llamastack-my-local-stack/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 104, in resolve_impls_with_routing
    impl = await instantiate_provider(spec, deps, configs[api])
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 174, in instantiate_provider
    impl = await instantiate_provider(
  File "/Users/friedahuang/local/llama-stack/llama_stack/distribution/resolver.py", line 192, in instantiate_provider
    impl = await fn(*args)
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/impls/meta_reference/safety/__init__.py", line 16, in get_provider_impl
    await impl.initialize()
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/impls/meta_reference/safety/safety.py", line 48, in initialize
    model_dir = resolve_and_get_path(shield_cfg.model)
  File "/Users/friedahuang/local/llama-stack/llama_stack/providers/impls/meta_reference/safety/safety.py", line 35, in resolve_and_get_path
    assert model is not None, f"Could not resolve model {model_name}"
AssertionError: Could not resolve model Prompt-Guard-86M
Error occurred in script at line: 42

I then excluded the prompt guard. It looks like postgres is not supported yet, as I got the NotImplementedError(). Any idea as to when it will get implemented or should I go ahead code a version for it?

ashwinb commented 1 week ago

@frieda-huang Oh boy -- can you try https://github.com/meta-llama/llama-stack/pull/177 to see if your prompt guard troubles are over?

About postgres -- what did you want postgres for?

ashwinb commented 1 week ago

@cheesecake100201 try updating the llama-stack package (v0.0.38) and let me know if that fixes the issues for you

frieda-huang commented 1 week ago

postgres

prompt guard troubles are clear! Thank you!

About postgres -- I'm using pgvector to store embeddings for agentic search.