Inference Failed Because of '500 Internal Server Error'

[dawenxi-007]

After launching the distribution server by "llama distribution start --name local-llama-8b --port 5000 --disable-ipv6 ", running any inference example, for example "python examples/scripts/vacation.py localhost 5000 --disable-safety" will give the following error:

Traceback (most recent call last):
  File "/home/ubuntu/taoz/llama-agentic-system/examples/scripts/hello.py", line 34, in <module>
    fire.Fire(main)
  File "/home/ubuntu/miniforge3/envs/llama-stack/lib/python3.10/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/ubuntu/miniforge3/envs/llama-stack/lib/python3.10/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/ubuntu/miniforge3/envs/llama-stack/lib/python3.10/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/home/ubuntu/taoz/llama-agentic-system/examples/scripts/hello.py", line 18, in main                                                                                          [58/582]
    asyncio.run(
  File "/home/ubuntu/miniforge3/envs/llama-stack/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/home/ubuntu/miniforge3/envs/llama-stack/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/taoz/llama-agentic-system/examples/scripts/multi_turn.py", line 41, in run_main
    client = await get_agent_system_instance(
  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/agentic_system/utils.py", line 121, in get_agent_system_instance
    create_response = await api.create_agentic_system(create_request)
  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/agentic_system/client.py", line 56, in create_agentic_system
    response.raise_for_status()                                                                                                                                                      [47/582]
  File "/home/ubuntu/miniforge3/envs/llama-stack/lib/python3.10/site-packages/httpx/_models.py", line 761, in raise_for_status
    raise HTTPStatusError(message, request=request, response=self)
httpx.HTTPStatusError: Server error '500 Internal Server Error' for url 'http://localhost:5000/agentic_system/create'
For more information check: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/500

[hardikjshah]

Hey @dawenxi-007 , can you print the distribution side logs if you see any.

[dawenxi-007]

Here is what I saw:

  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/distribution/server.py", line 202, in endpoint
    await func(request)
  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/agentic_system/meta_reference/agentic_system.py", line 86, in create_agentic_system
    raise ValueError("Wolfram API key not defined in config")
ValueError: Wolfram API key not defined in config
INFO:     127.0.0.1:37490 - "POST /agentic_system/create HTTP/1.1" 500 Internal Server Error
Wolfram API key not defined in config
Traceback (most recent call last):
  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/distribution/server.py", line 202, in endpoint
    await func(request)
  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/agentic_system/meta_reference/agentic_system.py", line 86, in create_agentic_system
    raise ValueError("Wolfram API key not defined in config")
ValueError: Wolfram API key not defined in config
INFO:     127.0.0.1:58112 - "POST /agentic_system/create HTTP/1.1" 500 Internal Server Error
Wolfram API key not defined in config
Traceback (most recent call last):
  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/distribution/server.py", line 202, in endpoint
    await func(request)
  File "/home/ubuntu/taoz/llama-stack/llama_toolchain/agentic_system/meta_reference/agentic_system.py", line 86, in create_agentic_system
    raise ValueError("Wolfram API key not defined in config")
ValueError: Wolfram API key not defined in config
INFO:     127.0.0.1:54880 - "POST /agentic_system/create HTTP/1.1" 500 Internal Server Error

It was asking for Wolfram API key. I didn't see this message if I run the flow from the llama agentic system git repo.

[hardikjshah]

Yeah we changed some things on how distributions manage keys ( its now in the config instead of using env variables ). Can you run llama distribution configure --name local-llama-8b again , that should help set up the keys properly and resolve this issue.

[dawenxi-007]

Yes, I noticed that change. Now the ask for set up the keys is at the configuration stage. However, shouldn't the setting for the key should be optional? Like in this case, it only asks for Wolfram API key but not Brave Search API key. Also, there could be some inference scenario may not use the related tool APIs.

[ashwinb]

Like in this case, it only asks for Wolfram API key but not Brave Search API key.

I think it just fails as soon as it needs to use the Wolfram API key. See https://github.com/meta-llama/llama-stack/blob/main/llama_toolchain/agentic_system/meta_reference/agentic_system.py#L83 The next iteration of that loop (if you did provide the wolfram key but did not provide Brave Search key) will fail at the Brave Search step.