Local LLM agent experimentation

[bdzimmer]

Updates for a version of bank_agent that uses a local LLM. I've gotten the best results by far with this model: phind-codellama-34b-v2. You can download the quantized version from HuggingFace Hub here: https://huggingface.co/TheBloke/Phind-CodeLlama-34B-v2-GGUF (phind-codellama-34b-v2.Q4_K_M.gguf) and put it in the MODELS_DIR_PATH defined in project

• Model File URL

Additional requirements:

• CMAKE_ARGS="-DLLAMA_METAL=on" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.82
  • The CMAKE_ARGS / FORCE_CMAKE variables are required to build with metal support for running on Apple Silicon GPUs and won't be necessary / may not work on other platforms.

Entry points:

• python -m examples.bank_agent_llama - local llama version of examples.bank_agent.
  • This follows Justin's bankagent / react_chat closely for the sake of comparison. I've tried to leave comments here and in neosophia.agents.simplelocal to highlight the differences, mostly prompt changes and additional formatting reminders.
• python -m examples.bank_agent_eval - run evaluation and compare agents using different LLMs, including OpenAI and local.
  • An update of note here is that the system abstraction now uses the run index as a seed, so you can get a range of behavior out of deterministic local LLMs. The seed is ignored for OpenAI LLMs.
  • I've added additional questions, some of which don't always work with the local LLM.

If you're not running on Apple Silicon, you should probably set GPU_ENABLE = False.

[jmpounders]

Is llama_cpp the only additional package that needs to be installed? Recommend updating the requirements.txt, esp if a certain version is needed.

[jmpounders]

Is llama_cpp the only additional package that needs to be installed? Recommend updating the requirements.txt, esp if a certain version is needed.

Reading the description closer now... can you update the requirements for llama_cpp and still get the right cmake flags, or should that always be a one-off installation?

[bdzimmer]

@jmpounders I'll look into it, might have to make an install script since it's setting environment variables but there may be a way to do that in requirements.txt. That being said, might be better to keep the metal flags separate since that's platform specific. And I'm realizing I should have made a note about that somewhere in the PR, heh.