[LocalNet] Add infrastructure to run LLM inference

[okdas]

Summary

Adds infrastructure to run and develop against LLM on LocalNet.

Issue

• 130

Type of change

Select one or more:

• [x] New feature, functionality or library
• [ ] Bug fix
• [ ] Code health or cleanup
• [ ] Documentation
• [ ] Other (specify)

Testing

Documentation changes (only if making doc changes)

• [ ] make docusaurus_start; only needed if you make doc changes

Local Testing (only if making code changes)

• [ ] Unit Tests: make go_develop_and_test
• [ ] LocalNet E2E Tests: make test_e2e
• See quickstart guide for instructions

PR Testing (only if making code changes)

• [ ] DevNet E2E Tests: Add the devnet-test-e2e label to the PR.
  • THIS IS VERY EXPENSIVE, so only do it after all the reviews are complete.
  • Optionally run make trigger_ci if you want to re-trigger tests without any code changes
  • If tests fail, try re-running failed tests only using the GitHub UI as shown here

Sanity Checklist

• [ ] I have tested my changes using the available tooling
• [ ] I have commented my code
• [ ] I have performed a self-review of my own code; both comments & source code
• [ ] I create and reference any new tickets, if applicable
• [ ] I have left TODOs throughout the codebase, if applicable

[okdas]

Note: this functionality is behind the gate and is turned off by default to avoid downloading and serving an LLM to preserve resources. Turn on ollama in localnet_config.yaml when needed.

The infrastructure by itself works. Can run the request with curl:

kubectl exec "$(tilt get kd validator -ojsonpath='{.status.pods[0].name}')" -- \
curl -X POST http://ollama:11434/v1/chat/completions -H "Content-Type: application/json" \
    -d '{
        "model": "qwen:0.5b",
        "messages": [
            {
                "role": "system",
                "content": "You are a helpful assistant."
            },
            {
                "role": "user",
                "content": "Hello!"
            }
        ]
    }'

However, it doesn't seem like we support anything but json-rpc at the moment: https://github.com/pokt-network/poktroll/blob/aba098d8e851ed4883c418f6b8c7cf8a22ed760f/pkg/partials/partial.go#L50

I get the following error:

{"level":"error","error":"got: {\n        \"model\": \"qwen:0.5b\",\n        \"messages\": [\n            {\n                \"role\": \"system\",\n                \"content\": \"You are a helpful assistant.\"\n            },\n            {\n                \"role\": \"user\",\n                \"content\": \"Hello!\"\n            }\n        ]\n    }: unrecognised request format in partial payload","service_id":"ollama","message":"failed getting error reply"}
{"level":"error","error":"got: {\n        \"model\": \"qwen:0.5b\",\n        \"messages\": [\n            {\n                \"role\": \"system\",\n                \"content\": \"You are a helpful assistant.\"\n            },\n            {\n                \"role\": \"user\",\n                \"content\": \"Hello!\"\n            }\n        ]\n    }: unrecognised request format in partial payload","message":"failed getting request type"}

I suggest we merge this as is to unblock work on other than json-rpc request types.

Btw, I picked qwen:0.5b as it was one of the smallest recent LLMs. We don't get hardware optimizations in that environment, so it makes sense to use the smallest possible. We can go crazy on DevNet, though.

[Olshansk]

Great find @okdas.

@red-0ne We'll have to prioritize adding support for gRPC, REST and all the other stuff shortly so we're not limited to just json-rpc.

[Olshansk]

@red-0ne With @okdas OOO for the next week, can you update the branch so we can merge it in please?

It'll help unlock development on non json-rpc.

[red-0ne]

@Olshansk , I added ollama services to supplier_stake_configs with a small change to the config parser to support lower/upper case rpc type values.

[github-actions[bot]]

The CI will now also run the e2e tests on devnet, which increases the time it takes to complete all CI checks. If you just created a pull request, you might need to push another commit to produce a container image DevNet can utilize to spin up infrastructure. You can use make trigger_ci to push an empty commit.

[Olshansk]

@red-0ne I added this TODO in the code: # TODO(#511): Add support forRESTand enabled this.

Assuming E2E tests pass, let's merge it in assuming there are no further changes you deem necessary.

[Olshansk]

@red-0ne - @okdas helped me figure out the issue with E2E bugs, which I resolved in [1]. Are you okay with approving this so we can merge it in and iterate on REST later?

[1] https://github.com/pokt-network/protocol-infra/pull/18