Support Docker

[bodybreaker]

Support docker for load balancer

BUILD

docker build -t paddler-load-balancer .

RUN

docker run --rm -p 8085:8085 -p 8080:8080 paddler-load-balancer \
    --management-host 127.0.0.1 \
    --management-port 8085 \
    --reverseproxy-host 196.168.2.10 \
    --reverseproxy-port 8080

[mcharytoniuk]

@bodybreaker Thank you for your contribution! :)

I will merge it, but I need one change first. Can you please modify CMD ["paddler", "balancer"] to ENTRYPOINT ["paddler"]? The current Dockerfile can only run the balancer; with ENTRYPOINT, it would be possible to run both balancer and agent.

For example (with ENTRYPOINT):

Balancer

docker run --rm -p 8085:8085 -p 8080:8080 paddler balancer \
    --management-host 127.0.0.1 \
    --management-port 8085 \
    --reverseproxy-host 196.168.2.10 \
    --reverseproxy-port 8080

Agent

docker run --rm -p 8085:8085 -p 8080:8080 paddler agent \
    --external-llamacpp-host 127.0.0.1 \
    --external-llamacpp-port 8088 \
    --local-llamacpp-host 127.0.0.1 \
    --local-llamacpp-port 8088 \
    --management-host 127.0.0.1 \
    --management-port 8085

[bodybreaker]

@bodybreaker Thank you for your contribution! :)

I will merge it, but I need one change first. Can you please modify CMD ["paddler", "balancer"] to ENTRYPOINT ["paddler"]? The current Dockerfile can only run the balancer; with ENTRYPOINT, it would be possible to run both balancer and agent.

For example (with ENTRYPOINT):

Balancer

docker run --rm -p 8085:8085 -p 8080:8080 paddler balancer \
    --management-host 127.0.0.1 \
    --management-port 8085 \
    --reverseproxy-host 196.168.2.10 \
    --reverseproxy-port 8080

Agent

docker run --rm -p 8085:8085 -p 8080:8080 paddler agent \
    --external-llamacpp-host 127.0.0.1 \
    --external-llamacpp-port 8088 \
    --local-llamacpp-host 127.0.0.1 \
    --local-llamacpp-port 8088 \
    --management-host 127.0.0.1 \
    --management-port 8085

It seems I missed that part. I have corrected it and committed the changes.

It seems like a good idea to run llama.cpp within a Docker container, with the container also including the Paddler Agent for operation. I'm working on it but do you have an any idea?

[mcharytoniuk]

Thank you! :)

It seems like a good idea to run llama.cpp within a Docker container, with the container also including the Paddler Agent for operation. I'm working on it but do you have an any idea?

llama.cpp has Docker packages: https://github.com/ggerganov/llama.cpp/pkgs/container/llama.cpp

I would start with those and add Paddler to them. Something like:

FROM ghcr.io/ggerganov/llama.cpp:server-cuda

# (...) install Paddler here (...)

Another approach would be to use composition (where Paddler and llama.cpp containers are running alongside and communicating with each other).

Also, by the way, I am ok with using Container Registry in this repo (similar to how llama.cpp does it: https://github.com/ggerganov/llama.cpp/pkgs/container/llama.cpp). That may make it easier to publish more Docker images.

[mcharytoniuk]

@bodybreaker This issue got closed because I merged your PR (thanks again!), but we can continue discussing adding more Docker containers. I am open to having them here and am happy you suggested it. I started a new topic here: https://github.com/distantmagic/paddler/discussions/14