My mac doesn't like the required Nvidia device allocation, so I needed to comment it out. Since docker-compose overrides don't really support removing things, I'm not sure what a good way to make this optional is. I was able to get llama3:8b working reasonably (a few words a second) on my macbook pro without this deploy section.
index cf24abd..f97c4b8 100644
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -9,13 +9,13 @@ services:
#ports:
# # Standard port for Ollama. Don't really want to expose this unencrypted.
# - 127.0.0.1:11434:11434
- deploy:
- resources:
- reservations:
- devices:
- - capabilities: [gpu]
- driver: nvidia
- count: all
+ # deploy:
+ # resources:
+ # reservations:
+ # devices:
+ # - capabilities: [gpu]
+ # driver: nvidia
+ # count: all
restart: unless-stopped
Also, maybe consider some additional env variables for setting the http/s ports. My mac doesn't like me exposing to privileged ports like 443 and 80:
I know how to layer in settings like the GPU settings. I just didn't get around to that in the first pass. Great idea though, and reminder here as an issue, to factor that out in a future pass.
My mac doesn't like the required Nvidia device allocation, so I needed to comment it out. Since docker-compose overrides don't really support removing things, I'm not sure what a good way to make this optional is. I was able to get llama3:8b working reasonably (a few words a second) on my macbook pro without this deploy section.
Also, maybe consider some additional env variables for setting the http/s ports. My mac doesn't like me exposing to privileged ports like 443 and 80: