search

pygmystack / pygmy

the pygmy stack is a container stack for local development
MIT License
25 stars 12 forks source link

[feature] Support Colima/Docker Contexts #397

Closed mathieuhelie closed 1 year ago

mathieuhelie commented 1 year ago

Colima is now a popular free and open alternative to Docker for Mac. I would like to use Colima to run Pygmy for Lagoon development.

When attempting to execute Pygmy with a Colima context instantiated, Pygmy reports Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

The colima daemon resides at ~/.colima/daemon.sock. It appears that pygmy hardcodes unix:///var/run/docker.sock as its source of docker daemon.

Describe the solution you'd like Support the docker context feature, as in https://stackoverflow.com/questions/72557053/why-does-colima-failed-to-find-docker-daemon#comment128492293_72560928

Describe alternatives you've considered I've attempted manually creating a symlink such as in this project: https://github.com/cloudfoundry-incubator/kubo-release/issues/329 This worked for running the pygmy update command, but the pygmy up command failed with error during connect: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/containers/amazeeio-ssh-agent-add-key/start": EOF - this is likely because a link is also needed within a container.

fubarhouse commented 1 year ago

A configurable socket path should be well and truly possible, would this satisfy your requirements?

If all that's needed is a path change, support shouldn't be an issue.

I'll be getting my laptop back next week, so assuming this would help you, I can have a look at this then..

mathieuhelie commented 1 year ago

@fubarhouse I'm not certain of the extent of the requirements imposed by Pygmy on DockerForMac. After finagling the daemon aliases I was able to initiate most of the containers, but the networking element on DNSmasq eluded me. The ssh key agent didn't work at all. There may be more.

fubarhouse commented 1 year ago

Could you please try out this one, and we'll see how far we can get to support Colima. I've been curious for some time, though I don't use MacOS much anymore...

https://github.com/pygmystack/pygmy/pull/401

Would love to know what you find!

mathieuhelie commented 1 year ago

I was unsuccessful.

pygmy/pygmy/builds   issue/397  ./pygmy-darwin-arm64 up
Using config file: /Users/mathieu.helie/.pygmy.yml
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x1027bea04]

goroutine 1 [running]:
encoding/json.(*Decoder).refill(0x1400007c000)
    /usr/local/go/src/encoding/json/stream.go:165 +0x144
encoding/json.(*Decoder).readValue(0x1400007c000)
    /usr/local/go/src/encoding/json/stream.go:140 +0x9c
encoding/json.(*Decoder).Decode(0x1400007c000, {0x102cc5560, 0x1400000e020})
    /usr/local/go/src/encoding/json/stream.go:63 +0x60
github.com/pygmystack/pygmy/service/interface/docker.DockerPull({0x102b9dca0, 0x14})
    /go/src/github.com/pygmystack/pygmy/service/interface/docker/docker.go:146 +0x47c
github.com/pygmystack/pygmy/service/interface.(*Service).Setup(0x1400026f738)
    /go/src/github.com/pygmystack/pygmy/service/interface/interface.go:33 +0x7c
github.com/pygmystack/pygmy/service/library.Up({{0x140003075a0, 0x1, 0x1}, {0x0, 0x0}, 0x0, {0x0, 0x0, 0x0}, 0x0, ...})
    /go/src/github.com/pygmystack/pygmy/service/library/up.go:61 +0x18cc
github.com/pygmystack/pygmy/cmd.glob..func9(0x103128c20?, {0x102b95f96?, 0x0?, 0x0?})
    /go/src/github.com/pygmystack/pygmy/cmd/up.go:69 +0x1bc
github.com/spf13/cobra.(*Command).execute(0x103128c20, {0x103164418, 0x0, 0x0})
    /go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:856 +0x4ac
github.com/spf13/cobra.(*Command).ExecuteC(0x10312a2a0)
    /go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:960 +0x34c
github.com/spf13/cobra.(*Command).Execute(...)
    /go/pkg/mod/github.com/spf13/cobra@v1.1.3/command.go:897
github.com/pygmystack/pygmy/cmd.Execute()
    /go/src/github.com/pygmystack/pygmy/cmd/root.go:58 +0x28
main.main()
    /go/src/github.com/pygmystack/pygmy/main.go:28 +0x1c

pygmy/pygmy/builds   issue/397  cat ~/.pygmy.yml
socket: /Users/mathieu.helie/.colima/default/docker.sock

pygmy/pygmy/builds   issue/397  ./pygmy-darwin-arm64 version
Using config file: /Users/mathieu.helie/.pygmy.yml
Pygmy version unidentifiable.

pygmy/pygmy/builds   issue/397  ./pygmy-darwin-arm64 status
Using config file: /Users/mathieu.helie/.pygmy.yml
[*] amazeeio-mailhog is able to start on port 1025
[*] amazeeio-dnsmasq is able to start on port 6053
[*] amazeeio-haproxy is able to start on port 80
[*] amazeeio-haproxy is able to start on port 443

[ ] amazeeio-haproxy is not running
[ ] amazeeio-mailhog is not running
[ ] amazeeio-ssh-agent is not running
[ ] amazeeio-dnsmasq is not running
[*] Resolv MacOS Resolver is properly connected
fubarhouse commented 1 year ago

Alright I've got colima going myself and worked some things out... This would be a pretty major refactor so I will need some more time.

I'll keep you posted! 😃

fubarhouse commented 1 year ago

I've reverted, the changes were... desired but significant. I'll open a new issue for that change, but in the meantime, could you please try something like the following on the same branch?

PYGMY_SOCKET=unix:///var/run/docker.sock go run main.go status

This should at least give a basic indication of if colima could be compatible.

mathieuhelie commented 1 year ago 
pygmy/pygmy   issue/397  git pull
remote: Enumerating objects: 44, done.
remote: Counting objects: 100% (44/44), done.
remote: Compressing objects: 100% (15/15), done.
remote: Total 44 (delta 29), reused 43 (delta 28), pack-reused 0
Unpacking objects: 100% (44/44), 9.38 KiB | 436.00 KiB/s, done.
From https://github.com/pygmystack/pygmy
   631f8f2..ff9a818  issue/397  -> origin/issue/397
   ddf4365..8d4ce86  issue/398  -> origin/issue/398
Updating 631f8f2..ff9a818
Fast-forward
 service/haproxy/haproxy.go         |  6 +++---
 service/interface/docker/docker.go | 68 +++++++++++++++++++++++++++++---------------------------------------
 service/interface/interface.go     | 16 +---------------
 service/interface/types.go         |  2 --
 service/library/network.go         |  3 +--
 service/library/setup.go           |  6 +-----
 6 files changed, 35 insertions(+), 66 deletions(-)
pygmy/pygmy   issue/397  PYGMY_SOCKET=unix:///var/run/docker.sock go run main.go status
Using config file: /Users/mathieu.helie/.pygmy.yml
[*] amazeeio-dnsmasq is able to start on port 6053
[*] amazeeio-haproxy is able to start on port 80
[*] amazeeio-haproxy is able to start on port 443
[*] amazeeio-mailhog is able to start on port 1025

[ ] amazeeio-dnsmasq is not running
[ ] amazeeio-haproxy is not running
[ ] amazeeio-mailhog is not running
[ ] amazeeio-ssh-agent is not running
[*] Resolv MacOS Resolver is properly connected
pygmy/pygmy   issue/397  ls -lah /var/run/docker.sock
ls: /var/run/docker.sock: No such file or directory

Seems to work.

dasrecht commented 1 year ago

Confirmed the PYGMY_SOCKET approach on colima and that worked without any issue :)

fubarhouse commented 1 year ago

I'll clean this up and get it merged in the next few days. Thanks for helping!

mathieuhelie commented 1 year ago

Before you merge, note that the "pygmy up" is still failing.

Perhaps that is part of your cleanup?

fubarhouse commented 1 year ago

I'm pretty sure the branch is now ready for merging @mathieuhelie - did you want to give it one more try?

Don't forget to set PYGMY_SOCKET! 😃

mathieuhelie commented 1 year ago

Appears successful! Thank you!

Note: the unix:// path of the socket is important.

PYGMY_SOCKET=unix:///Users/mathieu.helie/.colima/default/docker.sock ./builds/pygmy-darwin-arm64 up
Using config file: /Users/mathieu.helie/.pygmy.yml
Successfully started amazeeio-dnsmasq
Successfully started amazeeio-haproxy
Successfully started amazeeio-mailhog
Already connected amazeeio-haproxy to amazeeio-network
Already connected amazeeio-mailhog to amazeeio-network
Already connected amazeeio-ssh-agent-add-key to amazeeio-network
Already connected amazeeio-ssh-agent-add-key to amazeeio-network
Password:
Successfully configured resolvr MacOS Resolver
 - http://mailhog.docker.amazee.io (amazeeio-mailhog)
 - http://docker.amazee.io/stats (amazeeio-haproxy)

Without unix:// the command returns an ambiguous error.

PYGMY_SOCKET=/Users/mathieu.helie/.colima/docker.sock ./builds/pygmy-darwin-arm64 up
Using config file: /Users/mathieu.helie/.pygmy.yml
unable to parse docker host `/Users/mathieu.helie/.colima/docker.sock`
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x71 pc=0x1012831e4]

goroutine 1 [running]:
github.com/docker/docker/client.(*Client).getAPIPath(0x1400011c750?, {0x1015e1230?, 0x14000126008?}, {0x1013d3d69, 0x10}, 0x5?)
    /go/pkg/mod/github.com/docker/docker@v20.10.17+incompatible/client/client.go:183 +0x34
mathieuhelie commented 1 year ago

Unfortunately, the containers themselves appear confused about the location of the socket:

docker logs amazeeio-haproxy
2022/11/28 14:59:37 Error retrieving docker server version info: cannot connect to Docker endpoint
2022/11/28 14:59:37 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 14:59:37 Error listing containers: cannot connect to Docker endpoint
2022/11/28 14:59:37 Watching docker events
2022/11/28 14:59:37 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 14:59:37 Error listing containers: cannot connect to Docker endpoint
2022/11/28 14:59:37 Docker daemon connection interrupted
2022/11/28 14:59:47 Watching docker events
2022/11/28 14:59:47 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 14:59:47 Error listing containers: cannot connect to Docker endpoint
2022/11/28 14:59:48 Docker daemon connection interrupted
2022/11/28 14:59:58 Watching docker events
2022/11/28 14:59:58 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 14:59:58 Error listing containers: cannot connect to Docker endpoint
2022/11/28 14:59:58 Docker daemon connection interrupted
2022/11/28 15:00:08 Watching docker events
2022/11/28 15:00:08 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 15:00:08 Error listing containers: cannot connect to Docker endpoint
2022/11/28 15:00:08 Docker daemon connection interrupted
2022/11/28 15:00:18 Watching docker events
2022/11/28 15:00:18 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 15:00:18 Error listing containers: cannot connect to Docker endpoint
2022/11/28 15:00:19 Docker daemon connection interrupted
2022/11/28 15:00:29 Watching docker events
2022/11/28 15:00:29 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 15:00:29 Error listing containers: cannot connect to Docker endpoint
2022/11/28 15:00:29 Docker daemon connection interrupted
2022/11/28 15:00:39 Watching docker events
2022/11/28 15:00:39 Error retrieving docker server info: cannot connect to Docker endpoint
2022/11/28 15:00:39 Error listing containers: cannot connect to Docker endpoint
2022/11/28 15:00:39 Docker daemon connection interrupted
 docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED         STATUS          PORTS                                                                          NAMES
3063f8f09cde   pygmystack/mailhog     "MailHog"                3 minutes ago   Up 3 minutes    80/tcp, 8025/tcp, 0.0.0.0:1025->1025/tcp, :::1025->1025/tcp                    amazeeio-mailhog
a19ae3a0c1b2   pygmystack/haproxy     "/app/docker-entrypo…"   3 minutes ago   Up 3 minutes    0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp       amazeeio-haproxy
34d50e988376   pygmystack/dnsmasq     "dnsmasq -k --log-fa…"   3 minutes ago   Up 3 minutes    0.0.0.0:6053->53/tcp, 0.0.0.0:6053->53/udp, :::6053->53/tcp, :::6053->53/udp   amazeeio-dnsmasq
c8abf9868cf7   pygmystack/ssh-agent   "/run.sh ssh-agent"      2 months ago    Up 46 minutes                                                                                  amazeeio-ssh-agent
 docker logs amazeeio-dnsmasq
dnsmasq[1]: started, version 2.85 cachesize 150
dnsmasq[1]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth cryptohash DNSSEC loop-detect inotify dumpfile
dnsmasq[1]: reading /etc/resolv.conf
dnsmasq[1]: using nameserver 1.1.1.1#53
dnsmasq[1]: using nameserver 8.8.8.8#53
dnsmasq[1]: read /etc/hosts - 7 addresses
fubarhouse commented 1 year ago

Appreciate those details - I'll continue on this tonight.

fubarhouse commented 1 year ago

I've added input validation to ensure the raw socket path pygmy uses includes the prefix unix:// I've also added some validation to the output to indicate if HAProxy is able or unable to connect to the docker endpoint.

I just need to get Colima working locally now to do some testing...

edit: feature technically works, but looks like I'll be in the weeds configuring colima before I can get a positive outcome.

yusufhm commented 1 year ago

Hi guys, I started working on this locally before seeing this issue or the PR @fubarhouse created.

I wanted to take advantage of the docker contexts directly. I initially wanted to use the docker cli packages to work out the context as done here, but there were conflicts when importing the packages.

So I just created simple structs to read the docker config & manifest files directly and read them.

This should allow us to use pygmy without making any changes.

Developed and tested on Mac M1.

fubarhouse commented 1 year ago

Handy to know. I've so far been unable to test the Colima PR properly. If there's a lower-friction way to achieve this, we should document it so that more people know they can do this.

Ideally, Docker should just work with the active context because Pygmy does not attempt to target any specific context.