[bug] pygmy runs extremely slow with Docker Desktop 4.29 on Mac

sonnykt commented 6 months ago

Describe the bug Since I upgraded to Docker Desktop 4.29, all pygmy commands take at least 5 minutes to complete. pygmy status takes 5min, pygmy start takes 12min.

Removing all Pygmy docker images/containers doesn't resolve the issue.

To Reproduce Steps to reproduce the behavior:

Upgrade Docker Desktop to 4.29
Run pygmy commands: status, start, stop

Output A copy of the terminal/shell output where possible/useful.

$ pygmy version
Pygmy version vv0.12.0

$ docker version
Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.0.0
 API version:       1.45
 Go version:        go1.21.8
 Git commit:        2ae903e
 Built:             Wed Mar 20 15:14:46 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.29.0 (145265)
 Engine:
  Version:          26.0.0
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.8
  Git commit:       8b79278
  Built:            Wed Mar 20 15:18:02 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker image ls pygmystack/*
REPOSITORY             TAG       IMAGE ID       CREATED       SIZE
pygmystack/dnsmasq     latest    c06065ce5e6e   5 weeks ago   6.72MB
pygmystack/haproxy     latest    07b85c62e291   5 weeks ago   43.5MB
pygmystack/ssh-agent   latest    e413f08b3d85   5 weeks ago   17.3MB
pygmystack/mailhog     latest    37ddcfddfab8   5 weeks ago   19.8MB

$ time pygmy status
[*] amazeeio-dnsmasq is able to start on port 6053
[*] amazeeio-haproxy is able to start on port 80
[*] amazeeio-haproxy is able to start on port 443
[*] amazeeio-mailhog is able to start on port 1025
[ ] amazeeio-dnsmasq is not running
[ ] amazeeio-haproxy is not running
[ ] amazeeio-mailhog is not running
[ ] amazeeio-ssh-agent is not running
[ ] Resolv MacOS Resolver is not properly connected
pygmy status  0.26s user 0.15s system 0% cpu 4:10.95 total

$ time pygmy start
Successfully started amazeeio-ssh-agent
Successfully started amazeeio-dnsmasq
Successfully started amazeeio-haproxy
Successfully started amazeeio-mailhog
Successfully connected amazeeio-ssh-agent to amazeeio-network
Successfully connected amazeeio-haproxy to amazeeio-network
Successfully connected amazeeio-mailhog to amazeeio-network
Password:
Successfully configured resolvr MacOS Resolver
Validation success for SSH key /Users/sonny/.ssh/id_rsa
Identity added: /Users/sonny/.ssh/id_rsa (/Users/sonny/.ssh/id_rsa)
 - http://docker.amazee.io/stats (amazeeio-haproxy)
 - http://mailhog.docker.amazee.io (amazeeio-mailhog)
pygmy start  0.90s user 0.52s system 0% cpu 12:55.19 total

$ time pygmy status
[*] amazeeio-mailhog: Running as container amazeeio-mailhog
[*] amazeeio-haproxy: Running as container amazeeio-haproxy
[*] amazeeio-dnsmasq: Running as container amazeeio-dnsmasq
[*] amazeeio-ssh-agent: Running as container amazeeio-ssh-agent
[*] Resolv MacOS Resolver is properly connected
W2048 SHA256:630daDeGtjw7OQ2+wVUKQo0a7hm4Ko1LJsmtQLNtcus /Users/sonny/.ssh/id_rsa (RSA)
 - http://docker.amazee.io/stats

 - http://mailhog.docker.amazee.io

pygmy status  0.25s user 0.12s system 34% cpu 1.051 total

Additional context

When Docker is running, all pygmy commands take forever to complete. However, once pygmy start command completes, all lagoon Docker-based projects can use the DNS resolver.
When Docker is NOT running, pygmy status command responds immediately.
Downgrading Docker Desktop to 4.28 seems to resolve the issue.

Exported configuration

Run pygmy export --output output.yml and print the contents from output.yml below:

Defaults: true
Domain: docker.amazee.io
JSONFormat: false
JSONStatus:
  networks: null
  port_availability: null
  resolvers: null
  service_status: null
  ssh_messages: null
  url_validations: null
  volumes: null
Keys: null
Networks:
  amazeeio-network:
    Attachable: false
    ConfigFrom:
      Network: ""
    ConfigOnly: false
    Containers: null
    Created: "0001-01-01T00:00:00Z"
    Driver: ""
    EnableIPv6: false
    IPAM:
      Config:
      - Gateway: 10.99.99.1
        Subnet: 10.99.99.0/24
      Driver: ""
      Options: null
    Id: ""
    Ingress: false
    Internal: false
    Labels:
      pygmy.name: amazeeio-network
    Name: amazeeio-network
    Options: null
    Scope: ""
Resolvers:
- Data: |
    # Generated by amazeeio pygmy
    nameserver 127.0.0.1
    domain docker.amazee.io
    port 6053
  Enabled: true
  File: docker.amazee.io
  Folder: /etc/resolver
  Name: MacOS Resolver
Services:
  amazeeio-dnsmasq:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd:
      - --log-facility=-
      - -A
      - /docker.amazee.io/127.0.0.1
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/dnsmasq
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-dnsmasq
        pygmy.weight: "13"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd:
      - NET_ADMIN
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        53/tcp:
        - HostIp: ""
          HostPort: "6053"
        53/udp:
        - HostIp: ""
          HostPort: "6053"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    Image: pygmystack/dnsmasq
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-haproxy:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env:
      - AMAZEEIO_URL=docker.amazee.io
      Hostname: ""
      Image: pygmystack/haproxy
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-haproxy
        pygmy.network: amazeeio-network
        pygmy.url: http://docker.amazee.io/stats
        pygmy.weight: "14"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds:
      - /var/run/docker.sock:/tmp/docker.sock
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: ""
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        80/tcp:
        - HostIp: ""
          HostPort: "80"
        443/tcp:
        - HostIp: ""
          HostPort: "443"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    Image: pygmystack/haproxy
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-mailhog:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env:
      - MH_UI_BIND_ADDR=0.0.0.0:80
      - MH_API_BIND_ADDR=0.0.0.0:80
      - AMAZEEIO=AMAZEEIO
      - AMAZEEIO_URL=mailhog.docker.amazee.io
      ExposedPorts:
        80/tcp: {}
        1025/tcp: {}
        8025/tcp: {}
      Hostname: ""
      Image: pygmystack/mailhog
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-mailhog
        pygmy.network: amazeeio-network
        pygmy.url: http://mailhog.docker.amazee.io
        pygmy.weight: "15"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: "0"
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: ""
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        1025/tcp:
        - HostIp: ""
          HostPort: "1025"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    Image: pygmystack/mailhog
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-ssh-agent:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/ssh-agent
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-ssh-agent
        pygmy.network: amazeeio-network
        pygmy.output: "false"
        pygmy.purpose: sshagent
        pygmy.weight: "10"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings: null
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    Image: pygmystack/ssh-agent
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-ssh-agent-add-key:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/ssh-agent
      Labels:
        pygmy.defaults: "true"
        pygmy.discrete: "true"
        pygmy.enable: "true"
        pygmy.interactive: "true"
        pygmy.name: amazeeio-ssh-agent-add-key
        pygmy.network: amazeeio-network
        pygmy.output: "false"
        pygmy.purpose: addkeys
        pygmy.weight: "31"
      OnBuild: null
      OpenStdin: true
      StdinOnce: false
      Tty: true
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings: null
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: ""
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom:
      - amazeeio-ssh-agent
    Image: pygmystack/ssh-agent
    NetworkConfig:
      EndpointsConfig: null
SortedServices:
- amazeeio-ssh-agent
- amazeeio-dnsmasq
- amazeeio-haproxy
- amazeeio-mailhog
- amazeeio-ssh-agent-add-key
Volumes: {}

sonnykt commented 6 months ago

Updates:

First upgrade of Docker Desktop 4.29 reveals the issue on pygmy.
Downgrade Docker to 4.28 seems to fix the issue.
Second upgrade of Docker to 4.29 confirms the issue.
Second downgrade of Docker to 4.28 does NOT fix the issue.

I noticed that when there is at least one Lagoon-based project running (with docker compose up), pygmy command runs as fast as expected. Stopping ALL Lagoon-based projects will make pygmy slow again. I can reproduce this behaviour on both Docker 4.29 and 4.28.

tobybellwood commented 6 months ago

ooh - that helps @sonnykt - I can reproduce on my M1, so will dig in a bit.

tobybellwood commented 6 months ago

We've been looking into this in the background, just trying to get a debug handle on where the slowness is...

and then Docker released 4.30, and it seems to be fine again 🤷

tobybellwood@toby-macos drupal-opensearch % time pygmy status
[*] amazeeio-dnsmasq is able to start on port 6053
[*] amazeeio-haproxy is able to start on port 80
[*] amazeeio-haproxy is able to start on port 443
[*] amazeeio-mailhog is able to start on port 1025
[ ] amazeeio-mailhog is not running
[ ] amazeeio-ssh-agent is not running
[ ] amazeeio-dnsmasq is not running
[ ] amazeeio-haproxy is not running
[*] Resolv MacOS Resolver is properly connected
pygmy status  0.26s user 0.08s system 28% cpu 1.231 total
tobybellwood@toby-macos drupal-opensearch % docker version
Client:
 Cloud integration: v1.0.35+desktop.13
 Version:           26.1.1
 API version:       1.45
 Go version:        go1.21.9
 Git commit:        4cf5afa
 Built:             Tue Apr 30 11:44:56 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.30.0 (149282)
 Engine:
  Version:          26.1.1

twerthmueller commented 6 months ago

I can confirm that. I had also the problem with the long running startup and now with the version 4.30 everything runs smooth again! 🎉

sonnykt commented 6 months ago

COnfirming that the issue disappears on Docker Desktop 4.30.

sonnykt commented 3 months ago

Closing this issue as it no longer occurs with Pygmy 0.13 and Docker Desktop 4.33.

pygmystack / pygmy

[bug] pygmy runs extremely slow with Docker Desktop 4.29 on Mac #546