search

pygmystack / pygmy

the pygmy stack is a container stack for local development
MIT License
25 stars 12 forks source link

[bug] Pygmy dnsmasq issues with Colima #471

Open keyle opened 1 year ago

keyle commented 1 year ago

Describe the bug We use the pygmy stack with amazee projects on macOS and I made the transition to Colima from Docker App (enough of that turd).

Everything appears honky dory except for the routing. Dnsmasq is not happy with *.docker.amazee.io. We also found a temporary fix.

To Reproduce Steps to reproduce the behavior:

  1. Install Colima (after uninstalling Docker Desktop)
  1. colima start --cpu 8 --memory 10 --network-address
  2. pygmy up
  3. docker-compose up on an amazee project

Expected behavior Routing should work to project.docker.amazee.io

Output Browser attempts to resolve the name and fails after about 30s.

ERR_NAME_NOT_RESOLVED

12:18 ~ docker exec amazeeio-dnsmasq ping project.docker.amazee.io
ping: bad address 'project.docker.amazee.io'
12:29 ~ docker exec amazeeio-dnsmasq ping amazee.io
PING amazee.io (151.101.2.191): 56 data bytes
64 bytes from 151.101.2.191: seq=0 ttl=63 time=2.217 ms

Additional context We have the following fix: rm -f /etc/resolver/docker.amazee.io following every pygmy up. And it works without it. I am not exactly sure why it works without it, but it certainly doesn't work with it.

Contents of that file:

# Generated by amazeeio pygmy
nameserver 127.0.0.1
domain docker.amazee.io
port 6053

Exported configuration

Defaults: true
Domain: docker.amazee.io
Keys: null
Networks:
  amazeeio-network:
    Attachable: false
    ConfigFrom:
      Network: ""
    ConfigOnly: false
    Containers: null
    Created: "0001-01-01T00:00:00Z"
    Driver: ""
    EnableIPv6: false
    IPAM:
      Config:
      - Gateway: 10.99.99.1
        Subnet: 10.99.99.0/24
      Driver: ""
      Options: null
    Id: ""
    Ingress: false
    Internal: false
    Labels:
      pygmy.name: amazeeio-network
    Name: amazeeio-network
    Options: null
    Scope: ""
Resolvers:
- Data: |
    # Generated by amazeeio pygmy
    nameserver 127.0.0.1
    domain docker.amazee.io
    port 6053
  Enabled: true
  File: docker.amazee.io
  Folder: /etc/resolver
  Name: MacOS Resolver
Services:
  amazeeio-dnsmasq:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd:
      - --log-facility=-
      - -A
      - /docker.amazee.io/127.0.0.1
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/dnsmasq
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-dnsmasq
        pygmy.weight: "13"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd:
      - NET_ADMIN
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        53/tcp:
        - HostIp: ""
          HostPort: "6053"
        53/udp:
        - HostIp: ""
          HostPort: "6053"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-haproxy:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env:
      - AMAZEEIO_URL=docker.amazee.io
      Hostname: ""
      Image: pygmystack/haproxy
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-haproxy
        pygmy.network: amazeeio-network
        pygmy.url: http://docker.amazee.io/stats
        pygmy.weight: "14"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds:
      - /var/run/docker.sock:/tmp/docker.sock
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: ""
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        80/tcp:
        - HostIp: ""
          HostPort: "80"
        443/tcp:
        - HostIp: ""
          HostPort: "443"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-mailhog:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env:
      - MH_UI_BIND_ADDR=0.0.0.0:80
      - MH_API_BIND_ADDR=0.0.0.0:80
      - AMAZEEIO=AMAZEEIO
      - AMAZEEIO_URL=mailhog.docker.amazee.io
      ExposedPorts:
        80/tcp: {}
        1025/tcp: {}
        8025/tcp: {}
      Hostname: ""
      Image: pygmystack/mailhog
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-mailhog
        pygmy.network: amazeeio-network
        pygmy.url: http://mailhog.docker.amazee.io
        pygmy.weight: "15"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: "0"
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: ""
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings:
        1025/tcp:
        - HostIp: ""
          HostPort: "1025"
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-ssh-agent:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/ssh-agent
      Labels:
        pygmy.defaults: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-ssh-agent
        pygmy.network: amazeeio-network
        pygmy.output: "false"
        pygmy.purpose: sshagent
        pygmy.weight: "10"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings: null
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: unless-stopped
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom: null
    NetworkConfig:
      EndpointsConfig: null
  amazeeio-ssh-agent-add-key:
    Config:
      AttachStderr: false
      AttachStdin: false
      AttachStdout: false
      Cmd: null
      Domainname: ""
      Entrypoint: null
      Env: null
      Hostname: ""
      Image: pygmystack/ssh-agent
      Labels:
        pygmy.defaults: "true"
        pygmy.discrete: "true"
        pygmy.enable: "true"
        pygmy.name: amazeeio-ssh-agent-add-key
        pygmy.network: amazeeio-network
        pygmy.output: "false"
        pygmy.purpose: addkeys
        pygmy.weight: "31"
      OnBuild: null
      OpenStdin: false
      StdinOnce: false
      Tty: false
      User: ""
      Volumes: null
      WorkingDir: ""
    HostConfig:
      AutoRemove: false
      Binds: null
      BlkioDeviceReadBps: null
      BlkioDeviceReadIOps: null
      BlkioDeviceWriteBps: null
      BlkioDeviceWriteIOps: null
      BlkioWeight: 0
      BlkioWeightDevice: null
      CapAdd: null
      CapDrop: null
      Cgroup: ""
      CgroupParent: ""
      CgroupnsMode: ""
      ConsoleSize:
      - 0
      - 0
      ContainerIDFile: ""
      CpuCount: 0
      CpuPercent: 0
      CpuPeriod: 0
      CpuQuota: 0
      CpuRealtimePeriod: 0
      CpuRealtimeRuntime: 0
      CpuShares: 0
      CpusetCpus: ""
      CpusetMems: ""
      DeviceCgroupRules: null
      DeviceRequests: null
      Devices: null
      Dns: null
      DnsOptions: null
      DnsSearch: null
      ExtraHosts: null
      GroupAdd: null
      IOMaximumBandwidth: 0
      IOMaximumIOps: 0
      IpcMode: private
      Isolation: ""
      Links: null
      LogConfig:
        Config: null
        Type: ""
      MaskedPaths: null
      Memory: 0
      MemoryReservation: 0
      MemorySwap: 0
      MemorySwappiness: null
      NanoCpus: 0
      NetworkMode: ""
      OomKillDisable: null
      OomScoreAdj: 0
      PidMode: ""
      PidsLimit: null
      PortBindings: null
      Privileged: false
      PublishAllPorts: false
      ReadonlyPaths: null
      ReadonlyRootfs: false
      RestartPolicy:
        MaximumRetryCount: 0
        Name: ""
      SecurityOpt: null
      ShmSize: 0
      UTSMode: ""
      Ulimits: null
      UsernsMode: ""
      VolumeDriver: ""
      VolumesFrom:
      - amazeeio-ssh-agent
    NetworkConfig:
      EndpointsConfig: null
SortedServices:
- amazeeio-ssh-agent
- amazeeio-dnsmasq
- amazeeio-haproxy
- amazeeio-mailhog
- amazeeio-ssh-agent-add-key
Volumes: {}
fubarhouse commented 1 year ago

I'm afraid I'm facing issues getting Colima working on my systems too - facing similar issues. If there's some guidance the community can recommend that would be ideal...

Frankly I have no way to replicate this exactly, but DNS is basically broken in Linux under Colima.

rsau commented 1 year ago

@keyle I had exactly same issue and I can confirm that rm -f /etc/resolver/docker.amazee.io is the only thing that worked for me. I get following if I run pygmy status but everything works without an issue.

image

@fubarhouse

I also had issues with Colima in the past on Mac M1 but after running following, all started working https://github.com/abiosoft/colima/tree/main#upgrading 

colima delete # delete existing instance
colima start
fubarhouse commented 1 year ago

That makes a bit of sense - though it doesn't help my situation.

If you could confirm this @keyle, I could perhaps write a command to write out a config value to disable the resolver.

rsau commented 1 year ago

I also forgot to mention that I always have to run pygmy up twice after switching to Colima from Docker Desktop

First time I always get

image

Second time it starts without any issue.

keyle commented 1 year ago

I can confirm, I delete the resolver after every pygmy up... Otherwise it doesn't work.

fubarhouse commented 1 year ago

@rsau you might be interested in the following solution - it is known but unrelated.

https://github.com/pygmystack/pygmy/issues/442#issuecomment-1558776329

rsau commented 1 year ago

Thanks @fubarhouse In Linux it so easy to fix that but on Mac it is another story.