portainer / agent

The Portainer agent
https://www.portainer.io
zlib License
319 stars 71 forks source link

Portainer Agent: Unable to retrieve local agent IP address #161

Open kevdogg opened 3 years ago

kevdogg commented 3 years ago

Hi I'm running Agent version 2.0.0 on Arch linux host with Docker

I'm running docker in swarm mode, however I'm just connecting single instances (no replicas) over the swarm network:

The actual portainer installation is on a different Docker Daemon.

My docker compose file appears as the following (these are snippets):

networks:
  net:
    name: net
    driver: bridge
  watchtower-ubuntumc:
    name: watchtower_ubuntumc
    driver: bridge
  openldap-net:
    name: openldap-net
    driver: overlay
    attachable: true
    ipam:
      config:
        - subnet: 10.90.0.0/24

services: 
  portainer-agent:
    container_name: portainer-agent
    image: portainer/agent:2.0.0
    labels:
      - "com.centurylinklabs.watchtower.scope=archzfsproxy"
    restart: always
    secrets:
      - portainer-agent_secret
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    ports:
      - target: 9001
        published: 9001
        protocol: tcp
        mode: host
    networks:
      - net
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]
    environment:
      - AGENT_SECRET=/run/secrets/portainer-agent_secret
      - LOG_LEVEL=debug

I'm getting the following errors starting the portainer agent:

2020/12/26 08:39:05 [INFO] [main] [message: Agent running on Docker platform]
2020/12/26 08:39:05 [DEBUG] [main,configuration] [Member tags: &{AgentPort:9001 EdgeKeySet:false NodeName:archZFSProxy.domain.com DockerConfiguration:{EngineStatus:2 Leader:true NodeRole:1} KubernetesConfiguration:{}}]
2020/12/26 08:39:05 [INFO] [main] [message: Agent running on a Swarm cluster node. Running in cluster mode]
2020/12/26 08:39:05 [DEBUG] [docker] [network_name: net] [scope: local] [ingress: false] [message: Skipping invalid container network]
2020/12/26 08:39:05 [ERROR] [main,docker] [message: Unable to retrieve local agent IP address] [error: unable to retrieve the address on which the agent can advertise. Check your network settings]
 sudo docker inspect portainer-agent
[
    {
        "Id": "071c8718c5936d231ac13d9747f2c278a00c97d5e62ae3fa0d736687c151baa7",
        "Created": "2020-12-26T08:34:55.386882817Z",
        "Path": "./agent",
        "Args": [],
        "State": {
            "Status": "restarting",
            "Running": true,
            "Paused": false,
            "Restarting": true,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 1,
            "Error": "",
            "StartedAt": "2020-12-26T08:40:07.95189253Z",
            "FinishedAt": "2020-12-26T08:40:08.046910704Z"
        },
        "Image": "sha256:6b367a5c4fe3e272ae9fb6cf8393856562c99a6eb41d2922c8ab75a3e37d1813",
        "ResolvConfPath": "/var/lib/docker/containers/071c8718c5936d231ac13d9747f2c278a00c97d5e62ae3fa0d736687c151baa7/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/071c8718c5936d231ac13d9747f2c278a00c97d5e62ae3fa0d736687c151baa7/hostname",
        "HostsPath": "/var/lib/docker/containers/071c8718c5936d231ac13d9747f2c278a00c97d5e62ae3fa0d736687c151baa7/hosts",
        "LogPath": "/var/lib/docker/containers/071c8718c5936d231ac13d9747f2c278a00c97d5e62ae3fa0d736687c151baa7/071c8718c5936d231ac13d9747f2c278a00c97d5e62ae3fa0d736687c151baa7-json.log",
        "Name": "/portainer-agent",
        "RestartCount": 14,
        "Driver": "zfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/var/run/docker.sock:/var/run/docker.sock:rw",
                "/var/lib/docker/volumes:/var/lib/docker/volumes:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "net",
            "PortBindings": {
                "9001/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "9001"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "always",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "Mounts": [
                {
                    "Type": "bind",
                    "Source": "/etc/docker/compose/authelia/portainer-agent.secret",
                    "Target": "/run/secrets/portainer-agent_secret",
                    "ReadOnly": true
                }
            ],
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": {
                "Dataset": "tank/ROOT/default/a640d79fa430beb1bf25c16f10f7c32a261518057af75bacd6383ddd61721168",
                "Mountpoint": "/var/lib/docker/zfs/graph/a640d79fa430beb1bf25c16f10f7c32a261518057af75bacd6383ddd61721168"
            },
            "Name": "zfs"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/var/run/docker.sock",
                "Destination": "/var/run/docker.sock",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rprivate"
            },
            {
                "Type": "bind",
                "Source": "/var/lib/docker/volumes",
                "Destination": "/var/lib/docker/volumes",
                "Mode": "rw",
                "RW": true,
                "Propagation": "rslave"
            },
            {
                "Type": "bind",
                "Source": "/etc/docker/compose/authelia/portainer-agent.secret",
                "Destination": "/run/secrets/portainer-agent_secret",
                "Mode": "",
                "RW": false,
                "Propagation": "rprivate"
            }
        ],
        "Config": {
            "Hostname": "071c8718c593",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "9001/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "AGENT_SECRET=/run/secrets/portainer-agent_secret",
                "LOG_LEVEL=debug",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": null,
            "Image": "portainer/agent:2.0.0",
            "Volumes": {
                "/var/lib/docker/volumes": {},
                "/var/run/docker.sock": {}
            },
            "WorkingDir": "/app",
            "Entrypoint": [
                "./agent"
            ],
            "OnBuild": null,
            "Labels": {
                "com.centurylinklabs.watchtower.scope": "archzfsproxy",
                "com.docker.compose.config-hash": "109b860d435924a1cde851eeea0fd7b5a528fd8df1de9faeb2bc641cbd2ece1f",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "authelia",
                "com.docker.compose.project.config_files": "docker-compose.yml",
                "com.docker.compose.project.working_dir": "/etc/docker/compose/authelia",
                "com.docker.compose.service": "portainer-agent",
                "com.docker.compose.version": "1.27.4"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "b93d11a21fd3359f1e0f633bf24e9cd591f13da25d9e4eb2b32bac058f4ed9da",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/b93d11a21fd3",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "net": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "071c8718c593",
                        "portainer-agent"
                    ],
                    "NetworkID": "9ef3e7f522b228bd92824e6c142a2b04b7e2f7eba9e2e6b75655a541b25dc023",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "",
                    "DriverOpts": null
                }
            }
        }
    }
]

I've seen this error reported about 1 year ago (https://github.com/portainer/agent/issues/95), however I didn't see what the actual solution was. Obviously if I use the TCP/Direct Route with OpenSSL self-signed client server certs -- I bypass the agent and don't get the error.

deviantony commented 3 years ago

Hi @kevdogg

The agent must be part of an overlay network to work properly on a Swarm cluster. Have you tried to update the driver property of the net network to overlay?

realdeadbeef commented 3 years ago

I have this same problem, I've got a feeling that it's to do with the fact that my server has a Wi-Fi connection and Ethernet connection both with different IP addresses, is there a way to get the agent to just bind to one of the IP addresses. eg: [docker deployment command] --bind 192.168.10.4:9001

KopfKrieg commented 3 years ago

I've got a feeling that it's to do with the fact that my server has a Wi-Fi connection and Ethernet connection both with different IP addresses

I don't think that's the issue. My homeserver has a (internal) static ip address (via ethernet, no Wi-Fi connection), however, my public IP address changes (but I'm using DynDNS, so this shouldn't be a problem).

is there a way to get the agent to just bind to one of the IP addresses. eg: [docker deployment command] --bind 192.168.10.4:9001

That's possible via the -p/publish argument: https://docs.docker.com/config/containers/container-networking/

LeehamElectronics commented 2 years ago

I am still getting this issue, I have tried many different version of the agent:

2022/03/08 04:39:20 [INFO] [main] [message: Agent running on Docker platform]
2022/03/08 04:39:20 [INFO] [main] [message: Agent running on a Swarm cluster node. Running in cluster mode]
2022/03/08 04:39:20 [ERROR] [main,docker] [message: Unable to retrieve local agent IP address] [error: unable to retrieve the address on which the agent can advertise. Check your network settings]

Any ideas? It's running on a bridge network, I have tried to change it over to an overlay network but it does not let me select any overlay networks from the drop-down box in the Portainer UI.

LeehamElectronics commented 2 years ago

nvm, this fixed it here: https://github.com/portainer/agent/issues/95#issuecomment-902167711

I had the same issue. I fix this running this command

sudo docker swarm leave --force

(In my case, I don't need swarm)

AIndoria commented 2 years ago

nvm, this fixed it here: #95 (comment)

I had the same issue. I fix this running this command sudo docker swarm leave --force (In my case, I don't need swarm)

this isn't a solution for many of us who use docker swarm.

deviantony commented 2 years ago

With the merge of https://github.com/portainer/agent/pull/239 this shouldn't be an issue anymore I believe. Give a try to portainerci/agent:develop to see if that could solve the issue.

DennisYildirim commented 2 years ago

Sorry @deviantony does this mean that in the next version of the portainer, this won't be a problem anymore.

I'm using portainer 2.14.2 on both Desktop PC (with linux subsystem docker) and Synology Server. Have been trying to connect both to each other via using the agent, but it gives the following error log on the Desktop PC.

Both dockers are not srawm enabled. They are stand alone dockers, and I'm purely trying to connect them via the agent which was the easiest way.

Is there a solution to this yet?

I even added network host option to agent, but that did not solve the issue too.

2022/08/21 11:18:31 [INFO] [main] [message: Agent running on Docker platform] 2022/08/21 11:18:31 [WARN] [main] [message: Unable to retrieve agent container IP address, using '0.0.0.0' instead] [error: Error: No such container: docker-desktop] 2022/08/21 11:18:31 [INFO] [edge] [message: Edge key loaded from options] 2022/08/21 11:18:31 [INFO] [edge,registry] [message: Starting registry credential server] 2022/08/21 11:18:31 [INFO] [http] [server_addr: 0.0.0.0] [server_port: 9001] [use_tls: false] [api_version: 2.14.2] [message: Starting Agent API server] 2022/08/21 11:18:51 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:19:02 client: Connection error: dial tcp 172.67.198.19:8000: operation was canceled 2022/08/21 11:19:02 client: Give up 2022/08/21 11:19:38 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:19:48 client: Connection error: dial tcp 172.67.198.19:8000: operation was canceled 2022/08/21 11:19:48 client: Give up 2022/08/21 11:20:37 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:20:47 client: Connection error: dial tcp 172.67.198.19:8000: operation was canceled 2022/08/21 11:20:47 client: Give up 2022/08/21 11:21:37 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:21:48 client: Connection error: dial tcp 172.67.198.19:8000: operation was canceled 2022/08/21 11:21:48 client: Give up 2022/08/21 11:22:37 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:22:48 client: Connection error: dial tcp 172.67.198.19:8000: operation was canceled 2022/08/21 11:22:48 client: Give up 2022/08/21 11:23:37 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:23:47 client: Connection error: dial tcp 104.21.52.103:8000: operation was canceled 2022/08/21 11:23:47 client: Give up 2022/08/21 11:24:38 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:24:46 client: Connection error: dial tcp 104.21.52.103:8000: operation was canceled 2022/08/21 11:24:46 client: Give up 2022/08/21 11:25:52 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:26:02 client: Connection error: dial tcp 104.21.52.103:8000: operation was canceled 2022/08/21 11:26:02 client: Give up 2022/08/21 11:27:18 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:27:27 client: Connection error: dial tcp 104.21.52.103:8000: operation was canceled 2022/08/21 11:27:27 client: Give up 2022/08/21 11:28:18 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:28:27 client: Connection error: dial tcp 104.21.52.103:8000: operation was canceled 2022/08/21 11:28:27 client: Give up 2022/08/21 11:28:37 client: Connecting to ws://portainer.costxforbreakfast.com:8000 2022/08/21 11:28:47 client: Connection error: dial tcp 104.21.52.103:8000: operation was canceled 2022/08/21 11:28:47 client: Give up

nwrkbiz commented 1 year ago

Any updates on this issue?

amedveshchek commented 8 months ago

Found a solution in the #95 (here), TL;DR: add the env var AGENT_CLUSTER_ADDR=localhost:

docker run -d \
  -e AGENT_CLUSTER_ADDR=localhost \
  -p 9001:9001 \
  --name portainer_agent \
  --restart=always \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /var/lib/docker/volumes:/var/lib/docker/volumes \
  portainer/agent:2.19.4
jsorondo commented 1 month ago

Modifying the stack like this fixed both my issues(set your tag as needed):

version: '3.2'

services:
  agent:
    image: portainer/agent:latest
    ports:
      - "9001:9001"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/volumes:/var/lib/docker/volumes
    networks:
      - agent_network
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]

  portainer:
    image: portainer/portainer-ee:latest
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    ports:
      - "9443:9443"
      - "9000:9000"
      - "8000:8000"
    volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - agent_network
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

networks:
  agent_network:
    driver: overlay
    attachable: true

volumes:
  portainer_data: