Portainer On swarm with agent, agent_secret doesn't fix authentication issue

[psyciknz]

services:
  agent:
    image: portainer/agent
    environment:
      AGENT_CLUSTER_ADDR: tasks.agent
      AGENT_PORT: 9001
      #AGENT_SECRET: mysecrettoken
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /hdd/zdocker/volumes:/var/lib/docker/volumes
    networks:
      - agent-network
    ports:
      - target: 9001
        published: 9001
        protocol: tcp
        mode: host
    deploy:
      mode: global
      placement:
        constraints: [node.platform.os == linux]
  portainer:
    image: portainer/portainer:latest
    command: -H tcp://tasks.agent:9001 --tlsskipverify
    volumes:
      - portainer-data:/data
    ports:
      - "9000:9000"
    networks:
      - agent-network
    environment:
          TZ: Pacific/Auckland
          SERVICE_PORTS: "9000"
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints: [node.role == manager]

Trying to get portainer to work in a swarm. I had it working, before I added agent_secret - which was added in efforts to fix an authentication issue when the swarm manager died and the portainer gui transferred to a 2nd manager (ref: https://portainer.readthedocs.io/en/stable/agent.html#shared-secret)

But now, upon starting the service, the ui asks for initial credentials, and then shows the primary, until you click on it, and then it reports down.

I see in the logs for the agent where the ui is running that it is reporting a 403 error.

[deviantony]

@psyciknz I am not sure that this is an agent related issue. If you do not share the Portainer data across the Swarm cluster (through a shared filesystem or something else) then each time Portainer will restart it will start with an empty database...

If you don't have any solution to share volume data across nodes in your Swarm, then I would recommend pinning the Portainer container onto a specific node inside the cluster via a constraint.

[psyciknz]

But with the agent_secret, even if it created a new DB each time.....shouldn't it actually start? I got the impression from reading the docs that without agent_secret each time it moved it wouldn't allow signon

[deviantony]

What do you mean by "it wouldn't allow signon" ?

[psyciknz]

gets a 403 error in the agent logs

[deviantony]

Please share the Portainer logs as well as the agent logs with us so that we can get more insight on this.

[psyciknz]

I'll have to start again as I was working out how to use swarm at the same time and probably did something wrong.

[psyciknz]

Agent logs ( on swarm manager)

default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:47:56 [INFO] [main] [message: Agent running on a Swarm cluster node. Running in cluster mode]
default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:47:59 [INFO] serf: EventMemberJoin: pi01-bb3cb887e25f 10.0.1.14
default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:48:13 [INFO] [http] [server_addr: 0.0.0.0] [server_port: 9001] [secured: true] [api_version: 1.5.0] [message: Starting Agent API server]
default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:48:25 http error: Missing request signature headers (err=Unauthorized) (code=403)
default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:48:25 http error: Invalid request signature (err=Unauthorized) (code=403)
default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:48:25 http error: Invalid request signature (err=Unauthorized) (code=403)
default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:50:21 [INFO] serf: EventMemberJoin: pi03-3de470bb36df 10.0.1.20
default_agent.0.al34304tibyt@pi01    | 2019/10/15 20:50:58 http error: Invalid request signature (err=Unauthorized) (code=403)
default_agent.0.nftrpt94r210@pi03    | 2019/10/15 20:50:18 [INFO] [main] [message: Agent running on a Swarm cluster node. Running in cluster mode]
default_agent.0.nftrpt94r210@pi03    | 2019/10/15 20:50:21 [INFO] serf: EventMemberJoin: pi03-3de470bb36df 10.0.1.20
default_agent.0.nftrpt94r210@pi03    | 2019/10/15 20:50:21 [INFO] serf: EventMemberJoin: pi01-bb3cb887e25f 10.0.1.14
default_agent.0.nftrpt94r210@pi03    | 2019/10/15 20:50:38 [INFO] [http] [server_addr: 0.0.0.0] [server_port: 9001] [secured: true] [api_version: 1.5.0] [message: Starting Agent API server]
default_agent.0.nftrpt94r210@pi03    | 2019/10/15 20:52:07 http error: Invalid request signature (err=Unauthorized) (code=403)
default_agent.0.nftrpt94r210@pi03    | 2019/10/15 20:52:07 http error: Invalid request signature (err=Unauthorized) (code=403)

Portainer service (I was slow to initially connect, this is why it complained of lack of admin user)

p01:/hdd/docker-data/default $ docker service logs -f default_portainer
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:48:25 Templates already registered inside the database. Skipping template import.
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:48:25 http error: endpoint snapshot error (endpoint=primary, URL=tcp://tasks.agent:9001) (err=Error response from daemon: Invalid request signature)
default_portainer.1.cqz8v6nw3cvh@pi01    | 2019/10/15 20:47:59 Get https://tasks.agent:9001/_ping: dial tcp 10.0.1.14:9001: connect: connection refused
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:48:26 Starting Portainer 1.22.1 on :9000
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:48:26 [DEBUG] [chisel, monitoring] [check_interval_seconds: 10.000000] [message: starting tunnel management process]
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:48:26 server: Reverse tunnelling enabled
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:48:26 server: Fingerprint 7a:97:70:e4:22:2a:8c:60:ff:20:98:79:bc:59:db:ec
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:48:26 server: Listening on 0.0.0.0:8000...
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:50:47 http error: No administrator account found inside the database (err=Object not found inside the database) (code=404)
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:50:47 http error: No administrator account found inside the database (err=Object not found inside the database) (code=404)
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:52:07 background schedule error (endpoint snapshot). Unable to create snapshot (endpoint=primary, URL=tcp://tasks.agent:9001) (err=Error response from daemon: Invalid request signature)
default_portainer.1.hdrtnfr79ctr@pi01    | 2019/10/15 20:53:25 background schedule error (endpoint snapshot). Unable to create snapshot (endpoint=primary, URL=tcp://tasks.agent:9001) (err=Error response from daemon: Invalid request signature)
default_portainer.1.i9l9esahg2wz@pi01    | 2019/10/15 20:48:11 Templates already registered inside the database. Skipping template import.
default_portainer.1.i9l9esahg2wz@pi01    | 2019/10/15 20:48:11 Get https://tasks.agent:9001/_ping: dial tcp 10.0.1.14:9001: connect: connection refused

This is when the stack created with the AGENT_SECRET set

[jhonny-oliveira]

Assuming AGENT_SECRET can have any random value, for instance the instructions default ("mysecrettoken"), then I have the same issue.

[LostOnTheLine]

I never have it ask me for the Agent_Secret when I try to connect to the Agent environment I click Connect & then I just get

Failure
Get "https://192.168.3.17:9001/ping": dial tcp 192.168.3.17:9001: connect: connection refused

With no place to ever enter the Secret

If I have the secret line in both compose files, for the one managing & the one managed it won't connect. Does it need to be in a different place in the managed/manager?