Use Traefik for Edge Agent service

[robdyke]

Portainer communicates with the edge agent over port 8000 (as per Edge Agent Guide)

The example Traefik docker-compose does not expose :8000.

As Traefik can proxy any TCP traffic, let's use it?

[xe-nvdk]

Hi. No need to expose 8000. When you set up the Edge agent, you need to change the Portainer server URL to match the Edge entry point.

[robdyke]

Hummmm. So I set https://domain.tld in the config by the edge agent complained that it couldn't reach ws://endpoint on :8000 although it could reach the https:// on :443

[xe-nvdk]

When you add an edge agent, you need to change the Portainer server's URL to point edge.yourdomain.com. With the current configuration, with any request that came for that URL, Traefik will take and redirect to the port 800 in the container.

Take note that you need to specify one URL for portainer UI that works in port 9000 and another URL for edge.

Anyway, your proposal is valid, so, please, write in a new file to have both alternatives available.

Thank you again for your contribution is very appreciated. Ignacio

[robdyke]

Thanks and will do

On Thu, 19 Nov 2020, 11:49 Ignacio Van Droogenbroeck, < notifications@github.com> wrote:

When you add an edge agent, you need to change the Portainer server's URL to point edge.yourdomain.com. With the current configuration, with any request that came for that URL, Traefik will take and redirect to the port 800 in the container.

Take note that you need to specify one URL for portainer UI that works in port 9000 and another URL for edge.

Anyway, your proposal is valid, so, please, write in a new file to have both alternatives available.

Thank you again for your contribution is very appreciated. Ignacio

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/portainer/portainer-compose/issues/24#issuecomment-730321635, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHF5AFYSY6YK3SVAB3ZZT3SQUA5TANCNFSM4T2SRRAQ .

[baskinsy]

I probably have the same issue or missing something. I have deployed portainer with compose and I have a resolvable second URL (edge.mydomain.com) but edge agents are unable to connect with the following ERROR:

2020/12/22 15:04:25 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:30 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:35 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]
2020/12/22 15:04:40 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]

Edge URL is set when creating the edge endpoint as https://edge.mydomain.com and it is listening but the edge agent refuses to associate. My understanding was that proxying 8000 port with traefik on the separate URL is enough.

[robdyke]

Paste me the logs from Portainer && agent side @baskinsy

On Tue, 22 Dec 2020 at 15:11, baskinsy notifications@github.com wrote:

I probably have the same issue or missing something. I have deployed portainer with compose and I have a resolvable second URL ( edge.mydomain.com) but edge agents are unable to connect with the following ERROR:

2020/12/22 15:04:25 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] 2020/12/22 15:04:30 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] 2020/12/22 15:04:35 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed] 2020/12/22 15:04:40 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: short poll request failed]

Edge URL is set when creating the edge endpoint as https://edge.mydomain.com and it is listening but the edge agent refuses to associate. My understanding was that proxying 8000 port with traefik on the separate URL is enough.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/portainer/portainer-compose/issues/24#issuecomment-749589511, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHF5AAZW54P25TOJCL7TVDSWCZJFANCNFSM4T2SRRAQ .

[baskinsy]

The only log on edge side is the above, on portainer side logs are not displaying anything at all. I made a clean test again to verify that and the only log I can find is the above ERROR on egde agent side....

[baskinsy]

I think the tokens are broken somehow. I get "Incorrect padding" when i try to decode the token copied from portainer here https://www.base64code.com/decode

[robdyke]

@baskinsy I had those errors when using the provided docker-compose. I opened :8000 direct to portainer and the agent connected. Try the compose file that proxies agent through Traefik

[baskinsy]

Yes I read your proposal to proxy also 8000 but then is the communication secured? I would try it but that means the provided docker-compose does not work for edge agents as it seems.

[robdyke]

@baskinsy the ws:// on :8000 is (as I understand) encrypted

[baskinsy]

Found the issue and was confirmed by a staff member on slack. The KEY generation is broken in my case due to I'm using a four level domain for URL (edge.staging.mydomain.com).

https://github.com/portainer/portainer/issues/4647

[robdyke]

Glad yr sorted. Opened an issue upsteam? I often use four.level.doma.is

[baskinsy]

Yes seems the KEY cannot be decoded when a four level URL is used. When IP is used the decode works. I'll redeploy tomorrow with a third level URL on edge traefic vhost and report back if it works.

[hSinding]

@baskinsy Did you get it to work with a third level URL? When i'm trying to decode my KEY i'm getting Incorrect padding aswell, but with an IP it works.

xxxx.mydomain.com

[baskinsy]

@hSinding Yes I had success on decoding a KEY with a third level domain but I have still issues to connect the edge agent, it is correctly added and registered but cannot be browsed. Although the KEY issue can be circumvent with a third level domain, at least for me.

[reinoldus]

Same issue here, edge just won't connect:

2021/03/17 13:09:08 [ERROR] [internal,edge,poll] [message: an error occured during short poll] [error: Get https://edge1.domain.com/api/endpoints/5/status: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)]

But it seems to be a traefik misconfiguration because when I call the api endpoint manually I get 404

[charnesp]

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :

      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80

You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key. Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

[alackmann]

@charnesp this was such a handy tip. Many thanks. Portainer-peeps - it'd be fantastic to be able to generate an EDGE_KEY from the UI that supports this configuration. At very least an additional (optional) field for the edge URL would help.

[BerkeleyTrue]

@charnesp Thanks for the help.

I'd like to add I had to add an extra step. With the new key, it complained about a deprecated MD5 fingerprint so I had to update it with a SHA256 fingerprint. No idea why.

This really needs to be an option in the UI, or at least update the docs so it's not point edge as a separate url that doesn't work without the 8000 port open.

[mfruhner]

Hello, I am currently trying to achieve the same: Manage an Edge Agent in a VM via Portainer behind Traefik running in WSL.

As i described here, I try to browse an Edge Agent after successful association. The request to open the tunnel connection however goes to 127.0.0.1 and the connection fails.. Any advice?

[SAOPP]

I'm had made it before like was described at this post https://github.com/portainer/portainer-compose/issues/24#issuecomment-942389178

But agent communication is possible to solve with dns:port and entrypoint also, just needed added your port of edge into traefik ports:

...
      - 8000:8000
...

And entrypoints:

...
      - --entrypoints.edge.address=:8000   
...   

And add second route for this service at portainer's side:

...
    labels:
      - traefik.enable=true
      #portainer route
      - traefik.http.services.portainer.loadbalancer.server.port=9000
      - traefik.http.routers.portainer.rule=Host(`portainer.domain.ltd`)
      - traefik.http.routers.portainer.entrypoints=websecure
      - traefik.http.routers.portainer.service=portainer
      - traefik.http.routers.portainer.tls=true
      - traefik.http.routers.portainer.tls.certresolver=yourresolver
      #edge route
      - traefik.http.services.edge.loadbalancer.server.port=8000
      - traefik.http.routers.edge.rule=Host(`portainer.domain.ltd`)
      - traefik.http.routers.edge.entrypoints=edge
      - traefik.http.routers.edge.service=edge
      - traefik.http.routers.edge.tls=true
      - traefik.http.routers.edge.tls.certresolver=yourresolver
...

So, that expected, we are have portainer at portainer.domain.ltd and edge at portainer.domain.ltd:8000

Now, to access your edge, you just need to add the port to the dns name.

[tarmacx]

In my case had to disable the tls and certresolver to make it work. Are the data still encrypted by usinf ws instead of wss and disabling tls ?

[agoodshort]

Thanks @SAOPP your steps here helped me, but same as @tarmacx I had to disable tls as I was facing the below error:

2022/10/31 19:53:49 client: Connecting to ws://portainer.mydomain.com:8000
2022/10/31 19:53:49 client: Connection error: websocket: bad handshake
2022/10/31 19:53:49 client: Give up

---

Now, to access your edge, you just need to add the port to the dns name.

I think I didn't understand this part. Where do you add the port?

---

By the way, it seems that we are getting really close to the correct config template. We should edit the template on the official portainer docs

[SAOPP]

I think I didn't understand this part. Where do you add the port?

Hi! I mean portainer.mydomain.com:8000 port ova here.

[Jigsaw5279]

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :

      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80

You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key. Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

I tried setting things up like this but I'm getting bad handshake on the agent machine. I can't see what I did wrong there, any hints?

[charnesp]

Hi,

Maybe an incorrectly generated key.

You can use the following script for generating a corrected key :

#!/usr/bin/python3
from base64 import urlsafe_b64encode, urlsafe_b64decode
import argparse

INITIAL_EDGE_URL = "portainer.mydomain.com:8000"
FINAL_EDGE_URL = "https://edge.mydomain.com"

def replace_edge_address_edge_key(edge_key, initial_edge_url = INITIAL_EDGE_URL, final_edge_url = FINAL_EDGE_URL ):
    base64_bytes = edge_key.encode('utf-8')

    message_bytes = urlsafe_b64decode(base64_bytes + b'=' * (-len(base64_bytes) % 4))
    decoded_initial_key = message_bytes.decode('utf-8')
    decoded_corrected_key = decoded_initial_key.replace(initial_edge_url, final_edge_url)
    base64_bytes = decoded_corrected_key.encode('utf-8')
    message_bytes = urlsafe_b64encode(base64_bytes)
    corrected_key = message_bytes.decode('utf-8').replace('=',"")

    return corrected_key, decoded_initial_key

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('edge_key', action='store', type=str,
                        help='Initial encoded edge key')

    parser.add_argument('--initial_edge_url', action='store', type=str,
                         default=INITIAL_EDGE_URL,
                         help='Initial edge url (default: %s)' % INITIAL_EDGE_URL)

    parser.add_argument('--final_edge_url', action='store', type=str,
                         default=FINAL_EDGE_URL,
                         help='Final edge url (default: %s)' % FINAL_EDGE_URL)

    args = parser.parse_args()

    edge_key=args.edge_key
    initial_edge_url=args.initial_edge_url
    final_edge_url=args.final_edge_url

    corrected_key, decoded_initial_key = replace_edge_address_edge_key(edge_key,initial_edge_url = initial_edge_url, final_edge_url = final_edge_url )
    print("Initial decoded key: {} \nCorrected encoded key: {}".format(decoded_initial_key, corrected_key))

if __name__ == '__main__':
    main()

[Jigsaw5279]

Nope, that didn't do the trick

[charnesp]

Did you disabled TLS for your environment, or setup SSL certificate for Portainer? As Portainer is behind Traefik, which handles the HTTPS connections, you should use plain HTTP protocol.

[Jigsaw5279]

I've found the issue.. I had mistyped the service name in the traefik label, so the request wouldn't ever reach portainer. "Bad handshake" is a really misleading error though

[SAOPP]

@Jigsaw5279 Please share the details for those who are likely to experience this in the future.

[TimoVerbrugghe]

@Jigsaw5279 / @SAOPP / @tarmacx anybody that was able to solve the bad handshake issue?

2023/01/10 10:07:01 client: Connecting to ws://portainer.mydomain.com:8000
2023/01/10 10:07:01 client: Connection error: websocket: bad handshake
2023/01/10 10:07:01 client: Give up

I followed the steps from @SAOPP:

• Exposed port 8000 on my traefik container & port forwarded from my router
• Added edge as additional entrypoint
• Added an additional route to my portainer docker container (doublechecked the service labels)

Traefik uses letsencrypt certs. I added the edge agent with these settings:

It does work when I remove the tls and certresolver labels without changing any other labels, so I'm quite sure it's not a mistyping of service names in the traefik labels...

EDIT: Tried both with force https on/off, no changes. I have to remove the tls and certresolver labels to make this work...

[radarsu]

For long time I had similar problem, but error was a bit different (something with ws connection give up). Then I base64 decoded EDGE_KEY and replaced second parameter (edge server address) with portainer.example.com:443, because I have found no other way to change default 8000 port which I didn't want to use.

Error has changed to bad handshake and then commenting out labels fixed the issue. Although that means certificates won't be automatically renewed at least it works. So I'm posting full configuration (in TypeScript instead of yaml) as it might help someone.

Portainer + Portainer Edge

services[portainerConfig.serviceName] = {
    container_name: portainerConfig.serviceName,
    image: `portainer/portainer-ee:2.16.2-alpine`,

    command: `-H unix:///var/run/docker.sock`,
    labels: [
        `traefik.enable=true`,

        // portainer-http
        `traefik.http.routers.${portainerConfig.serviceName}-http.entrypoints=web`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.rule=Host(\`${portainerConfig.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.middlewares=${traefikConfig.serviceName}-redirect-to-https`,
        `traefik.http.routers.${portainerConfig.serviceName}-http.service=${portainerConfig.serviceName}`,

        // portainer
        `traefik.http.routers.${portainerConfig.serviceName}.entrypoints=websecure`,
        `traefik.http.routers.${portainerConfig.serviceName}.rule=Host(\`${portainerConfig.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}.service=${portainerConfig.serviceName}`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls.certresolver=acmeresolver`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls.domains[0].main=${portainerConfig.domain}`,
        `traefik.http.routers.${portainerConfig.serviceName}.tls=true`,
        `traefik.http.services.${portainerConfig.serviceName}.loadbalancer.server.port=9000`,

        // portainer-edge-http
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.entrypoints=web`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.rule=Host(\`${portainerConfig.edge.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.middlewares=${traefikConfig.serviceName}-redirect-to-https`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge-http.service=${portainerConfig.serviceName}-edge`,

        // portainer-edge
        `traefik.http.routers.${portainerConfig.serviceName}-edge.entrypoints=websecure`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge.rule=Host(\`${portainerConfig.edge.domain}\`)`,
        `traefik.http.routers.${portainerConfig.serviceName}-edge.service=${portainerConfig.serviceName}-edge`,
        // Commenting those out fixed bad handshake issue.
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls.certresolver=acmeresolver`,
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls.domains[0].main=${portainerConfig.edge.domain}`,
        // `traefik.http.routers.${portainerConfig.serviceName}-edge.tls=true`,
        `traefik.http.services.${portainerConfig.serviceName}-edge.loadbalancer.server.port=8000`,
    ],
    networks: [`traefik-network`],
    restart: `unless-stopped`,
    volumes: [`/var/run/docker.sock:/var/run/docker.sock`, `${sharedConfig.docker.volumes.root}/${portainerConfig.serviceName}/data:/data`],
};

Portainer Edge Agent (different machine on the Internet)

services[portainerAgentConfig.serviceName] = {
    container_name: portainerAgentConfig.serviceName,
    image: `portainer/agent:2.16.2-alpine`,

    environment: {
        EDGE: `1`,
        EDGE_ID: `<ID>`,
        EDGE_INSECURE_POLL: `0`,
        EDGE_KEY: `<BASE64_DECODED_MODIFIED_AND_ENCODED_BACK_MANUALLY>`
    },
    labels: [`traefik.enable=false`],
    networks: [`traefik-network`],
    restart: `unless-stopped`,
    volumes: [
        `/var/run/docker.sock:/var/run/docker.sock`,
        `/var/lib/docker/volumes:/var/lib/docker/volumes`,
        `/:/host`,
        `${sharedConfig.docker.volumes.root}/${portainerAgentConfig.serviceName}/data:/data`,
    ],
};

[dmr138]

With a small hack I was able to make it work through Traefik :

Portainer is setup in Traefik with :

      # Frontend
      - "traefik.enable=true"
      - "traefik.http.routers.frontend-portainer.rule=Host(`portainer.mydomain.com`)"
      - "traefik.http.services.frontend-portainer.loadbalancer.server.port=9000"
      - "traefik.http.routers.frontend-portainer.service=frontend-portainer"
      - "traefik.http.routers.frontend-portainer.tls.certresolver=lets-encrypt"

      # Edge
      - traefik.http.routers.edge-portainer.rule=Host(`edge.mydomain.com`)
      - traefik.http.services.edge-portainer.loadbalancer.server.port=8000
      - traefik.http.routers.edge-portainer.service=edge-portainer
      - traefik.http.routers.edge-portainer.tls.certresolver=lets-encrypt
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.regex=^http://(.*)
      - traefik.http.middlewares.edge-portainer-redirect.redirectregex.replacement=https://$${1}
      - traefik.http.routers.http.middlewares=edge-portainer-redirect
      - traefik.http.services.http.loadbalancer.server.port=80

You first need to decode the EDGE_KEY (using https://www.base64decode.org for example) in order to obtain the fingerprint and the endpoint ID.

You should obtain something like: https://edge.mydomain.com|edge.mydomain.com:8000|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

3 being the endpoint ID, aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00 being the server fingerprint.

I modified it like that : https://portainer.mydomain.com|https://edge.mydomain.com|aa:bb:cc:dd:ee:ff:00:00:00:01:00:00:00:00:00:00|3

and used https://www.base64encode.org (with URL-safe encoding enabled) to regenerate the key. Then simply use this generated EDGE_KEY on the agent and it should work without exposing any 8000port

Thank you much this worked perfectly!