Teleport web UI and Application Access running behind reverse proxy

stefangrosaru commented 2 years ago

What

What would you like Teleport to do differently? Allow the teleport web UI and its features to work behind a reverse proxy such as traefik, nginx, etc.

How

How would you implement this? As an example we have teleport.example.com which is the url for the web ui and *.teleport.example.com which is used for application access. We would then have these two names configured in the reverse proxy, for example nginx, so that they pass the traffic to teleport.

Why

Why do you need this? The application access service of teleport is very useful, especially when using unsecure apps, but it would be nice to have the option to decide which apps should be put behind that layer of security and which ones should not.

Workaround

There is no current workaround as far as I am aware.

alexbarcelo commented 2 years ago

I will leave this here, in case is correct (which I'm not 100% sure) and becomes useful for somebody:

Workaround

Configure your reverse proxy to perform SSL termination. Configure your certificates (I personally use a wildcard domain *.tele.my.domain plus the UI tele.my.domain).

Remove ACME configuration from the Teleport yaml. Add both certificates in the https_keypairs (in proxy_service).

I may be mistaken because I was testing things and DNS may be propagating and being naughty in general. But I believe that it is working with this configuration.

It seems a bit silly, as it is unwrapping and rewrapping the SSL with the same certificate; but with this configuration the LAN connections can be performed directly (without going through the reverse proxy) and everything seems to work. The ALPN is not performed because ALPN doesn't seem to behave nicely with reverse proxies, AFAICT. That's why a wildcard seems to be needed.

Hopefully someone will correct me if I'm wrong.

stefangrosaru commented 2 years ago

@alexbarcelo Thanks for taking the time to help with this. I have had a few attempts to achieve something similar, however I have encountered issues with some of the teleport functionality, namely application access and the ability for server nodes to connect to the cluster in "reverse tunnel" mode.

alexbarcelo commented 2 years ago

DISCLAIMER: I believe that my solution works for web-based access, but it seems to fail for the tsh login flow.

Maybe there is some client certificate shenanigans? I don't really know the answer, but keep that in mind if you try my workaround.

Application access seems to work if you connect directly to the node (instead of going through the reverse proxy). I suppose that the same mechanism is failing, whatever it is.

stefangrosaru commented 2 years ago

Yes, I wonder if you can get around this by creating another teleport service that deals with application access, but still not sure how to get the auth service to work.

Maybe doing L4 load balancing instead of L7 could also help, the only issue with that is that you lose the ability to do hostname based routing.

It seems that some teleport services are incompatible with each other, but I was not able to find any info on this in the documentation.

toby1991 commented 2 years ago

After 3 years, this problem still exist......

KasperSkytte commented 2 years ago

I have trouble with adding applications too. I use nginx as reverse-proxy for the main domain (like most self-hosters nowadays?), and then teleport on a subdomain teleport.domain.com. When I set my email in the ACME setting in the server teleport.yml I get:

2022-03-16T21:43:17Z WARN [ALPN:PROX] Failed to handle client connection. error:[
ERROR REPORT:
Original Error: *errors.errorString acme/autocert: missing server name
Stack Trace:
    /go/src/github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:344 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).handleConn
    /go/src/github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:282 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).Serve.func1
    /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: acme/autocert: missing server name] alpnproxy/proxy.go:290

When I set http_keypairs to link to both the domain.com and teleport.domain.com certificates and private keys, I'm not allowed to connect through the browser because of HSTS, invalid certificate. The ladder case also happens when I leave either of acme or http_keypairs settings blank. I have the required ports forwared on the router.

The app I'm starting also shows the following line in the output:

2022-03-17T09:28:01+01:00 INFO [APP:SERVI] Failed to ping web proxy "teleport.domain.com:3024" addr: Get "https://teleport.domain.com:3024/webapi/find": tls: first record does not look like a TLS handshake leaseID:1 target:teleport.domain.com:3024 reversetunnel/agent.go:269

but a simple nmap teleport.domain.com -p 3024 shows the port is open. I have set *.teleport.domain.com and teleport.domain.com DNS CNAME records.

What am I doing wrong? The browser doesn't get the correct certificate presented, but why?

webvictim commented 2 years ago

@KasperSkytte That error (Failed to ping web proxy "teleport.domain.com:3024") isn't really an error, it's just some debugging output that should be silenced. I'm going to raise an issue to get this fixed.

With regard to your other issue, did you get it resolved? If not, can you please share your full /etc/teleport.yaml config file (with any secrets/tokens redacted)?

In general you don't need to run Teleport behind nginx or anything else, it expects to be able to terminate its own TLS (which is why it has the ACME support to get certificates automatically from LetsEncrypt)

KasperSkytte commented 2 years ago

@webvictim Sorry I totally forgot about this. The trouble was caused by both contradicting certificates between teleport and the reverse proxy, and that I forgot to add *.teleport.domain.com in addition to teleport.domain.com to the reverse-proxy config. So pretty basic mistakes :)

KasperSkytte commented 2 years ago

So both the https_keypairs and acme settings under proxy_service in teleport.yaml are blank. I let the reverse proxy handle things and it works like a charm.

abjoseph commented 2 years ago

So both the https_keypairs and acme settings under proxy_service in teleport.yaml are blank. I let the reverse proxy handle things and it works like a charm.

@KasperSkytte Could you share your /etc/teleport.yaml? If your https_keypairs field is empty, how are you configuring teleport to use your proxied domain instead of its default domains --> teleport.cluster.local and *.teleport.cluster.local

abjoseph commented 2 years ago

@webvictim Is there a way to configure teleport to use a proxied domain instead of its default domains --> teleport.cluster.local and *.teleport.cluster.local when behind a reverse proxy like nginx? Does one always have to configure the certificates within the https_keypairs field?

webvictim commented 2 years ago

@abjoseph teleport.cluster.local is irrelevant here - that's just an internal SNI name that Teleport uses to refer to its own internal connections to the Teleport auth service.

Here's a working config that I use for putting Teleport behind Caddy:

/etc/teleport.yaml:

version: v1
teleport:
  log:
    output: stderr
    severity: DEBUG
auth_service:
  enabled: yes
  cluster_name: teleport.example.com
  authentication:
    type: github
    second_factor: webauthn
    webauthn:
      rp_id: teleport.example.com
ssh_service:
  enabled: yes
  labels:
    type: node
proxy_service:
  enabled: yes
  web_listen_addr: 0.0.0.0:3080
  # public address which caddy hosts in front of Teleport
  public_addr: teleport.example.com:443
  ssh_public_addr: teleport.example.com:3023
  tunnel_public_addr: teleport.example.com:3024
  kube_public_addr: teleport.example.com:3026

/etc/caddy/Caddyfile.d/teleport.conf:

teleport.example.com, *.teleport.example.com {
    tls {
        dns cloudflare {env.CLOUDFLARE_AUTH_TOKEN}
    }
    reverse_proxy https://localhost:3080 {
                transport http {
                        tls
                        tls_insecure_skip_verify
                }
        }
}

How it works:

The DNS for teleport.example.com and *.teleport.example.com points to the public IP of the server
Caddy listens externally for HTTPS traffic on port 443, and automatically fetches ACME certs for teleport.example.com and *.teleport.example.com using a DNS-01 challenge (Cloudflare is my DNS provider)
Caddy forwards any requests for either of these hosts to https://localhost:3080, which is the port for the web UI on Teleport's proxy service.
As there are no https_keypairs configured, Teleport generates a self-signed certificate and serves the web UI using this certificate.
- Caddy ignores the self-signed cert, due to the tls_insecure_skip_verify in its config.
- It's technically possible to remove this by issuing Teleport a certificate from your own self-signed CA and configuring the reverse proxy with the CA's public key - I just haven't bothered as this isn't a production setup and it doesn't bother me.
Any requests for Teleport's SSH proxy (port 3023), SSH tunnel (port 3024) and Kubernetes proxy (port 3026) are sent to those ports, which are opened to the outside world to allow direct connections to each of those Teleport services outside of Caddy.
I should note as well that pretty much everything works perfectly like this. tsh login is fine, application access via browser is fine, reverse tunnels are fine, etc. There are two things which don't work with this setup:
- command-line application access via Teleport (with tsh app login). This doesn't work because the reverse proxy doesn't pass client certificates through to Teleport, and I don't currently know of any workarounds.
- multiplexing database traffic on port 443 (which is the default for PostgreSQL, MongoDB and maybe others when you use them with Teleport). You can work around this by setting separate listeners and public addresses in Teleport's config file if needed:
```
proxy_service:
postgres_listen_addr: 0.0.0.0:5432
postgres_public_addr: teleport.example.com:5432
```

Other reverse proxies like nginx or Apache should work similarly - it's just a question of splitting the traffic such that you forward HTTPS through the reverse proxy and all other traffic goes direct to Teleport. This requires that you use version: v1 in the Teleport config file, and remove proxy_listener_mode: multiplex from the Teleport config so it listens on separate ports. It isn't possible to terminate TLS in front of Teleport with a reverse proxy when using multiplexing mode (which is the default in Teleport 8+)

abjoseph commented 2 years ago

@webvictim Thank you for the thorough reply. Please see below my complete setup which is still not working when trying to register a node.

teleport config --> `/etc/teleport.yaml`

version: v1
teleport:
  nodename: teleport
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: []
  diag_addr: ""
auth_service:
  enabled: "yes"
  #listen_addr: 0.0.0.0:3025
  cluster_name: teleport.example.com
  #authentication:
    #type: github
    #second_factor: webauthn
    #webauthn:
      #rp_id: teleport.example.com
  #public_addr: teleport.example.com:3025
  #proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
  labels:
    env: lan-net
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  #listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:3080
  public_addr: teleport.example.com:443
  ssh_public_addr: teleport.example.com:3023
  tunnel_public_addr: teleport.example.com:3024
  #tunnel_listen_addr: 0.0.0.0:3024

nginx config --> `teleport.example.conf`

server {
    listen                   443 ssl http2;

    server_name              teleport.example.com *.teleport.example.com;

    ssl_certificate          /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key      /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_trusted_certificate  /etc/letsencrypt/live/example.com/fullchain.pem;

    include                  /etc/nginx/snippets/ssl.conf;

    location / {

        proxy_buffering      off;
        proxy_set_header     Host $host;
        proxy_set_header     X-Real-IP $remote_addr;
        proxy_set_header     X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header     X-Forwarded-Proto "https";

        # WebSocket support
        proxy_http_version   1.1;
        proxy_set_header     Upgrade $http_upgrade;
        proxy_set_header     Connection "upgrade";
        proxy_read_timeout   86400;

        proxy_pass           https://teleport.localdomain.net:3080;
    }
}

Additional context:

The following ports are been port-forwarded directly to the teleport server.

- 3023
- 3024
- 3025

After adding a new user with tctl users add teleport-admin --roles=editor,access --logins=root,ubuntu I'm able to login to the webui at https://teleport.example.com.

In the webui, I click the "Add Server" button in the top right corner and copy the command from the "Manually" tab. e.g teleport start --roles=node --token=9bed84d4c9ad70210f528962b14881cf --auth-server=teleport.example.com:443

When I go to the linux host (protected-resource) that I want to add as a node in teleport and run the above command, I get the below error:

root@rport:~# sudo teleport start \
> --roles=node \
> --auth-server=teleport.example.com:443  \
> --token=b289190e4c8c68f54ba5610a65578242 \
> --labels=env=lab-net

INFO             Generating new host UUID: e7091a70-29fd-421b-9259-7475351f4d9b. service/service.go:689
INFO [PROC:1]    Joining the cluster with a secure token. service/connect.go:395
INFO [PROC:1]    Node has obtained credentials to connect to the cluster. service/connect.go:429
ERRO [PROC:1]    "Node failed to establish connection to cluster: Failed to connect to Auth Server directly or over tunnel, no methods remaining.\n\tGet \"https://teleport.cluster.local/v2/domain\": x509: certificate is valid for *.example.com, not 74656c65706f72742e6d616368696e656a61722e636f6d.teleport.cluster.local, Get \"https://teleport.cluster.local/v2/domain\": ssh: handshake failed: EOF." service/connect.go:86

Below is the debug version of the above command:

root@rport:~# teleport start --debug --roles=node --token=b289190e4c8c68f54ba5610a65578242 --auth-server=teleport.example.com:443
[SQLITE]    DEBU Connected to: file:%2Fvar%2Flib%2Fteleport%2Fproc%2Fsqlite.db?_busy_timeout=10000&_sync=FULL&_txlock=immediate, poll stream period: 1s lite/lite.go:225
[SQLITE]    DEBU journal_mode=delete, synchronous=2, busy_timeout=10000 lite/lite.go:272
[KEYGEN]    DEBU SSH cert authority started with no keys pre-compute. native/native.go:106
DEBU [PROC:1]    Adding service to supervisor. service:register.node service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:ssh.node service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:ssh.shutdown service/supervisor.go:201
DEBU [PROC:1]    Adding service to supervisor. service:common.rotate service/supervisor.go:201
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:262
DEBU [PROC:1]    Service has started. service:register.node service/supervisor.go:262
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:883
DEBU [PROC:1]    Service has started. service:ssh.node service/supervisor.go:262
DEBU [PROC:1]    Service has started. service:ssh.shutdown service/supervisor.go:262
INFO [PROC:1]    Joining the cluster with a secure token. service/connect.go:395
DEBU [PROC:1]    Generating new key pair for Node first-time-connect. service/connect.go:302
[AUTH]      DEBU Registering node to the cluster. auth-servers:[{teleport.example.com:443 tcp }] auth/register.go:153
[AUTH]      DEBU The first specified auth server appears to be a proxy. auth/register.go:167
[AUTH]      INFO Attempting registration via proxy server. auth/register.go:173
[CLIENT]    DEBU HTTPS client init(proxyAddr=teleport.example.com:443, insecure=false) client/weblogin.go:228
[CLIENT]    DEBU Attempting https://teleport.example.com:443/v1/webapi/host/credentials client/https_client.go:92
[AUTH]      INFO Successfully registered via proxy server. auth/register.go:180
DEBU [PROC:1]    Deleted generated key pair Node first-time-connect. service/connect.go:288
INFO [PROC:1]    Node has obtained credentials to connect to the cluster. service/connect.go:429
DEBU [PROC:1]    Attempting to connect to Auth Server directly. auth-addrs:[teleport.example.com:443] service/connect.go:863
DEBU [PROC:1]    Failed to connect to Auth Server directly. auth-addrs:[teleport.example.com:443] service/connect.go:870
DEBU [PROC:1]    Attempting to discover reverse tunnel address. auth-addrs:[teleport.example.com:443] service/connect.go:879
DEBU [PROC:1]    Attempting to connect to Auth Server through tunnel. auth-addrs:[teleport.example.com:443] service/connect.go:881
            DEBU Attempting GET teleport.example.com:443/webapi/find webclient/webclient.go:112
            DEBU Attempting GET teleport.example.com:3024/webapi/find webclient/webclient.go:112
            DEBU Attempting GET teleport.example.com:3024/webapi/find webclient/webclient.go:112
DEBU [PROC:1]    Failed to ping web proxy "teleport.example.com:3024" addr: Get "https://teleport.example.com:3024/webapi/find": tls: first record does not look like a TLS handshake reversetunnel/transport.go:100
[HTTP:PROX] DEBU No proxy set in environment, returning direct dialer. proxy/proxy.go:269
DEBU [PROC:1]    Failed to connect to Auth Server directly. auth-addrs:[teleport.example.com:443] error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError Get &#34;https://teleport.cluster.local/v2/domain&#34;: x509: certificate is valid for *.example.com, not 74656c65706f72742e6d616368696e656a61722e636f6d.teleport.cluster.local
Stack Trace:
        /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:133 github.com/gravitational/teleport/lib/httplib.ConvertResponse
        /go/src/github.com/gravitational/teleport/lib/auth/clt.go:288 github.com/gravitational/teleport/lib/auth.(*Client).Get
        /go/src/github.com/gravitational/teleport/lib/auth/clt.go:379 github.com/gravitational/teleport/lib/auth.(*Client).GetDomainName
        /go/src/github.com/gravitational/teleport/lib/auth/clt.go:1538 github.com/gravitational/teleport/lib/auth.(*Client).GetLocalClusterName
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:968 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClientDirect
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:865 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:437 github.com/gravitational/teleport/lib/service.(*TeleportProcess).firstTimeConnect
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:143 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:125 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService
        /go/src/github.com/gravitational/teleport/lib/service/connect.go:64 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService
        /go/src/github.com/gravitational/teleport/lib/service/service.go:2051 github.com/gravitational/teleport/lib/service.(*TeleportProcess).registerWithAuthServer.func1
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:487 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
        /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:263 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
        /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: Get &#34;https://teleport.cluster.local/v2/domain&#34;: x509: certificate is valid for *.example.com, not 74656c65706f72742e6d616368696e656a61722e636f6d.teleport.cluster.local] service/connect.go:888
DEBU [PROC:1]    Failed to connect to Auth Server through tunnel. auth-addrs:[teleport.example.com:443] error:[Get "https://teleport.cluster.local/v2/domain": ssh: handshake failed: EOF] service/connect.go:889
ERRO [PROC:1]    "Node failed to establish connection to cluster: Failed to connect to Auth Server directly or over tunnel, no methods remaining.\n\tGet \"https://teleport.cluster.local/v2/domain\": x509: certificate is valid for *.example.com, not 74656c65706f72742e6d616368696e656a61722e636f6d.teleport.cluster.local, Get \"https://teleport.cluster.local/v2/domain\": ssh: handshake failed: EOF." service/connect.go:86
DEBU [PROC:1]    Failed to ping web proxy "teleport.example.com:3024" addr: Get "https://teleport.example.com:3024/webapi/find": context canceled reversetunnel/transport.go:100
[HTTP:PROX] DEBU No proxy set in environment, returning direct dialer. proxy/proxy.go:269

webvictim commented 2 years ago

What output do you get when you run telnet teleport.example.com 3023 and telnet teleport.example.com 3024? The error here is that the SSH handshake is failing, so it seems maybe the traffic can't make it through on those ports.

abjoseph commented 2 years ago

Both ports are open and accessible as shown below using both telnet and netcat.

I know you stated that teleport.cluster.local is irrelevant here but why is it referring to that fqdn when erroring out here? and why is it trying to validate the cert for that specific fqdn as shown in the output above x509: certificate is valid for *.example.com, not 74656c65706f72742e6d616368696e656a61722e636f6d.teleport.cluster.local ?

The impression I'm getting is that it's expecting to see *.teleport.cluster.local within the SAN of the certificate on my reverse proxy; I'm assuming that's due to the internal cert of teleport been autogenerated versus configuring a custom cert. If this is not the case then the error message is really unhelpful with regards to diagnosing the true issue.

ubuntu@oracle-cloudhost-01:~$ telnet teleport.example.com 3023
Trying 69.143.XX.XX...
Connected to teleport.example.com.
Escape character is '^]'.
SSH-2.0-Teleport

ubuntu@oracle-cloudhost-01:~$ telnet teleport.example.com 3024
Trying 69.143.XX.XX...
Connected to teleport.example.com.
Escape character is '^]'.
SSH-2.0-Teleport

ubuntu@oracle-cloudhost-01:~$ nc -zv teleport.example.com 3023
Connection to teleport.example.com 3023 port [tcp/*] succeeded!

ubuntu@oracle-cloudhost-01:~$ nc -zv teleport.example.com 3024
Connection to teleport.example.com 3024 port [tcp/*] succeeded!

webvictim commented 2 years ago

The teleport.cluster.local thing is a red herring; see https://github.com/gravitational/teleport/issues/11289 and other linked issues.

The HTTPS part of the connection works fine and the node joins the cluster successfully:

[AUTH]      DEBU Registering node to the cluster. auth-servers:[{teleport.example.com:443 tcp }] auth/register.go:153
[AUTH]      DEBU The first specified auth server appears to be a proxy. auth/register.go:167
[AUTH]      INFO Attempting registration via proxy server. auth/register.go:173
[CLIENT]    DEBU HTTPS client init(proxyAddr=teleport.example.com:443, insecure=false) client/weblogin.go:228
[CLIENT]    DEBU Attempting https://teleport.example.com:443/v1/webapi/host/credentials client/https_client.go:92
[AUTH]      INFO Successfully registered via proxy server. auth/register.go:180

The issue here appears to be with the reverse tunnel connection. If you look at the end of that message, the actual error is: ssh: handshake failed: EOF." service/connect.go:86 - this is why I was suggesting that you telnet to port 3024 to see whether the SSH tunnel listener is responding. It does seem to be responding as expected, so there's likely some other issue here.

What versions of Teleport are you running on your auth/proxy and your node?

abjoseph commented 2 years ago

Understood, regarding teleport.cluster.local-thanks for adding some context.

Both the server and node are using the same version:

root@teleport:~# teleport version
Teleport v9.1.1 git:v9.1.1-0-gb0129ff5e go1.17.9

KasperSkytte commented 2 years ago

@abjoseph Here are my configuration files using the SWAG Docker container as nginx reverse proxy. I just had to make sure a certificate was retrieved for *.teleport.domain.com, which in my case is done by just setting an environment variable EXTRA_DOMAINS: "*.teleport.domain.com" when starting the SWAG Docker container. I run teleport in a Docker container named teleport on the same Docker network as the reverse-proxy, so it's referred to by container name

Ansible task for the reverse-proxy container

- name: "Deploy SWAG container"
  docker_container:
    name: swag
    recreate: false
    restart_policy: unless-stopped
    image: ghcr.io/linuxserver/swag:latest
    published_ports:
      - 443:443/tcp
    env:
      TZ: "{{ TZ }}"
      PUID: "1000"
      PGID: "1000"
      URL: domain.com
      SUBDOMAINS: wildcard
      VALIDATION: dns
      CERTPROVIDER: zerossl
      EMAIL: <redacted>
      DNSPLUGIN: cloudflare
      ONLY_SUBDOMAINS: "false"
      PROPAGATION: "10"
      EXTRA_DOMAINS: "*.teleport.domain.com"
      STAGING: "false"
    networks:
      - name: swag_network
    volumes:
      - "local_conf_dir_somewhere:/config"
    capabilities:
      - NET_ADMIN

teleport nginx proxy conf

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name teleport.*;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app teleport;
        set $upstream_port 443;
        set $upstream_proto https;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }

}

server {
    listen 443 ssl;
    listen [::]:443 ssl;

    server_name *.teleport.domain.com;

    include /config/nginx/ssl.conf;

    client_max_body_size 0;

    location / {
        include /config/nginx/proxy.conf;
        include /config/nginx/resolver.conf;
        set $upstream_app teleport;
        set $upstream_port 443;
        set $upstream_proto https;
        proxy_pass $upstream_proto://$upstream_app:$upstream_port;

    }
}

`teleport.yml`

teleport:
  nodename: teleport.domain.com
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: <redacted>
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  public_addr: teleport.domain.com:3025
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:443
  ssh_public_addr: teleport.domain.com:3023
  public_addr: teleport.domain.com:443
  tunnel_listen_addr: 0.0.0.0:3024
  https_keypairs: []
  acme: {}

abjoseph commented 2 years ago

@KasperSkytte Thanks for replying and providing your configuration. Could you also add your docker-compose file for the teleport container? I want to try and mirror your setup and see if I run into the same issue.

KasperSkytte commented 2 years ago

@abjoseph Sure. I use ansible to manage everything, but docker arguments are the same. In addition to the above I also create a docker network with an explicit subnet to use, but that's not necessary if you use docker-compose since a network will be created for all containers in the compose file. It might come in handy to read https://docs.linuxserver.io/general/swag. I've really enjoyed SWAG so far. I have created ansible roles for setting up teleport clients and apps if you need.

- name: "Create network swag_network"
  docker_network:
    name: swag_network
    state: present
    ipam_config:
      - subnet: "172.16.96.0/24"

- name: "Deploy teleport"
  docker_container:
    name: teleport
    recreate: false
    restart_policy: unless-stopped
    entrypoint: /bin/sh
    hostname: teleport.domain.com
    image: quay.io/gravitational/teleport:9.1.0
    command: -c "/bin/dumb-init teleport start -c /etc/teleport/teleport.yaml"
    published_ports:
      - "3023:3023"
      - "3024:3024"
      - "3025:3025"
    exposed_ports:
      - "3080:443"
    networks:
      - name: swag_network
    volumes:
      - "{{ ansible_env.HOME }}/teleport/config:/etc/teleport"
      - "{{ ansible_env.HOME }}/teleport/data:/var/lib/teleport"

ColaIan commented 2 years ago

I'm trying to setup something similar to @abjoseph with nginx and teleport that is not inside of a container, however whenever I set the public_addr to tele.:443 instead of tele., my applications will have a redirection loop between tele. and app.tele., and this would show up in log:

teleport[821333]: 2022-05-02T07:25:49-07:00 WARN [APP:SERVI] Failed to create remote tunnel: failed to dial: all auth methods failed, conn: <nil>. leaseID:255 target:tele.<my_domain>:443 reversetunnel/agent.go:393

Details here: https://github.com/gravitational/teleport/discussions/12349#discussioncomment-2671012

I have also tried adding the following proxy_service configs but nothing changed either:

  web_listen_addr: 0.0.0.0:3080
  public_addr: tele.pydrox.com:443
  listen_addr: 0.0.0.0:3023
  ssh_public_addr: tele.pydrox.com:3023
  tunnel_listen_addr: 0.0.0.0:3024
  tunnel_public_addr: tele.pydrox.com:3024

KasperSkytte commented 2 years ago

@ColaIan Do you have all the required ports open and forwarded correctly?

ColaIan commented 2 years ago

@ColaIan Do you have all the required ports open and forwarded correctly?

Yes I do (basically the firewall is completely open, nothing is blocked), and it looks like a weird redirection loop

sbourdette commented 2 years ago

Same for me. I'm trying to do exactly the same with the same infinite loop ... Any solutions or PR ?

csiber commented 2 years ago

I had a similar problem, also the meeting of nginx proxy and teleport. After all, the Cloudflare proxy was the cause of the error. I turned it off and it worked immediately.

mtgnoah commented 2 years ago

Has anyone got this working with cloudflare tunnel? I am exploring that option because it is impossible for me to get a static ip.

sbourdette commented 1 year ago

I had a similar problem, also the meeting of nginx proxy and teleport. After all, the Cloudflare proxy was the cause of the error. I turned it off and it worked immediately.

Could you hare us your nginx config ?

avluis commented 1 year ago

So I ended up turning this around a bit as I enjoy the convenience of tsh ssh and wanted to extend that to ssh proxy after upping my ssh_config game

As I already had a nginx setup for reverse proxy duties I was also seeking answers to handling this

What works for me is I've ended up setting up a tunnel to that nginx instance and updated the relevant DNS records in Cloudflare (including wildcards); this has the added benefit of being able to really lock things down!

From here, double check only required ports are open in your teleport host and update your router DNAT to point to it -- lock it down in there as well; if the default port (3022 for ssh) fails to connect, teleport/tsh automatically tries 443 (you can of course make needed changes if this is too slow for you)

sbourdette commented 1 year ago

Hi, Any news or tips to make it working ?

webvictim commented 1 year ago

This is now fully supported with the release of Teleport 13.

Here is a guide on how to configure Teleport behind nginx as a reverse proxy: https://github.com/gravitational/teleport/discussions/26445

lordrip commented 1 year ago

Cool, thanks for the heads-up @webvictim, I'm looking forward to give it a try :tada:

gravitational / teleport