search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.63k stars 1.76k forks source link

teleport issues behind web proxy #23895

Closed juju4 closed 1 year ago

juju4 commented 1 year ago

Observing similar issue to https://github.com/gravitational/teleport/issues/18828 when attempting to connect from behind web proxy. teleport server is in allowed list but get immediate disconnected message. Unfiltered client gets normal ssh session Seems to also affect applications (web forwarding - error "NetworkError when attempting to fetch resource.") and Desktop (Windows - spinning forever)

On proxy side, all logs return 200. On unfiltered client, pcap only shows tcp/443 traffic. Disabling TLS inspection for the domain helps to get web ssh session working intermittently (?) but not applications and desktops. still getting invalid session server side. likely because of websockets. This was not affecting cli ssh connections.

Expected behavior: Connecting properly through web ssh/applications/desktops

Current behavior: Can't connect

Bug details:

webvictim commented 1 year ago

Can you explain how the web proxy you're using works? Is it an HTTP CONNECT proxy, or is it doing TLS termination?

juju4 commented 1 year ago

both. As said, TLS decryption was enabled initially and disabled after for the domain. Now just CONNECT but still issues Tested with squid.

webvictim commented 1 year ago

Interesting. Can you share the /etc/teleport.yaml config file from your cluster?

juju4 commented 1 year ago

here is a redacted version

version: v3
teleport:
  nodename: ip-172-16-x-y
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: 172.16.x.y:3000
  storage:
    type: dynamodb
    region: "us-east-2"
    table_name: teleport-pcq1234
    audit_events_uri:  ['dynamodb://teleport-pcq1234-events', 'file:///var/lib/teleport/audit/events', 'stdout://']
    audit_sessions_uri: "s3://teleport-kqn1w/records"
    audit_retention_period: 90d
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: teleport.example.com
  proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  public_addr: teleport.example.com:443
  https_keypairs: []
  acme:
    enabled: "yes"
    email: admin@teleport.example.com
app_service:
  enabled: "yes"
  apps:
    - name: wiki
      uri: "http://172.16.1.1:8080"
      labels:
        env: "wiki"
    - name: "logs"
      uri: "http://172.16.1.2:8000"
      #insecure_skip_verify: true
      rewrite:
        headers:
          - "Host: logs.teleport.example.com"
      labels:
        env: logging
    - name: "grafana"
      uri: "https://172.16.1.3:443"
      insecure_skip_verify: true
      labels:
        env: intra
      rewrite:
        headers:
          - "Host: grafana.teleport.example.com"
          - "Origin: https://grafana.teleport.example.com"
windows_desktop_service:
  enabled: yes
  listen_addr: 0.0.0.0:3028
  public_addr: "teleport.example.com:3028"
  ldap:
    addr:     '172.16.2.1:636'
    domain:   'example.internal'
    username: 'ADDOMAIN\svc-teleport'
    sid: 'S-1-5-21-2369732838-0123456798-123456789-1234'
    server_name: 'dc.example.internal'
    insecure_skip_verify: false
    ldap_ca_cert: |
        -----BEGIN CERTIFICATE-----
        REDACTED
        -----END CERTIFICATE-----
  discovery:
    base_dn: '*'
  labels:
    teleport.internal/resource-id: REDACTED
webvictim commented 1 year ago

Thanks. As you've correctly surmised, I suspect that the issue with Windows sessions is due to (a lack of) websocket proxying - Windows Desktop access requires the use of websockets to work correctly. Presumably what you're saying is that Windows Desktop sessions work fine with no HTTP CONNECT proxy in use, but fails when it's being used?

With regard to application access, I would expect it to work fine in the browser. Application access via CLI (tsh app login) requires the use of client certificates which may also be interfered with somewhat by the proxy.

Does Application Access in the browser work correctly without the use of the HTTP CONNECT proxy?

juju4 commented 1 year ago

Access when unfiltered/no proxy is working fine.

When filtered (firewall+proxy), only the cli tsh+ssh is working fine. Web applications get either "NetworkError when attempting to fetch resource.", either infinite reload loop. Desktops/Win side keeps in connection spinning. Web app access is intermittently working with just the proxy in connect mode, no tls interception. Maybe an extra port is needed? but from proxy logs, everything is allowed.

Note that normally with just connect and no tls inspection, websockets should work through proxy as is, at least the wss ones, I believe.

webvictim commented 1 year ago

Can you check the network inspector tab in your browser when trying to connect to a Windows session? It should attempt to start a websocket - see whether the call succeeds or whether there's an error. It might also be an idea to turn up the proxy's logging to see whether it sheds any light on what's happening.

Also, just to be clear - what you're doing is this, correct?

Browser -> HTTP proxy -> Teleport

The "invalid session" message in the browser might indicate some issue with the application access session being created in the backend database, but this generally only happens with HA setups - do you have more than one Teleport auth/proxy server?

juju4 commented 1 year ago

On Desktops connections, I only see repetead GET teleport.example.com on status file with 200 status but going slowly and repeating 5 times+ initiated by app.<random?>.js. This happens after a context and active file. On working side, I see a connect before status, identified as websocket and 101 status. In both case, there is favicon.ico file call that happenings. got 200 on working side and seems stalling on non-working side or not exactly as I have a response icon but for some reason no status code returned or displayed.

So, websocket call does not even seem to happen.

I already have usual logging on the proxy or do you mean full debug logs? without the tls inspection, I don't think it will give more than the CONNECT on teleport server.

On connection flow, yes. more complete flow: Browser -> HTTP/HTTPS Proxy -> Internet -> Teleport server -> Targeted infra And only single teleport server, no HA.

zmb3 commented 1 year ago

Closing due to inactivity. I would suggest starting a GitHub discussion or joining our community Slack workspace to see if anyone else has seen similar issues with the squid proxy.