tsh: invalid character '<' looking for beginning of value

d9705996 commented 2 years ago

Description

What happened:

tsh login --proxy teleport.example.com:443 --insecure
ERROR: invalid character '<' looking for beginning of value

What you expected to happen:

tsh to successfully connect to the cluster

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

Setup teleport cluster with TLS routing
Successfully create a user /usr/local/bin/tctl users add dwalker --roles=editor,auditor,access
Complete user setup in Web UI
Confirm user can authenticate to Web UI

Server Details

Teleport version (run teleport version): Teleport v8.0.5 git:v8.0.5-0-g0e304c3 go1.17.3
Server OS (e.g. from /etc/os-release): CentOS Linux release 7.9.2009 (Core)
Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): Dedicated Hardware
Additional details: HAProxy is configured to pass SSL connection to teleport.example.com:443 to the teleport service unmolested.

Client Details

Tsh version (tsh version): Teleport v8.0.1 git:v8.0.1-0-gb95fb530b go1.17.3
Computer OS (e.g. Linux, macOS, Windows): Windows
Browser version (for UI-related issues): N/A
Installed via (e.g. apt, yum, brew, website download): website download
Additional details:

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

Start Teleport with --debug flag (teleport --debug)
Run tsh with --debug flag (tsh --debug)

tsh login --proxy teleport.example.com:443 --debug --insecure
DEBU [CLIENT]    open C:\Users\dwalker\.tsh\teleport.example.com.yaml: The system cannot find the file specified. client\api.go:733
INFO [CLIENT]    No teleport login given. defaulting to dwalker client\api.go:1061
INFO [CLIENT]    no host login given. defaulting to dwalker client\api.go:1071
ERRO [CLIENT]    [KEY AGENT] Unable to connect to SSH agent on socket: "". client\api.go:2877
DEBU [CLIENT]    not using loopback pool for remote proxy addr: teleport.example.com:443 client\api.go:2842
DEBU             Attempting GET teleport.example.com:443/webapi/ping webclient\webclient.go:62

ERROR REPORT:
Original Error: *json.SyntaxError invalid character &#39;&lt;&#39; looking for beginning of value
Stack Trace:
        /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/teleport/api/client/webclient/webcli
ent.go:143 github.com/gravitational/teleport/api/client/webclient.Ping
        /go/src/github.com/gravitational/teleport/lib/client/api.go:2432 github.com/gravitational/teleport/lib/client.
(*TeleportClient).Ping
        /go/src/github.com/gravitational/teleport/lib/client/api.go:2286 github.com/gravitational/teleport/lib/client.
(*TeleportClient).Login
        /go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:839 main.onLogin
        /go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:617 main.Run
        /go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:274 main.main
        /opt/go/src/runtime/proc.go:255 runtime.main
        /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: invalid character &#39;&lt;&#39; looking for beginning of value

Manually querying the URI on the same client returns a valid JSON response

curl -Ss https://teleport.example.com:443/webapi/ping | jq
{
  "auth": {
    "type": "local",
    "second_factor": "otp",
    "preferred_local_mfa": "otp",
    "has_motd": false
  },
  "proxy": {
    "kube": {
      "enabled": true,
      "listen_addr": "0.0.0.0:443"
    },
    "ssh": {
      "listen_addr": "0.0.0.0:443",
      "tunnel_listen_addr": "0.0.0.0:443",
      "public_addr": "teleport.example.com:443"
    },
    "db": {
      "mysql_listen_addr": "0.0.0.0:443"
    },
    "tls_routing_enabled": true
  },
  "server_version": "8.0.5",
  "min_client_version": "7.0.0"
}

stefansedich commented 2 years ago

Not sure if this is related but I am seeing the following from our agents trying to connect to the auth server upon upgrading to 8.0.5:

App failed to establish connection to cluster: special characters are not allowed in resource names, please use name composed only from characters, hyphens, dots, and plus signs: "/tokens/<redacted>", invalid character '<' looking for beginning of value. service/connect.go:68

d9705996 commented 2 years ago

Same issue after upgrading client and server to 8.0.6

tsh version
Teleport v8.0.6 git:v8.0.6-0-g807c6d662 go1.17.3

teleport version
Teleport v8.0.6 git:v8.0.6-0-g807c6d6 go1.17.3

stefansedich commented 2 years ago

FWIW we were seeing the same issues with 8.0.3 and 8.0.5 and have had to downgrade to 7.3.5 to get everthing up and running again in our environment.

RiwanBodereau commented 2 years ago

Same issue here, downgrading tsh to 8.0.0 fixed it for me

stefansedich commented 2 years ago

So after some time of working fine with 7.3.5 this issue reared it's head again and after some digging the fix that worked for us was to generate new static tokens with no special characters.

This did the trick and we were also able to upgrade to 8.x successfully, not sure if this is your issue too @RiwanBodereau @d9705996 but worth looking at. The odd part was we have been using these tokens for a long time with 7.x with no issue then after a reboot the other day we began seeing them.

strideynet commented 2 years ago

I've reproduced this issue when attempting to log in with --proxy set to an address that is different to the one configured in proxy_service.public_addr, but where that name does route to the proxy and the proxy does have a valid certificate for that name. It appears this is caused because the Proxy HTTP API detecting that the host is not the host configured and attempting to redirect to /web/launch/$CONFIGURED_PROXY_ADDRESS.

Partially related to #13012.

strideynet commented 2 years ago

For some further detail, this error also appears in tbot during the join process when a proxy address is specified, but connecting via proxy fails and then the client falls back to trying to connect to it as an auth server.

You can see this within the aggregate error returned, which includes the real error and then this misleading red-herring:

ERROR: user "bot-bot1" has already been issued a renewable certificate and cannot be issued another; consider deleting and recreating the bot, invalid character '<' looking for beginning of value

We should better detect when this response comes from the proxy, rather than the auth server, and omit this error and output one suggesting that it has tried to connect to a proxy as if it were an auth server.

gravitational / teleport