tbot receives timeout when accessing /v1/webapi/host/credentials #39549

Closed dk-0x89 closed 7 months ago

Hello everyone,

I use Teleport in combination with GitLab and the pipeline. Unfortunately, the tbot receives a 404 error when it tries to reach /v1/webapi/host/credentials, and I can't quite figure out why. When I make the WebAPI call with curl using GET instead of POST, I also receive a 404, as stated in the documentation. When I use POST via curl, I get the error that my payload is not correct, which is fine because I'm not sending any data. However, this behavior still shouldn't result in a 404.

For completeness, I'm using the Teleport Docker container and Traefik in front of it. I expose only 80/443 with Traefik. tsh/tctl work perfectly; currently, the issue lies only with tbot.

Here are excerpts from my GitLab pipeline:

tbot.yaml

version: v2
proxy_server: teleport.tld.de:443
onboarding:
  join_method: gitlab
  token: [MASKED]
  ca_pins:
    - "sha256:0bf34063d299828xxxxxxxx5842ec371781b6cbddd26dxxxxxxxxxx"
oneshot: true
storage:
  type: memory
outputs: 
  - type: identity
    destination: 
      type: directory
      path: /opt/ssh-ansible

Curl on webapi/ping

$ curl -X GET https://teleport.tld.de:443/v1/webapi/ping
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     

{
  "auth": {
    "type": "local",
    "second_factor": "otp",
    "preferred_local_mfa": "otp",
    "local": {
      "name": ""
    },
    "private_key_policy": "none",
    "piv_slot": "",
    "device_trust_disabled": true,
    "device_trust": {
      "disabled": true
    },
    "has_motd": false,
    "default_session_ttl": "12h0m0s"
  },
  "proxy": {
    "kube": {
      "enabled": true,
      "listen_addr": "0.0.0.0:3080"
    },
    "ssh": {
      "listen_addr": "0.0.0.0:3080",
      "tunnel_listen_addr": "0.0.0.0:3080",
      "web_listen_addr": "0.0.0.0:3080",
      "public_addr": "teleport.tld.de:443"
    },
    "db": {
      "postgres_listen_addr": "0.0.0.0:3080",
      "mysql_listen_addr": "0.0.0.0:3080"
    },
    "tls_routing_enabled": true,
    "assist_enabled": false
  },
  "server_version": "15.1.8",
  "min_client_version": "14.0.0",
  "cluster_name": "teleport.tld.de",
  "automatic_upgrades": false
}

curl on credentials with post

$ curl -X POST https://teleport.tld.de:443/v1/webapi/host/credentials
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    61  100    61    0     0    539      0 --:--:-- --:--:-- --:--:--   535
{
    "error": {
        "message": "invalid request"
    }

tbot Version

$ tbot version
Teleport v15.1.8 git:v15.1.8-0-g86c1d3f go1.21.8

tbot debug error massage

$ TELEPORT_ANONYMOUS_TELEMETRY=1 tbot start -c tbot.yml --debug
INFO [TBOT]      Created directory "/opt/ssh-ansible" config/destination_directory.go:130
INFO [TBOT]      Anonymous telemetry is enabled. Find out more about Machine ID's anonymous telemetry at https://goteleport.com/docs/machine-id/reference/telemetry/ tbot/anonymous_telemetry.go:89
INFO [TBOT:IDEN] Initializing bot identity. tbot/service_bot_identity.go:148
INFO [TBOT:IDEN] Fetching bot identity using token. tbot/service_bot_identity.go:384
DEBU [AUTH]      Registering node to the cluster. proxy-server:{teleport.tld.de:443 tcp } auth/register.go:267
INFO [AUTH]      Attempting registration via proxy server. auth/register.go:288
DEBU [CLIENT]    HTTPS client init(proxyAddr=teleport.tld.de:443, insecure=false, extraHeaders=map[]) client/weblogin.go:346
DEBU [CLIENT]    Attempting https://teleport.tld.de:443/v1/webapi/host/credentials client/https_client.go:87
DEBU [TBOT]      Successfully transmitted anonymous telemetry distinct_id:51796926-1116-4d36-802e-32ec23f549c5 duration:592.867289ms tbot/anonymous_telemetry.go:127
DEBU [AUTH]      Registration via proxy server failed. error:[
ERROR REPORT:
Original Error: *trace.ConnectionProblemError net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Stack Trace:
Caught:
    github.com/gravitational/teleport/lib/httplib/httplib.go:216 github.com/gravitational/teleport/lib/httplib.ConvertResponse
    github.com/gravitational/teleport/lib/client/https_client.go:92 github.com/gravitational/teleport/lib/client.(*WebClient).PostJSONWithFallback
    github.com/gravitational/teleport/lib/client/weblogin.go:687 github.com/gravitational/teleport/lib/client.HostCredentials
    github.com/gravitational/teleport/lib/auth/register.go:345 github.com/gravitational/teleport/lib/auth.registerThroughProxy
    github.com/gravitational/teleport/lib/auth/register.go:289 github.com/gravitational/teleport/lib/auth.Register
    github.com/gravitational/teleport/lib/tbot/service_bot_identity.go:438 github.com/gravitational/teleport/lib/tbot.botIdentityFromToken
    github.com/gravitational/teleport/lib/tbot/service_bot_identity.go:184 github.com/gravitational/teleport/lib/tbot.(*identityService).Initialize
    github.com/gravitational/teleport/lib/tbot/tbot.go:184 github.com/gravitational/teleport/lib/tbot.(*Bot).Run
    github.com/gravitational/teleport/tool/tbot/main.go:417 main.onStart
    github.com/gravitational/teleport/tool/tbot/main.go:225 main.Run
    github.com/gravitational/teleport/tool/tbot/main.go:56 main.main
    runtime/proc.go:267 runtime.main
    runtime/asm_amd64.s:1650 runtime.goexit
User Message: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
] auth/register.go:292
DEBU [TBOT]      Unlocking bot storage. tbot/tbot.go:124
ERROR REPORT:
Original Error: trace.aggregate net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Stack Trace:
    github.com/gravitational/teleport/lib/auth/register.go:298 github.com/gravitational/teleport/lib/auth.Register
    github.com/gravitational/teleport/lib/tbot/service_bot_identity.go:438 github.com/gravitational/teleport/lib/tbot.botIdentityFromToken
    github.com/gravitational/teleport/lib/tbot/service_bot_identity.go:184 github.com/gravitational/teleport/lib/tbot.(*identityService).Initialize
    github.com/gravitational/teleport/lib/tbot/tbot.go:184 github.com/gravitational/teleport/lib/tbot.(*Bot).Run
    github.com/gravitational/teleport/tool/tbot/main.go:417 main.onStart
    github.com/gravitational/teleport/tool/tbot/main.go:225 main.Run
    github.com/gravitational/teleport/tool/tbot/main.go:56 main.main
    runtime/proc.go:267 runtime.main
    runtime/asm_amd64.s:1650 runtime.goexit
User Message: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Cleaning up project directory and file based variables 00:01
ERROR: Job failed: exit code 1

It seems that there weren't enough ports open to reach GitLab from the Teleport server... I'm sorry.

gravitational / teleport