search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.62k stars 1.76k forks source link

Invalid redirect returned to AWS CLI obscuring real error #10790

Open Tener opened 2 years ago

Tener commented 2 years ago

Description

What happened:

In case of non-operational AWS Console access using "tsh aws" results in confusing error message. Instead of XML with information we return HTTP 302 redirect, which the AWS CLI doesn't know how to handle. See #10789 for how to make the faulty setup which results in this.

What you expected to happen:

A meaningful error. Obtaining web session has failed, this should be caught by Teleport proxy handling the connection and reported as such.

Reproduction Steps

See #10789 for detailed setup steps. This is the same scenario, except using "tsh aws" command.

Server Details

Client Details

Debug Logs

Teleport logs. Note that these mix the original cause of failure (inability to create reverse tunnel from node to itself - dial tcp timeout) and the resulting failure to create the app session (which ultimately results in confusing redirect to /web).

Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [APP:SERVI] Dial to tele-1.aws.tener.io:3080 failed. error:[
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: ERROR REPORT:
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: Original Error: *net.OpError dial tcp 3.66.85.216:3080: i/o timeout
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: Stack Trace:
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/utils/proxy/proxy.go:129 github.com/gravitational/teleport/lib/utils/proxy.directDial.DialTimeout
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:292 github.com/gravitational/teleport/lib/reversetunnel.(*Agent).connect
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/agent.go:385 github.com/gravitational/teleport/lib/reversetunnel.(*Agent).run
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: User Message: dial tcp 3.66.85.216:3080: i/o timeout] leaseID:94 target:tele-1.aws.tener.io:3080 reversetunnel/agent.go:294
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z WARN [APP:SERVI] Failed to create remote tunnel: failed to dial: all auth methods failed, conn: <nil>. leaseID:94 target:tele-1.aws.tener.io:3080 reversetunnel/agent.go:387
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [APP:SERVI] Changing state connecting -> disconnected. leaseID:94 target:tele-1.aws.tener.io:3080 reversetunnel/agent.go:208
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [PROXY:AGE] Adding agent(leaseID=95,state=connecting) -> main:tele-1.aws.tener.io:3080. cluster:main reversetunnel/agentpool.go:308
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:337
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:244
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [APP:SERVI] Pool is closing agent. leaseID:94 target:tele-1.aws.tener.io:3080 reversetunnel/agentpool.go:238
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [AUTH]      ClientCertPool -> cert(main issued by main:84549874656568164947356731045716606294) auth/middleware.go:609
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [AUTH]      ClientCertPool -> cert(main issued by main:62115784750887467050448464695030025758) auth/middleware.go:609
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [PROXY:SER] Dialing from: "@web-proxy" to: "@local-node". trace.fields:map[cluster:main] reversetunnel/localsite.go:198
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [PROXY:SER] Error occurred while dialing through a tunnel. address:@local-node error:[
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: ERROR REPORT:
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: Original Error: *trace.NotFoundError no tunnel connection found: no app reverse tunnel for 638f1d8b-bd06-4252-a498-644b1ed28fe1.main found
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: Stack Trace:
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:279 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).dialTunnel
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:303 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).getConn
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:200 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).DialTCP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:193 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).Dial
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/transport.go:238 github.com/gravitational/teleport/lib/web/app.dialAppServer
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/match.go:88 github.com/gravitational/teleport/lib/web/app.MatchHealthy.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/match.go:97 github.com/gravitational/teleport/lib/web/app.MatchAll.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/match.go:54 github.com/gravitational/teleport/lib/web/app.Match
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/session.go:72 github.com/gravitational/teleport/lib/web/app.(*Handler).newSession
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/handler.go:263 github.com/gravitational/teleport/lib/web/app.(*Handler).getSession
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/handler.go:177 github.com/gravitational/teleport/lib/web/app.(*Handler).authenticate
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/middleware.go:50 github.com/gravitational/teleport/lib/web/app.(*Handler).withAuth.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/middleware.go:124 github.com/gravitational/teleport/lib/web/app.makeHandler.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/net/http/server.go:2046 net/http.HandlerFunc.ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /tmp/gomodcache/github.com/julienschmidt/httprouter@v1.3.0/router.go:448 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/handler.go:135 github.com/gravitational/teleport/lib/web/app.(*Handler).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/apiserver.go:207 github.com/gravitational/teleport/lib/web.(*APIHandler).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /tmp/gomodcache/github.com/gravitational/oxy@v0.0.0-20211213172937-a1ba0900a4c9/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /tmp/gomodcache/github.com/gravitational/oxy@v0.0.0-20211213172937-a1ba0900a4c9/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/net/http/server.go:2878 net/http.serverHandler.ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/net/http/server.go:1929 net/http.(*conn).serve
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: User Message: no tunnel connection found: no app reverse tunnel for 638f1d8b-bd06-4252-a498-644b1ed28fe1.main found] trace.fields:map[cluster:main] reversetunnel/localsite.go:312
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [PROXY:SER] Dialing from: "@web-proxy" to: "@local-node". trace.fields:map[cluster:main] reversetunnel/localsite.go:198
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z DEBU [PROXY:SER] Error occurred while dialing through a tunnel. address:@local-node error:[
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: ERROR REPORT:
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: Original Error: *trace.NotFoundError no tunnel connection found: no app reverse tunnel for 638f1d8b-bd06-4252-a498-644b1ed28fe1.main found
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: Stack Trace:
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:279 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).dialTunnel
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:303 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).getConn
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:200 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).DialTCP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/reversetunnel/localsite.go:193 github.com/gravitational/teleport/lib/reversetunnel.(*localSite).Dial
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/transport.go:238 github.com/gravitational/teleport/lib/web/app.dialAppServer
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/match.go:88 github.com/gravitational/teleport/lib/web/app.MatchHealthy.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/match.go:97 github.com/gravitational/teleport/lib/web/app.MatchAll.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/match.go:54 github.com/gravitational/teleport/lib/web/app.Match
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/session.go:72 github.com/gravitational/teleport/lib/web/app.(*Handler).newSession
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/handler.go:263 github.com/gravitational/teleport/lib/web/app.(*Handler).getSession
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/handler.go:177 github.com/gravitational/teleport/lib/web/app.(*Handler).authenticate
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/middleware.go:50 github.com/gravitational/teleport/lib/web/app.(*Handler).withAuth.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/middleware.go:124 github.com/gravitational/teleport/lib/web/app.makeHandler.func1
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/net/http/server.go:2046 net/http.HandlerFunc.ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /tmp/gomodcache/github.com/julienschmidt/httprouter@v1.3.0/router.go:448 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/app/handler.go:135 github.com/gravitational/teleport/lib/web/app.(*Handler).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /go/src/github.com/gravitational/teleport/lib/web/apiserver.go:207 github.com/gravitational/teleport/lib/web.(*APIHandler).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /tmp/gomodcache/github.com/gravitational/oxy@v0.0.0-20211213172937-a1ba0900a4c9/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /tmp/gomodcache/github.com/gravitational/oxy@v0.0.0-20211213172937-a1ba0900a4c9/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/net/http/server.go:2878 net/http.serverHandler.ServeHTTP
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/net/http/server.go:1929 net/http.(*conn).serve
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: User Message: no tunnel connection found: no app reverse tunnel for 638f1d8b-bd06-4252-a498-644b1ed28fe1.main found] trace.fields:map[cluster:main] reversetunnel/localsite.go:312
Mar 03 14:48:03 ip-172-31-27-191.eu-central-1.compute.internal teleport[30590]: 2022-03-03T14:48:03Z WARN [APP:WEB]   Failed to get session: failed to match applications. app/handler.go:179

tsh logs. Notice double --debug flag: one applies to tsh, one to aws.

tsh --debug aws ec2 describe-instances --output table --region us-west-1 --debug
INFO [CLIENT]    no host login given. defaulting to tener client/api.go:1119
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.SFyEO41sCZ/Listeners" client/api.go:3021
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/tener/.tsh/keys/tele-1.aws.tener.io/tener-x509.pem" valid until "2022-03-03 22:17:20 +0000 UTC". client/keystore.go:285
DEBU [KEYSTORE]  Reading certificates from path "/Users/tener/.tsh/keys/tele-1.aws.tener.io/tener-ssh/main-cert.pub". client/keystore.go:308
INFO [KEYAGENT]  Loading SSH key for user "tener" and cluster "main". client/keyagent.go:191
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/tener/.tsh/keys/tele-1.aws.tener.io/tener-x509.pem" valid until "2022-03-03 22:17:20 +0000 UTC". client/keystore.go:285
DEBU [KEYSTORE]  Reading certificates from path "/Users/tener/.tsh/keys/tele-1.aws.tener.io/tener-ssh/main-cert.pub". client/keystore.go:308
DEBU [KEYSTORE]  Reading certificates from path "/Users/tener/.tsh/keys/tele-1.aws.tener.io/tener-app/main". client/keystore.go:308
DEBU [KEYSTORE]  Returning Teleport TLS certificate "/Users/tener/.tsh/keys/tele-1.aws.tener.io/tener-x509.pem" valid until "2022-03-03 22:17:20 +0000 UTC". client/keystore.go:285
DEBU [KEYSTORE]  Reading certificates from path "/Users/tener/.tsh/keys/tele-1.aws.tener.io/tener-app/main". client/keystore.go:308
2022-03-03 15:59:18,096 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.4.21 Python/3.8.8 Darwin/21.3.0 exe/x86_64
2022-03-03 15:59:18,096 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'describe-instances', '--output', 'table', '--region', 'us-west-1', '--debug', '--endpoint-url=https://localhost:61923/', '--ca-bundle=/var/folders/c0/_6x4ddps6zn5js_4y31xlz380000gn/T/2125775390_aws_local_proxy_cert.pem']
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fd978e38f70>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fd978c90550>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fd978c33d30>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fd978c39d30>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fd978e4a9d0>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fd978cdd3a0>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2022-03-03 15:59:18,107 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fd978e40c10>
2022-03-03 15:59:18,107 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fd978d87e50>
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fd978d909d0>
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fd978d90940>
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fd978d90af0>
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fd978d90a60>
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fd978ee6c80>
2022-03-03 15:59:18,109 - MainThread - botocore.session - DEBUG - Setting config variable for region to 'us-west-1'
2022-03-03 15:59:18,109 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.4.21 Python/3.8.8 Darwin/21.3.0 exe/x86_64 prompt/off
2022-03-03 15:59:18,109 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['ec2', 'describe-instances', '--output', 'table', '--region', 'us-west-1', '--debug', '--endpoint-url=https://localhost:61923/', '--ca-bundle=/var/folders/c0/_6x4ddps6zn5js_4y31xlz380000gn/T/2125775390_aws_local_proxy_cert.pem']
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fd978e3a5e0>
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fd9789ed280>
2022-03-03 15:59:18,109 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fd978eadaf0>
2022-03-03 15:59:18,110 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fd9789e6700>
2022-03-03 15:59:18,110 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fd978a1fca0>
2022-03-03 15:59:18,111 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2022-03-03 15:59:18,117 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fd978cdd280>
2022-03-03 15:59:18,117 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fd978c8e430>
2022-03-03 15:59:18,130 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/ec2/2016-11-15/service-2.json
2022-03-03 15:59:18,177 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler functools.partial(<function _remove_commands at 0x7fd978db1280>, commands_to_remove=['import-instance', 'import-volume'])
2022-03-03 15:59:18,177 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-instance
2022-03-03 15:59:18,177 - MainThread - awscli.customizations.removals - DEBUG - Removing operation: import-volume
2022-03-03 15:59:18,177 - MainThread - botocore.hooks - DEBUG - Event building-command-table.ec2: calling handler <function add_waiters at 0x7fd978e40c10>
2022-03-03 15:59:18,187 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/ec2/2016-11-15/waiters-2.json
2022-03-03 15:59:18,188 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('filters', <awscli.arguments.ListArgument object at 0x7fd998b14130>), ('instance-ids', <awscli.arguments.ListArgument object at 0x7fd998b14160>), ('dry-run', <awscli.arguments.BooleanArgument object at 0x7fd998b14190>), ('no-dry-run', <awscli.arguments.BooleanArgument object at 0x7fd998b141c0>), ('max-results', <awscli.arguments.CLIArgument object at 0x7fd998b141f0>), ('next-token', <awscli.arguments.CLIArgument object at 0x7fd998b14220>)])
2022-03-03 15:59:18,188 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function add_streaming_output_arg at 0x7fd978e3ab80>
2022-03-03 15:59:18,188 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function rename_arg.<locals>._rename_arg at 0x7fd978eafca0>
2022-03-03 15:59:18,188 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function rename_arg.<locals>._rename_arg at 0x7fd978eafd30>
2022-03-03 15:59:18,188 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler functools.partial(<function pull_up_bool at 0x7fd978e40670>, event_handler=<botocore.hooks.HierarchicalEmitter object at 0x7fd9789d3d90>)
2022-03-03 15:59:18,188 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function add_cli_input_json at 0x7fd978a26550>
2022-03-03 15:59:18,189 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function add_cli_input_yaml at 0x7fd978a26820>
2022-03-03 15:59:18,189 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function unify_paging_params at 0x7fd978c90c10>
2022-03-03 15:59:18,199 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/ec2/2016-11-15/paginators-1.json
2022-03-03 15:59:18,199 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: DescribeInstances
2022-03-03 15:59:18,199 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.ec2.describe-instances: calling handler <function add_generate_skeleton at 0x7fd978d87430>
2022-03-03 15:59:18,199 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.describe-instances: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fd998b14430>>
2022-03-03 15:59:18,199 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.describe-instances: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fd998b14400>>
2022-03-03 15:59:18,199 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.ec2.describe-instances: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fd998b148e0>>
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.ec2.describe-instances: calling handler functools.partial(<function validate_boolean_mutex_groups at 0x7fd978e40700>, boolean_pairs=[])
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.ec2.describe-instances: calling handler functools.partial(<function check_should_enable_pagination at 0x7fd978c90d30>, ['next-token', 'max-results'], {}, OrderedDict([('filters', <awscli.arguments.ListArgument object at 0x7fd998b14130>), ('instance-ids', <awscli.arguments.ListArgument object at 0x7fd998b14160>), ('dry-run', <awscli.arguments.BooleanArgument object at 0x7fd998b14190>), ('no-dry-run', <awscli.arguments.BooleanArgument object at 0x7fd998b141c0>), ('max-results', <awscli.arguments.CLIArgument object at 0x7fd998b141f0>), ('next-token', <awscli.arguments.CLIArgument object at 0x7fd998b14220>), ('cli-input-json', <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fd998b14430>), ('cli-input-yaml', <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fd998b14400>), ('starting-token', <awscli.customizations.paginate.PageArgument object at 0x7fd998b145e0>), ('page-size', <awscli.customizations.paginate.PageArgument object at 0x7fd998b14760>), ('max-items', <awscli.customizations.paginate.PageArgument object at 0x7fd998b146d0>), ('generate-cli-skeleton', <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fd998b148e0>)]))
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.filters: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.instance-ids: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.dry-run: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.max-results: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.next-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,200 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.starting-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.max-items: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.ec2.describe-instances.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fd978f261c0>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-instances: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fd998b14430>>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-instances: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fd998b14400>>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-instances: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fd998b148e0>>
2022-03-03 15:59:18,201 - MainThread - botocore.hooks - DEBUG - Event calling-command.ec2.describe-instances: calling handler functools.partial(<function check_should_enable_pagination_call_parameters at 0x7fd978ca0160>, ['NextToken', 'MaxResults'])
2022-03-03 15:59:18,201 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2022-03-03 15:59:18,201 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2022-03-03 15:59:18,201 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/endpoints.json
2022-03-03 15:59:18,209 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fd99892b700>
2022-03-03 15:59:18,217 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.ec2: calling handler <function add_generate_presigned_url at 0x7fd998a14af0>
2022-03-03 15:59:18,220 - MainThread - botocore.endpoint - DEBUG - Setting ec2 timeout as (60, 60)
2022-03-03 15:59:18,222 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.ec2.DescribeInstances: calling handler <function base64_decode_input_blobs at 0x7fd978eaf280>
2022-03-03 15:59:18,222 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ec2.DescribeInstances: calling handler <bound method ParameterAlias.alias_parameter_in_call of <botocore.handlers.ParameterAlias object at 0x7fd978022f40>>
2022-03-03 15:59:18,222 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.ec2.DescribeInstances: calling handler <function generate_idempotent_uuid at 0x7fd9780285e0>
2022-03-03 15:59:18,222 - MainThread - botocore.hooks - DEBUG - Event before-call.ec2.DescribeInstances: calling handler <function inject_api_version_header_if_needed at 0x7fd97802fe50>
2022-03-03 15:59:18,222 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeInstances) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'Content-Type': 'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': 'aws-cli/2.4.21 Python/3.8.8 Darwin/21.3.0 exe/x86_64 prompt/off command/ec2.describe-instances'}, 'body': {'Action': 'DescribeInstances', 'Version': '2016-11-15'}, 'url': 'https://localhost:61923/', 'context': {'client_region': 'us-west-1', 'client_config': <botocore.config.Config object at 0x7fd9884d5940>, 'has_streaming_input': False, 'auth_type': None}}
2022-03-03 15:59:18,222 - MainThread - botocore.hooks - DEBUG - Event request-created.ec2.DescribeInstances: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fd9884d5a00>>
2022-03-03 15:59:18,222 - MainThread - botocore.hooks - DEBUG - Event choose-signer.ec2.DescribeInstances: calling handler <function set_operation_specific_signer at 0x7fd9780284c0>
2022-03-03 15:59:18,223 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2022-03-03 15:59:18,223 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-www-form-urlencoded; charset=utf-8
host:localhost:61923
x-amz-date:20220303T145918Z

content-type;host;x-amz-date
6171eb09865e32b0602af0f7957e26573a51f53caaedff02ff88883cb0275885
2022-03-03 15:59:18,223 - MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
20220303T145918Z
20220303/us-west-1/ec2/aws4_request
95d4ba2d2934e535c129c7597809c5c51074d9245e817ab2b0571b17e5be0626
2022-03-03 15:59:18,223 - MainThread - botocore.auth - DEBUG - Signature:
65a217af5387d44ad33892043b38ae8d34fd1f3a80f30bf44ed1da3114d597c4
2022-03-03 15:59:18,223 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://localhost:61923/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/2.4.21 Python/3.8.8 Darwin/21.3.0 exe/x86_64 prompt/off command/ec2.describe-instances', 'X-Amz-Date': b'20220303T145918Z', 'Authorization': b'AWS4-HMAC-SHA256 Credential=14c4b5ec-d6f0-4068-8ff9-f5d28c01fda2/20220303/us-west-1/ec2/aws4_request, SignedHeaders=content-type;host;x-amz-date, Signature=65a217af5387d44ad33892043b38ae8d34fd1f3a80f30bf44ed1da3114d597c4', 'Content-Length': '43'}>
2022-03-03 15:59:18,223 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): localhost:61923
2022-03-03 15:59:18,506 - MainThread - urllib3.connectionpool - DEBUG - https://localhost:61923 "POST / HTTP/1.1" 302 0
2022-03-03 15:59:18,506 - MainThread - botocore.parsers - DEBUG - Response headers: {'Content-Length': '0', 'Date': 'Thu, 03 Mar 2022 14:59:18 GMT', 'Location': 'https://tele-1.aws.tener.io:3080/web/launch/localhost'}
2022-03-03 15:59:18,506 - MainThread - botocore.parsers - DEBUG - Response body:
b''
2022-03-03 15:59:18,507 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/botocore/parsers.py", line 487, in _parse_xml_string_to_dom
  File "<string>", line None
xml.etree.ElementTree.ParseError: no element found: line 1, column 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "awscli/clidriver.py", line 459, in main
  File "awscli/clidriver.py", line 594, in __call__
  File "awscli/clidriver.py", line 770, in __call__
  File "awscli/clidriver.py", line 903, in invoke
  File "awscli/clidriver.py", line 925, in _display_response
  File "awscli/formatter.py", line 82, in __call__
  File "awscli/botocore/paginate.py", line 449, in build_full_result
  File "awscli/botocore/paginate.py", line 255, in __iter__
  File "awscli/botocore/paginate.py", line 332, in _make_request
  File "awscli/botocore/client.py", line 285, in _api_call
  File "awscli/botocore/client.py", line 601, in _make_api_call
  File "awscli/botocore/client.py", line 621, in _make_request
  File "awscli/botocore/endpoint.py", line 103, in make_request
  File "awscli/botocore/endpoint.py", line 135, in _send_request
  File "awscli/botocore/endpoint.py", line 167, in _get_response
  File "awscli/botocore/endpoint.py", line 221, in _do_get_response
  File "awscli/botocore/parsers.py", line 243, in parse
  File "awscli/botocore/parsers.py", line 610, in _do_error_parse
  File "awscli/botocore/parsers.py", line 540, in _do_error_parse
  File "awscli/botocore/parsers.py", line 489, in _parse_xml_string_to_dom
botocore.parsers.ResponseParserError: Unable to parse response (no element found: line 1, column 0), invalid XML received. Further retries may succeed:
b''

Unable to parse response (no element found: line 1, column 0), invalid XML received. Further retries may succeed:
b''

ERROR REPORT:
Original Error: *exec.ExitError exit status 255
Stack Trace:
    /tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/tool/tsh/aws.go:115 main.onAWS
    /tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:732 main.Run
    /tmp/build-darwin-amd64/go/src/github.com/gravitational/teleport/tool/tsh/tsh.go:310 main.main
    /var/folders/ys/8czjjsys38x504kj8172pd_m0000gp/T/drone-ChyAEOMKUxLFdO9I/home/drone/build-10400-1645573935-toolchains/go/src/runtime/proc.go:255 runtime.main
    /var/folders/ys/8czjjsys38x504kj8172pd_m0000gp/T/drone-ChyAEOMKUxLFdO9I/home/drone/build-10400-1645573935-toolchains/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: exit status 255
erozario commented 2 years ago

Same error

Server Details

GavinFrazar commented 1 year ago

When I added dynamodb to database-access I marshaled the error into JSON as the aws cli will expect it:

https://github.com/gravitational/teleport/blob/8144a7e843fcb7eb09f7771cddf57b54599d3f48/lib/srv/db/dynamodb/engine.go#L98-L141

Maybe we can fix this by doing the same for AWS app-access