gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
16.97k stars 1.7k forks source link

Make fips teleport log connection errors more clearly #7697

Open programmerq opened 2 years ago

programmerq commented 2 years ago

Description

What happened:

When attempting a login via OIDC, the auth server logs an error including this stacktrace:

Original Error: *trace.ConnectionProblemError failed to reach out to oidc connector keycloak, most likely URL "https://keycloak.example.com/auth/realms/teleport" is not valid or not accessible, check configuration and try to re-create the connector
Stack Trace:
    /go/src/github.com/gravitational/teleport/lib/auth/oidc.go:98 github.com/gravitational/teleport/lib/auth.(*Server).createOIDCClient
    /go/src/github.com/gravitational/teleport/lib/auth/oidc.go:50 github.com/gravitational/teleport/lib/auth.(*Server).getOrCreateOIDCClient
    /go/src/github.com/gravitational/teleport/lib/auth/oidc.go:182 github.com/gravitational/teleport/lib/auth.(*Server).CreateOIDCAuthRequest
    /go/src/github.com/gravitational/teleport/lib/auth/auth_with_roles.go:1667 github.com/gravitational/teleport/lib/auth.(*ServerWithRoles).CreateOIDCAuthRequest
    /go/src/github.com/gravitational/teleport/lib/auth/apiserver.go:1360 github.com/gravitational/teleport/lib/auth.(*APIServer).createOIDCAuthRequest
    /go/src/github.com/gravitational/teleport/lib/auth/apiserver.go:298 github.com/gravitational/teleport/lib/auth.(*APIServer).withAuth.func1
    /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:61 github.com/gravitational/teleport/lib/httplib.MakeHandlerWithErrorWriter.func1
    /go/src/github.com/gravitational/teleport/vendor/github.com/julienschmidt/httprouter/router.go:387 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:173 github.com/gravitational/teleport/lib/httplib.RewritePaths.func1
    /opt/go/src/net/http/server.go:2042 net/http.HandlerFunc.ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/auth/middleware.go:541 github.com/gravitational/teleport/lib/auth.(*Middleware).ServeHTTP
    /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
    /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
    /opt/go/src/net/http/server.go:2843 net/http.serverHandler.ServeHTTP
    /opt/go/src/net/http/server.go:1925 net/http.(*conn).serve
    /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
Caught:
    /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:131 github.com/gravitational/teleport/lib/httplib.ConvertResponse
    /go/src/github.com/gravitational/teleport/lib/auth/clt.go:295 github.com/gravitational/teleport/lib/auth.(*Client).PostJSON
    /go/src/github.com/gravitational/teleport/lib/auth/clt.go:1164 github.com/gravitational/teleport/lib/auth.(*Client).CreateOIDCAuthRequest
    /go/src/github.com/gravitational/teleport/lib/web/apiserver.go:886 github.com/gravitational/teleport/lib/web.(*Handler).oidcLoginWeb
    /go/src/github.com/gravitational/teleport/lib/web/apiserver.go:2309 github.com/gravitational/teleport/lib/web.(*Handler).WithRedirect.func1
    /go/src/github.com/gravitational/teleport/vendor/github.com/julienschmidt/httprouter/router.go:387 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
    /opt/go/src/net/http/server.go:2081 net/http.StripPrefix.func1
    /opt/go/src/net/http/server.go:2042 net/http.HandlerFunc.ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/web/apiserver.go:394 github.com/gravitational/teleport/lib/web.NewHandler.func1
    /opt/go/src/net/http/server.go:2042 net/http.HandlerFunc.ServeHTTP
    /go/src/github.com/gravitational/teleport/vendor/github.com/julienschmidt/httprouter/router.go:448 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:173 github.com/gravitational/teleport/lib/httplib.RewritePaths.func1
    /opt/go/src/net/http/server.go:2042 net/http.HandlerFunc.ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/web/apiserver.go:196 github.com/gravitational/teleport/lib/web.(*RewritingHandler).ServeHTTP
    /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
    /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
    /opt/go/src/net/http/server.go:2843 net/http.serverHandler.ServeHTTP
    /opt/go/src/net/http/server.go:1925 net/http.(*conn).serve
    /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
User Message: failed to reach out to oidc connector keycloak, most likely URL "https://keycloak.example.com/auth/realms/teleport" is not valid or not accessible, check configuration and try to re-create the connector
] web/apiserver.go:895

The keycloak instance has a well trusted certificate and is up and running.

This is a teleport-fips binary. A network stacktrace showed that the TLS-level error was that teleport was disconnecting because it did not trust the 4096 bit certificate since it was using a cipher not allowed by fips.

What you expected to happen:

At the very least, the fips connection failure reason should be made clear in the logs. The "is not valid or not accessible" is too general and is confusing in conjunction with the endpoint referenced being up and running and accepting other requests and working with other tools on the system such as curl.

Additionally, it may be appropriate to relax the stringent fips requirements when communicating with upstream services especially since the teleport admin may not have control over those.

Reproduction Steps

With a teleport fips binary, create an oidc connector that will cause teleport to try to make a TLS connection. Use a 4096 bit cert on that oidc server.

Server Details

Client Details

pschisa commented 2 years ago

Same general problem with running FIPS Teleport binaries with etcd. If the ETCD certs are invalid for FIPS, this was the only error produced in Teleport:

sudo teleport start
2022-03-15T07:26:26-06:00 INFO             Using license from /var/lib/teleport/license.pem expires at 2122-01-03 17:59:53.371934139 +0000 UTC,supports application access. process/process.go:64
2022-03-15T07:26:26-06:00 DEBU [SQLITE]    Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:172
2022-03-15T07:26:26-06:00 DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:217
2022-03-15T07:26:26-06:00 DEBU [KEYGEN]    SSH cert authority is going to pre-compute 25 keys. native/native.go:102
2022-03-15T07:26:26-06:00 DEBU [PROC:1]    Using etcd backend. service/service.go:3731
ERROR REPORT:
Original Error: context.deadlineExceededError context deadline exceeded
Stack Trace:
    /go/src/github.com/gravitational/teleport/lib/backend/etcdbk/etcd.go:359 [github.com/gravitational/teleport/lib/backend/etcdbk.(*EtcdBackend).reconnect](http://github.com/gravitational/teleport/lib/backend/etcdbk.(*EtcdBackend).reconnect)
    /go/src/github.com/gravitational/teleport/lib/backend/etcdbk/etcd.go:227 [github.com/gravitational/teleport/lib/backend/etcdbk.New](http://github.com/gravitational/teleport/lib/backend/etcdbk.New)
    /go/src/github.com/gravitational/teleport/lib/service/service.go:3744 [github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthStorage](http://github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthStorage)
    /go/src/github.com/gravitational/teleport/lib/service/service.go:1108 [github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthService](http://github.com/gravitational/teleport/lib/service.(*TeleportProcess).initAuthService)
    /go/src/github.com/gravitational/teleport/lib/service/service.go:779 [github.com/gravitational/teleport/lib/service.NewTeleport](http://github.com/gravitational/teleport/lib/service.NewTeleport)
    /go/src/github.com/gravitational/teleport/e/tool/teleport/process/process.go:67 [github.com/gravitational/teleport/e/tool/teleport/process.NewTeleport](http://github.com/gravitational/teleport/e/tool/teleport/process.NewTeleport)
    /go/src/github.com/gravitational/teleport/lib/service/service.go:503 [github.com/gravitational/teleport/lib/service.Run](http://github.com/gravitational/teleport/lib/service.Run)
    /go/src/github.com/gravitational/teleport/e/tool/teleport/main.go:23 main.main
    /opt/go/src/runtime/proc.go:255 runtime.main
    /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit
User Message: initialization failed
    context deadline exceeded

And on etcd side

Caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"<redacted>","server-name":"","error":"remote error: tls: bad certificate"}

This can lead to a lot of troubleshooting around how the TLS certificates and networking are set up between Teleport and etcd instead of focusing on a FIPS compliance failure in Teleport.

pschisa commented 2 years ago

For those interested, root of the above issue was: https://github.com/gravitational/teleport/issues/3519