Improve Session Remote Upload Error Handling

[travelton]

Description

In the event a remote storage session upload fails, Teleport will continue to retry uploading the session recording. In the case of Google Cloud Storage (GCS), an upload failure could be the result of a session having already exist at the remote storage target. This ultimately results in a continuous loop of attempting to upload the session.

What happened:

Teleport is in a continuous upload loop.

What you expected to happen:

In the event that a session object already exists on GCS, we should try to compare a hash of the object to ensure the local object is an exact copy on the remote storage target. If so, the local object may be deleted. If not, perhaps create a UUID conflict path on the remote storage and upload the object there?

How to reproduce it (as minimally and precisely as possible):

1. Configure a Teleport Cluster w/ GCS for session storage.
2. Start a session, but do not end the session.
3. Grab the session ID and create a blank object in your GCS bucket with the same session ID.
4. Complete the Teleport session.
5. Monitor the Teleport Auth server logs for the error.

Environment

• Teleport version (use teleport version): 4.3.6

• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): GCP

Relevant Debug Logs If Applicable

Here is an example stack trace:

WARN [AUDIT] "Session upload failed: 
ERROR REPORT:
Original Error: *trace.AlreadyExistsError recording for session \"96c1e47f-df2e-4b3b-9d2d-xxxxx\" already exists in GCS
Stack Trace:
    /go/src/github.com/gravitational/teleport/lib/events/gcssessions/gcshandler.go:221 github.com/gravitational/teleport/lib/events/gcssessions.(*Handler).Upload
    /go/src/github.com/gravitational/teleport/lib/events/auditlog.go:337 github.com/gravitational/teleport/lib/events.(*AuditLog).UploadSessionRecording
    /go/src/github.com/gravitational/teleport/e/lib/pro/auditlog.go:100 github.com/gravitational/teleport/e/lib/pro.(*AuditLog).UploadSessionRecording
    /go/src/github.com/gravitational/teleport/lib/auth/auth_with_roles.go:1386 github.com/gravitational/teleport/lib/auth.(*AuthWithRoles).UploadSessionRecording
    /go/src/github.com/gravitational/teleport/lib/auth/apiserver.go:1968 github.com/gravitational/teleport/lib/auth.(*APIServer).uploadSessionRecording
    /go/src/github.com/gravitational/teleport/lib/auth/apiserver.go:273 github.com/gravitational/teleport/lib/auth.(*APIServer).withAuth.func1
    /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:50 github.com/gravitational/teleport/lib/httplib.MakeHandler.func1
    /go/src/github.com/gravitational/teleport/vendor/github.com/julienschmidt/httprouter/router.go:300 github.com/gravitational/teleport/vendor/github.com/julienschmidt/httprouter.(*Router).ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:157 github.com/gravitational/teleport/lib/httplib.RewritePaths.func1
    /opt/go/src/net/http/server.go:2008 net/http.HandlerFunc.ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/auth/grpcserver.go:487 github.com/gravitational/teleport/lib/auth.(*GRPCServer).ServeHTTP
    /go/src/github.com/gravitational/teleport/lib/auth/middleware.go:348 github.com/gravitational/teleport/lib/auth.(*AuthMiddleware).ServeHTTP
    /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/ratelimit/tokenlimiter.go:119 github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
    /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/connlimit/connlimit.go:72 github.com/gravitational/teleport/vendor/github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
    /opt/go/src/net/http/server.go:2803 net/http.serverHandler.ServeHTTP
    /opt/go/src/net/http/server.go:1891 net/http.(*conn).serve
    /opt/go/src/runtime/asm_amd64.s:1358 runtime.goexit
User Message: recording for session \"96c1e47f-df2e-4b3b-9d2d-xxxxx\" already exists in GCS
" duration:16.761777ms session-id:96c1e47f-df2e-4b3b-9d2d-xxxxx events/auditlog.go:339

[zmb3]

Closing this, as the UploadSessionRecording code path that triggered the issue is no longer used.