AlibabaCloud S3-compatible storage does not work with Teleport

[webvictim]

Originally posted by @ZeroClover in https://github.com/gravitational/teleport/discussions/6491

I tried to configure S3 API-compatible storage for Teleport, but it doesn't work.

I can see in S3 file manager that Teleport has created the directory and uploaded the file in .tar format, but Teleport throws errors in the log.

Apr 16 19:06:49 hostname teleport[19753]: INFO [S3]        Upload created in 88.677244ms. s3sessions/s3stream.go:42
Apr 16 19:06:49 hostname teleport[19753]: INFO [S3]        UploadPart(A54DC4C8CD75435299374EAECAD8AEA1) part(1) uploaded in 13.520818ms. s3sessions/s3stream.go:63
Apr 16 19:06:49 hostname teleport[19753]: INFO [S3]        UploadPart(A54DC4C8CD75435299374EAECAD8AEA1) completed in 20.614992ms. s3sessions/s3stream.go:89
Apr 16 19:06:49 hostname teleport[19753]: INFO [AUDIT]     session.upload cluster_name:Teleport-HKG-Master code:T2005I ei:2.147483647e+09 event:session.upload sid:4366a95d-973c-4356-a393-3c5db0506ab9 time:2021-04-16T11:06:49.424Z url:s3://zerotp/4366a95d-973c-4356-a393-
Apr 16 19:06:49 hostname teleport[19753]: WARN [AUTH:GRPC] Failed to flush close the stream. error:[
Apr 16 19:06:49 hostname teleport[19753]: ERROR REPORT:
Apr 16 19:06:49 hostname teleport[19753]: Original Error: *trace.ConnectionProblemError emitter has been closed
Apr 16 19:06:49 hostname teleport[19753]: Stack Trace:
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/events/stream.go:422 github.com/gravitational/teleport/lib/events.(*ProtoStream).Close
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/events/emitter.go:453 github.com/gravitational/teleport/lib/events.(*CheckingStream).Close
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/auth_with_roles.go:1913 github.com/gravitational/teleport/lib/auth.(*streamWithRoles).Close
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/grpcserver.go:139 github.com/gravitational/teleport/lib/auth.(*GRPCServer).CreateAuditStream.func2
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/grpcserver.go:214 github.com/gravitational/teleport/lib/auth.(*GRPCServer).CreateAuditStream
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/github.com/gravitational/teleport/api/client/proto/authservice.pb.go:9114 github.com/gravitational/teleport/api/client/proto._AuthService_CreateAuditStream_Handler
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/lib/auth/middleware.go:386 github.com/gravitational/teleport/lib/auth.(*Middleware).StreamInterceptor
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/google.golang.org/grpc/server.go:1336 google.golang.org/grpc.(*Server).processStreamingRPC
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/google.golang.org/grpc/server.go:1409 google.golang.org/grpc.(*Server).handleStream
Apr 16 19:06:49 hostname teleport[19753]:         /go/src/github.com/gravitational/teleport/vendor/google.golang.org/grpc/server.go:746 google.golang.org/grpc.(*Server).serveStreams.func1.1
Apr 16 19:06:49 hostname teleport[19753]:         /opt/go/src/runtime/asm_amd64.s:1374 runtime.goexit
Apr 16 19:06:49 hostname teleport[19753]: User Message: emitter has been closed] auth/grpcserver.go:140

At the same time, I was unable to play back Session recordings in Teleport WebGUI, prompting "Recording for this session is not available."

I opened the browser console and found that when playing back the session, there was a 404 that went to https://<teleport url>/v1/webapi/sites/<cluster name>/sessions/4366a95d-973c-4356- a393-3c5db0506ab9/events .

[oanqa]

hello, i also have the same issue as this. right now i'm using teleport v13.4.9, from the oss console we can see the {sessions id}.tar but we cannot play it from teleport web interface

[webvictim]

What messages do you see in the Teleport auth server logs when you try to play back the session?

[oanqa]

hello, thank you for the response.

i've already solved this issue after changing the S3ForcePathStyle to false. see https://github.com/oanqa/teleport/commit/8ef11092689c672c112faaa7983bc45a26297004

from the alibaba oss documentation, the oss service doesn't support path style access.