search

authzed / spicedb

Open Source, Google Zanzibar-inspired permissions database to enable fine-grained authorization for customer applications
https://authzed.com/docs
Apache License 2.0
4.91k stars 264 forks source link

v1.34.0 Segfault on ReadRelationships before WriteSchema when using memstore #1985

Closed theronic closed 1 month ago

theronic commented 1 month ago

What platforms are affected?

macos

What architectures are affected?

amd64

What SpiceDB version are you using?

v1.34.0

Steps to Reproduce

Docker image sha256:5f81b3ea0702907c847e11cb20a6a58d60921811628da6776336ce2da7c60d68.

I can provide a Docker image and code if you send me an Image registry I can push to. I stopped a SpiceDB Docker container, and then restarted it with docker start -a <container-id> and attempted to ReadRelationships for some non-existent schema. The FailedPrecondition seems to be the cause of the segfault.

Expected Result

Spice should not segfault. ReadRelationships should return an error.

Actual Result

Segmentation Fault. Logs:

{"level":"warn","protocol":"grpc","grpc.component":"server","grpc.service":"authzed.api.v1.PermissionsService","grpc.method":"ReadRelationships","grpc.method_type":"server_stream","requestID":"edd969d0685a5dda472a2c43b57d41d0","peer.address":"192.168.65.1:60543","grpc.start_time":"2024-07-17T13:27:29Z","grpc.code":"FailedPrecondition","grpc.error":"object definition `team` not found","grpc.time_ms":0,"time":"2024-07-17T13:27:29Z","message":"finished call"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0xe55780]

goroutine 360 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x2db1268, 0x4000bec1b0}, {0x4000e7e238, 0x2, 0x40008bf1d0?}, {0x2dbdaa0, 0x4000bec240})
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:72 +0x2c0
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).CheckPermission(0x40008ca870, {0x2db1268, 0x4000bec1b0}, 0x4000c7d260)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:53 +0x154
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_CheckPermission_Handler.func1({0x2db1268, 0x4000bec1b0}, {0x25cd1c0?, 0x4000c7d260})
    /home/runner/go/pkg/mod/github.com/authzed/authzed-go@v0.10.1/proto/authzed/api/v1/permission_service_grpc.pb.go:342 +0x74
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.UnaryServerInterceptor.UnaryServerInterceptor.func4({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x4000e7e601?, 0x4000c9a8e8)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/interceptors/server.go:22 +0x1f0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x2db1268?, 0x4000bec0c0?}, {0x25cd1c0?, 0x4000c7d260?})
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:23 +0x40
github.com/authzed/spicedb/internal/middleware/handwrittenvalidation.UnaryServerInterceptor({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0xffff5aa3fca8?, 0x4000754dc0)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/handwrittenvalidation/handwrittenvalidation.go:26 +0xc0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x2db1268?, 0x4000bec0c0?}, {0x25cd1c0?, 0x4000c7d260?})
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:23 +0x40
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/validator.UnaryServerInterceptor.func1({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x30?, 0x4000754e00)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/interceptors/validator/interceptors.go:23 +0x7c
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x2db1268?, 0x4000bec0c0?}, {0x25cd1c0?, 0x4000c7d260?})
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:23 +0x40
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x4000e7e7f8?, 0x12c3528?)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:32 +0xb8
github.com/authzed/spicedb/internal/middleware/servicespecific.UnaryServerInterceptor({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000c9a8e8)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/servicespecific/servicespecific.go:25 +0x84
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000bec0c0}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func15({0x2db1268, 0x4000bec090}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000b518c0)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/consistency/consistency.go:191 +0x138
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000bec090}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func13({0x2db1268?, 0x4000bec060?}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51880)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/datastore/datastore.go:70 +0x90
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000bec060}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func11({0x2db1268?, 0x4000969fb0?}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51840)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/dispatcher/dispatcher.go:66 +0x90
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.UnaryServerInterceptor.func20({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51800)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.func8({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000b517c0)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/interceptors/auth/auth.go:48 +0xbc
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/grpc-ecosystem/go-grpc-prometheus.init.(*ServerMetrics).UnaryServerInterceptor.func3({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51780)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:107 +0x74
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969fb0}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x2db1268, 0x4000969d10}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20, 0x4000b51700)
    /home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.47.0/interceptor.go:326 +0x3f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969d10}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.UnaryServerInterceptor.UnaryServerInterceptor.func2({0x2db1268, 0x4000969ce0}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b516c0)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969ce0}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/cmd/server.DefaultUnaryMiddleware.UnaryServerInterceptor.UnaryServerInterceptor.func18({0x2db1268, 0x4000969800}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51680)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.getChainUnaryHandler.func1({0x2db1268, 0x4000969800}, {0x25cd1c0, 0x4000c7d260})
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1205 +0xa0
github.com/authzed/spicedb/pkg/middleware/requestid.UnaryServerInterceptor.UnaryServerInterceptor.func1({0x2db1268, 0x4000969770}, {0x25cd1c0, 0x4000c7d260}, 0x4000754c20?, 0x4000b51640)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.0.1/interceptors/server.go:22 +0x1f0
google.golang.org/grpc.NewServer.chainUnaryServerInterceptors.chainUnaryInterceptors.func1({0x2db1268, 0x4000969770}, {0x25cd1c0, 0x4000c7d260}, 0x4001303988?, 0x536b98?)
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1196 +0x88
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_CheckPermission_Handler({0x24f9e40?, 0x40008ca870}, {0x2db1268, 0x4000969770}, 0x4000b4eb00, 0x4000754940)
    /home/runner/go/pkg/mod/github.com/authzed/authzed-go@v0.10.1/proto/authzed/api/v1/permission_service_grpc.pb.go:344 +0x12c
google.golang.org/grpc.(*Server).processUnaryRPC(0x40005dc800, {0x2db1268, 0x40009696e0}, {0x2dbf358, 0x4001350000}, 0x40012406c0, 0x4000f16f30, 0x47d97d0, 0x0)
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1385 +0xb8c
google.golang.org/grpc.(*Server).handleStream(0x40005dc800, {0x2dbf358, 0x4001350000}, 0x40012406c0)
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1796 +0xc4c
google.golang.org/grpc.(*Server).serveStreams.func2.1()
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1029 +0x8c
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 261
    /home/runner/go/pkg/mod/google.golang.org/grpc@v1.61.0/server.go:1040 +0x150
josephschorr commented 1 month ago

@theronic Are you using an in-memory datastore?

josephschorr commented 1 month ago

Please reopen if you can reproduce. If you're using an in-memory datastore and resuming the container, it likely won't function.

theronic commented 2 weeks ago

Another segfault on ReadRelationships when running SpiceDB v1.31.0 on CircleCI, when no schema exists:

8:31AM INF telemetry reporter scheduled endpoint=https://telemetry.authzed.com interval=1h0m0s next=5m3s
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=LookupResources grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:56Z grpc.time_ms=0 peer.address=127.0.0.1:46884 protocol=grpc requestID=cqu6ne3tgl0dnppd3rig
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rj0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rjg
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rk0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rkg
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rl0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rlg
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x13dc74d]

goroutine 283 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x3604fb0, 0xc0005acc80}, {0xc000e02088, 0x2, 0x3617300?}, {0x3613a70, 0xc000a4d110})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:71 +0x34d
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).LookupResources(0xc001512a80, 0xc0001c1110, {0x3611720, 0xc000ea90f0})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:381 +0x191
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_LookupResources_Handler({0x2bf0460, 0xc001512a80}, {0x360d868, 0xc000a4cfc0})
        /home/runner/go/pkg/mod/github.com/authzed/authzed-go@v0.11.2-0.20240320204618-9622b72a72c6/proto/authzed/api/v1/permission_service_grpc.pb.go:407 +0x107
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.MustStreamServerInterceptor.func2({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cf60}, 0x2b5c620?, 0x2f32638)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/streamtimeout/streamtimeout.go:29 +0x136
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360ca68?, 0xc000a4cf60?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.StreamServerInterceptor.StreamServerInterceptor.func5({0x2bf0460, 0xc001512a80}, {0x360e4d0, 0xc000ea8fd0}, 0x2b32820?, 0xc000d19140)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360e4d0?, 0xc000ea8fd0?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/middleware/handwrittenvalidation.StreamServerInterceptor({0x2bf0460, 0xc001512a80}, {0x360e488, 0xc000b55698}, 0x2b963a0?, 0xc000d19160)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/handwrittenvalidation/handwrittenvalidation.go:33 +0x7a
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360e488?, 0xc000b55698?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/validator.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0x20?, 0xc000d19180)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/validator/interceptors.go:58 +0x9e
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360cab0?, 0xc000d19120?})
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0x1?, 0x7f9db5fba8e8?)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:56 +0xb8
github.com/authzed/spicedb/internal/middleware/servicespecific.StreamServerInterceptor({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0xc000b555f0, 0x2f32638)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/servicespecific/servicespecific.go:35 +0x83
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func21({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100}, 0xc000b555f0, 0xc0009ecb40)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/consistency/consistency.go:205 +0x1a2
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func19({0x2bf0460, 0xc001512a80}, {0x360c990?, 0xc000d19100?}, 0xc000b555f0?, 0xc0009ecb00)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/datastore/datastore.go:83 +0xbd
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func17({0x2bf0460, 0xc001512a80}, {0x360ca68?, 0xc000a4ce70?}, 0xc000b555f0?, 0xc0009ecac0)
        /home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/dispatcher/dispatcher.go:79 +0xbd
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4ce70})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.StreamServerInterceptor.func27({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d190e0}, 0xc000b555f0?, 0xc0009eca80)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d190e0})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func14({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cde0}, 0xc000b555f0, 0xc0009eca40)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/auth/auth.go:68 +0x126
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cde0})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.createServerMetrics.(*ServerMetrics).StreamServerInterceptor.StreamServerInterceptor.func6({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cdb0}, 0xc000b555f0?, 0xc0009eca00)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cdb0})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.StreamServerInterceptor.StreamServerInterceptor.func2({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000e76941?, 0xc0009ec980)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func11({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000b555f0, 0xc0009ec980)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/selector/selector.go:50 +0x1c4
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func7({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000b555f0, 0xc0009ec940)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/selector/selector.go:52 +0x1fa
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cc60}, 0xc000b555f0, 0xc0009ec880)
        /home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.49.0/interceptor.go:446 +0x5bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cc60})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.StreamServerInterceptor.func24({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cba0}, 0xc000b555f0?, 0xc0009ec840)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cba0})
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/middleware/requestid.StreamServerInterceptor.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360d4d0, 0xc000d5c4b0}, 0xc000b555f0?, 0xc0009ec800)
        /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x2bf0460, 0xc001512a80}, {0x360d4d0, 0xc000d5c4b0}, 0xc000b555f0, 0xc000ea8aa0?)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1523 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00051f400, {0x3604f78, 0xc000a4c660}, {0x3615060, 0xc000e26000}, 0xc00086fe60, 0xc00097e0f0, 0x51643c0, 0x0)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1687 +0x11e7
google.golang.org/grpc.(*Server).handleStream(0xc00051f400, {0x3615060, 0xc000e26000}, 0xc00086fe60)
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1801 +0xe36
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 262
        /home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1038 +0x125

Exited with code 2
theronic commented 2 weeks ago

@josephschorr using in-memory container for tests, but no container resumption in the latest segfault case.

theronic commented 2 weeks ago

Bumped SpiceDB version to 1.35.2 and same LookupResources segfault on empty schema with in-memory data store. We can switch to Postgres but this is inconvenient for tests.

Have you guys considered using Sqlite as a backing store?

tstirrat15 commented 2 weeks ago

@theronic can you explain the use case further? Are you using spicedb serve or spicedb serve-testing when running the container? Is this for running integration tests? What kinds of assertions are the tests making?

I agree that a segfault isn't the best behavior. What behavior are you expecting in this case? Should it be erroring or are you expecting that the schema should be present at this point in your workflow?

josephschorr commented 2 weeks ago

@theronic We've tried to reproduce to no avail and, unless we're missing something, a nil issue is not possible with the in-memory datastore, on the reported line of code in the stack trace.

At this point, we need exact reproduction instructions if possible.

theronic commented 2 weeks ago

@tstirrat15 running spicedb serve using SpiceDB 1.34.0 via Docker:

 docker run --rm -it \
    --name spicedb \
    -p 50051:50051 \
    authzed/spicedb \
    serve \
    --grpc-preshared-key "spicedb"

The behaviour I expect is that ReadSchema or ReadRelationships should never crash, even if no schema exists. Should return empty string for schema or empty set of relationships on ReadRelationships when blank schema. Segfault concerning for us because it implies under certain conditions if we Read from Spice, it will bring down Spice in prod, but hopefully that won't happen when using Postgres (only memstore).

@josephschorr I am attempting to reproduce a minimal example for you. It seems to be related to ReadSchema before any schema was written to in-memory store, followed by ReadRelationships (args are just resource_type = "server") and possibly also DeleteRelationships. I suspect there is some initial state that is null when no schema has been written, that is fixed after writing schema. This happens when running SpiceDB in Docker on macOS, not just on CircleCI:

11:52AM INF grpc server started serving addr=:50051 insecure=true network=tcp service=grpc workers=0
11:52AM INF running server datastore=*schemacaching.definitionCachingProxy
11:52AM INF telemetry reporter scheduled endpoint=https://telemetry.authzed.com interval=1h0m0s next=1m13s
11:52AM INF http server started serving addr=:9090 insecure=true service=metrics
11:52AM INF finished call grpc.code=NotFound grpc.component=server grpc.error="rpc error: code = NotFound desc = No schema has been defined; please call WriteSchema to start" grpc.method=ReadSchema grpc.method_type=unary grpc.service=authzed.api.v1.SchemaService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3km0
11:52AM INF finished call grpc.code=NotFound grpc.component=server grpc.error="rpc error: code = NotFound desc = No schema has been defined; please call WriteSchema to start" grpc.method=ReadSchema grpc.method_type=unary grpc.service=authzed.api.v1.SchemaService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3kmg
11:52AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3kn0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x10c86c8]

goroutine 249 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x33a0718, 0x40012e2750}, {0x4001041e90, 0x2, 0x4001041df8?}, {0x33b1fa8, 0x40012e27e0})
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:71 +0x288
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).CheckPermission(0x4000b70850, {0x33a0718, 0x40012e2750}, 0x40009cb980)
    /home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:58 +0x154
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_CheckPermission_Handler.func1({0x33a0718?, 0x40012e2750?}, {0x2a27f00?, 0x40009cb980?})
    /home/runner/go/pkg/mod/github.com/authzed/authzed-go@v0.12.1-0.20240607163830-a28f71a1b0e5/proto/authzed/api/v1/permission_service_grpc.pb.go:375 +0xd0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.UnaryServerInterceptor.UnaryServerInterceptor.func4({0x33a0718, 0x40012e2660}, {0x2a27f00, 0x40009cb980}, 0x4001042101?, 0x40012bc660)
    /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:22 +0x1f0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x33a0718?, 0x40012e2660?}, {0x2a27f00?, 0x40009cb980?})
...

Note I am calling this via Clojure using gRPC libraries, but SpiceDB actually crashes in my Docker, so I don't believe this is related to any gRPC impl. bugs.

theronic commented 2 weeks ago

I'm not a Go guy, but possibly one of these error values causing it when attempting ReadRelationships before writing any schema.

These are guesses, but maybe r.txSource() or tx.Lowerbound(tableNamespace, indexID) before indexID = "id" is written to memstore (pre schema write).

I am confident there is a memstore bug when reading before writing schema.

Seems to be intermittent on my side, which hints at cache-related, or memory allocation related. I cannot consistently reproduce it locally, but can do it consistently on CircleCI on our prod stack, which I'm trying to reduce down to a minimal example.