Closed theronic closed 1 month ago
@theronic Are you using an in-memory datastore?
Please reopen if you can reproduce. If you're using an in-memory datastore and resuming the container, it likely won't function.
Another segfault on ReadRelationships when running SpiceDB v1.31.0 on CircleCI, when no schema exists:
8:31AM INF telemetry reporter scheduled endpoint=https://telemetry.authzed.com interval=1h0m0s next=5m3s
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=LookupResources grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:56Z grpc.time_ms=0 peer.address=127.0.0.1:46884 protocol=grpc requestID=cqu6ne3tgl0dnppd3rig
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rj0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rjg
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `account` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rk0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rkg
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rl0
8:32AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-14T08:32:58Z grpc.time_ms=0 peer.address=127.0.0.1:46900 protocol=grpc requestID=cqu6nejtgl0dnppd3rlg
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x13dc74d]
goroutine 283 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x3604fb0, 0xc0005acc80}, {0xc000e02088, 0x2, 0x3617300?}, {0x3613a70, 0xc000a4d110})
/home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:71 +0x34d
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).LookupResources(0xc001512a80, 0xc0001c1110, {0x3611720, 0xc000ea90f0})
/home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:381 +0x191
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_LookupResources_Handler({0x2bf0460, 0xc001512a80}, {0x360d868, 0xc000a4cfc0})
/home/runner/go/pkg/mod/github.com/authzed/authzed-go@v0.11.2-0.20240320204618-9622b72a72c6/proto/authzed/api/v1/permission_service_grpc.pb.go:407 +0x107
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.MustStreamServerInterceptor.func2({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cf60}, 0x2b5c620?, 0x2f32638)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/streamtimeout/streamtimeout.go:29 +0x136
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360ca68?, 0xc000a4cf60?})
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.StreamServerInterceptor.StreamServerInterceptor.func5({0x2bf0460, 0xc001512a80}, {0x360e4d0, 0xc000ea8fd0}, 0x2b32820?, 0xc000d19140)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360e4d0?, 0xc000ea8fd0?})
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/middleware/handwrittenvalidation.StreamServerInterceptor({0x2bf0460, 0xc001512a80}, {0x360e488, 0xc000b55698}, 0x2b963a0?, 0xc000d19160)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/handwrittenvalidation/handwrittenvalidation.go:33 +0x7a
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360e488?, 0xc000b55698?})
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/validator.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0x20?, 0xc000d19180)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/validator/interceptors.go:58 +0x9e
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3.1.1({0x2bf0460?, 0xc001512a80?}, {0x360cab0?, 0xc000d19120?})
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:47 +0x34
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainStreamServer.func3({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0x1?, 0x7f9db5fba8e8?)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/chain.go:56 +0xb8
github.com/authzed/spicedb/internal/middleware/servicespecific.StreamServerInterceptor({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120}, 0xc000b555f0, 0x2f32638)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/servicespecific/servicespecific.go:35 +0x83
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360cab0, 0xc000d19120})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func21({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100}, 0xc000b555f0, 0xc0009ecb40)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/consistency/consistency.go:205 +0x1a2
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func19({0x2bf0460, 0xc001512a80}, {0x360c990?, 0xc000d19100?}, 0xc000b555f0?, 0xc0009ecb00)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/datastore/datastore.go:83 +0xbd
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d19100})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func17({0x2bf0460, 0xc001512a80}, {0x360ca68?, 0xc000a4ce70?}, 0xc000b555f0?, 0xc0009ecac0)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/middleware/dispatcher/dispatcher.go:79 +0xbd
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4ce70})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.StreamServerInterceptor.func27({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d190e0}, 0xc000b555f0?, 0xc0009eca80)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360c990, 0xc000d190e0})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func14({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cde0}, 0xc000b555f0, 0xc0009eca40)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/auth/auth.go:68 +0x126
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cde0})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.createServerMetrics.(*ServerMetrics).StreamServerInterceptor.StreamServerInterceptor.func6({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cdb0}, 0xc000b555f0?, 0xc0009eca00)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cdb0})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors/logging.StreamServerInterceptor.StreamServerInterceptor.func2({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000e76941?, 0xc0009ec980)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func11({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000b555f0, 0xc0009ec980)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/selector/selector.go:50 +0x1c4
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.func7({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900}, 0xc000b555f0, 0xc0009ec940)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/selector/selector.go:52 +0x1fa
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360d5f0, 0xc0009ec900})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cc60}, 0xc000b555f0, 0xc0009ec880)
/home/runner/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.49.0/interceptor.go:446 +0x5bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cc60})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/cmd/server.DefaultStreamingMiddleware.StreamServerInterceptor.StreamServerInterceptor.func24({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cba0}, 0xc000b555f0?, 0xc0009ec840)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.getChainStreamHandler.func1({0x2bf0460, 0xc001512a80}, {0x360ca68, 0xc000a4cba0})
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1532 +0xb2
github.com/authzed/spicedb/pkg/middleware/requestid.StreamServerInterceptor.StreamServerInterceptor.func1({0x2bf0460, 0xc001512a80}, {0x360d4d0, 0xc000d5c4b0}, 0xc000b555f0?, 0xc0009ec800)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:35 +0x2bb
google.golang.org/grpc.NewServer.chainStreamServerInterceptors.chainStreamInterceptors.func2({0x2bf0460, 0xc001512a80}, {0x360d4d0, 0xc000d5c4b0}, 0xc000b555f0, 0xc000ea8aa0?)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1523 +0x85
google.golang.org/grpc.(*Server).processStreamingRPC(0xc00051f400, {0x3604f78, 0xc000a4c660}, {0x3615060, 0xc000e26000}, 0xc00086fe60, 0xc00097e0f0, 0x51643c0, 0x0)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1687 +0x11e7
google.golang.org/grpc.(*Server).handleStream(0xc00051f400, {0x3615060, 0xc000e26000}, 0xc00086fe60)
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1801 +0xe36
google.golang.org/grpc.(*Server).serveStreams.func2.1()
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1027 +0x8b
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 262
/home/runner/go/pkg/mod/google.golang.org/grpc@v1.62.1/server.go:1038 +0x125
Exited with code 2
@josephschorr using in-memory container for tests, but no container resumption in the latest segfault case.
Bumped SpiceDB version to 1.35.2 and same LookupResources
segfault on empty schema with in-memory data store. We can switch to Postgres but this is inconvenient for tests.
Have you guys considered using Sqlite as a backing store?
@theronic can you explain the use case further? Are you using spicedb serve
or spicedb serve-testing
when running the container? Is this for running integration tests? What kinds of assertions are the tests making?
I agree that a segfault isn't the best behavior. What behavior are you expecting in this case? Should it be erroring or are you expecting that the schema should be present at this point in your workflow?
@theronic We've tried to reproduce to no avail and, unless we're missing something, a nil
issue is not possible with the in-memory datastore, on the reported line of code in the stack trace.
At this point, we need exact reproduction instructions if possible.
@tstirrat15 running spicedb serve
using SpiceDB 1.34.0 via Docker:
docker run --rm -it \
--name spicedb \
-p 50051:50051 \
authzed/spicedb \
serve \
--grpc-preshared-key "spicedb"
The behaviour I expect is that ReadSchema or ReadRelationships should never crash, even if no schema exists. Should return empty string for schema or empty set of relationships on ReadRelationships when blank schema. Segfault concerning for us because it implies under certain conditions if we Read from Spice, it will bring down Spice in prod, but hopefully that won't happen when using Postgres (only memstore).
@josephschorr I am attempting to reproduce a minimal example for you. It seems to be related to ReadSchema
before any schema was written to in-memory store, followed by ReadRelationships
(args are just resource_type = "server"
) and possibly also DeleteRelationships
. I suspect there is some initial state that is null when no schema has been written, that is fixed after writing schema. This happens when running SpiceDB in Docker on macOS, not just on CircleCI:
11:52AM INF grpc server started serving addr=:50051 insecure=true network=tcp service=grpc workers=0
11:52AM INF running server datastore=*schemacaching.definitionCachingProxy
11:52AM INF telemetry reporter scheduled endpoint=https://telemetry.authzed.com interval=1h0m0s next=1m13s
11:52AM INF http server started serving addr=:9090 insecure=true service=metrics
11:52AM INF finished call grpc.code=NotFound grpc.component=server grpc.error="rpc error: code = NotFound desc = No schema has been defined; please call WriteSchema to start" grpc.method=ReadSchema grpc.method_type=unary grpc.service=authzed.api.v1.SchemaService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3km0
11:52AM INF finished call grpc.code=NotFound grpc.component=server grpc.error="rpc error: code = NotFound desc = No schema has been defined; please call WriteSchema to start" grpc.method=ReadSchema grpc.method_type=unary grpc.service=authzed.api.v1.SchemaService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3kmg
11:52AM WRN finished call grpc.code=FailedPrecondition grpc.component=server grpc.error="object definition `server` not found" grpc.method=ReadRelationships grpc.method_type=server_stream grpc.service=authzed.api.v1.PermissionsService grpc.start_time=2024-08-15T11:52:26Z grpc.time_ms=0 peer.address=192.168.65.1:47865 protocol=grpc requestID=cquunumaahes73cq3kn0
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x10c86c8]
goroutine 249 [running]:
github.com/authzed/spicedb/internal/namespace.CheckNamespaceAndRelations({0x33a0718, 0x40012e2750}, {0x4001041e90, 0x2, 0x4001041df8?}, {0x33b1fa8, 0x40012e27e0})
/home/runner/actions-runner/_work/spicedb/spicedb/internal/namespace/util.go:71 +0x288
github.com/authzed/spicedb/internal/services/v1.(*permissionServer).CheckPermission(0x4000b70850, {0x33a0718, 0x40012e2750}, 0x40009cb980)
/home/runner/actions-runner/_work/spicedb/spicedb/internal/services/v1/permissions.go:58 +0x154
github.com/authzed/authzed-go/proto/authzed/api/v1._PermissionsService_CheckPermission_Handler.func1({0x33a0718?, 0x40012e2750?}, {0x2a27f00?, 0x40009cb980?})
/home/runner/go/pkg/mod/github.com/authzed/authzed-go@v0.12.1-0.20240607163830-a28f71a1b0e5/proto/authzed/api/v1/permission_service_grpc.pb.go:375 +0xd0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.UnaryServerInterceptor.UnaryServerInterceptor.func4({0x33a0718, 0x40012e2660}, {0x2a27f00, 0x40009cb980}, 0x4001042101?, 0x40012bc660)
/home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/v2@v2.1.0/interceptors/server.go:22 +0x1f0
github.com/authzed/spicedb/internal/services/v1.NewPermissionsServer.ChainUnaryServer.func1.1.1({0x33a0718?, 0x40012e2660?}, {0x2a27f00?, 0x40009cb980?})
...
Note I am calling this via Clojure using gRPC libraries, but SpiceDB actually crashes in my Docker, so I don't believe this is related to any gRPC impl. bugs.
I'm not a Go guy, but possibly one of these error values causing it when attempting ReadRelationships before writing any schema.
These are guesses, but maybe r.txSource()
or tx.Lowerbound(tableNamespace, indexID)
before indexID = "id" is written to memstore (pre schema write).
I am confident there is a memstore bug when reading before writing schema.
Seems to be intermittent on my side, which hints at cache-related, or memory allocation related. I cannot consistently reproduce it locally, but can do it consistently on CircleCI on our prod stack, which I'm trying to reduce down to a minimal example.
What platforms are affected?
macos
What architectures are affected?
amd64
What SpiceDB version are you using?
v1.34.0
Steps to Reproduce
Docker image
sha256:5f81b3ea0702907c847e11cb20a6a58d60921811628da6776336ce2da7c60d68
.I can provide a Docker image and code if you send me an Image registry I can push to. I stopped a SpiceDB Docker container, and then restarted it with
docker start -a <container-id>
and attempted toReadRelationships
for some non-existent schema. TheFailedPrecondition
seems to be the cause of the segfault.Expected Result
Spice should not segfault. ReadRelationships should return an error.
Actual Result
Segmentation Fault. Logs: