Embed SpiceDB as a library inside services written in Go

[stanislavprokopov]

Is it possible to embed SpiceDB as a library inside a service written in Go to handle permissions? If not then would you consider refactoring the code so that this use case would also be possible?

[jonwhitty]

One of the large concepts behind Google's Zanzibar design is that you distribute the query space or access evaluation across multiple nodes when evaluating an access control decision. This enables extremely large scalability and does not suffer many drawbacks because of the distributed caching that is involved in evaluating access control decisions across nodes. Consequently, a server driven design is more appropriate for this type of architecture because of how a spicedb cluster works to solve the scalability problem. To implement spicedb as a library would eliminate many of the design goals of Zanzibar and spicedb - one of which being extreme scalability.

Take a look at these source files for more insight:

https://github.com/authzed/spicedb/blob/main/internal/dispatch/remote/cluster.go https://github.com/authzed/spicedb/blob/main/internal/dispatch/client/consistentbackend/client.go

Also, Section 3 (Architecture and Implementation) of the Zanzibar paper is a useful read.

[josephschorr]

@stanislavprokopov As @jonwhitty mentioned, embedding as a library does lose a significant amount of value. Would you be able to expand upon your use case for SpiceDB as a library? Would this only be used for a small-scale system?

[stanislavprokopov]

I need to handle backoffice permissions for business client, there is no need in distribution or multi node SpiceDB setup, and there is no point to spin up a separate server to handle this, so thats why i would like to use a library. I have not looked at the code, but I imagine the actual engine that performs permission check against schema and relations surely does not know anything about distributed queries and multiple nodes, it simply takes the relation graph and checks if the user has required permission.

[josephschorr]

@stanislavprokopov Yeah, it is sort of supported already. In fact, the SpiceDB's developer API basically uses SpiceDB as a library, since it sets up and tears down the entire context per request. Take a look in https://github.com/authzed/spicedb/blob/main/internal/services/v0/devcontext.go, and if you encounter any functions or types you'd like to have exported to make using SpiceDB as a library easier, we can discuss those changes.

[jzelinskie]

@josephschorr Most of those libraries are internal, and thus unable to imported into other Go programs. This is done intentionally as we would like to control exactly what is stable for other programs to rely on from SpiceDB (those can be found in the pkg/ directory).

[josephschorr]

@jzelinskie Yep, hence why the idea to discuss to the changes necessary to externalize to support the use cases, once we have a better idea of them

[stanislavprokopov]

So tried to use the internal packages to see how they could be used if they were exposed. Tried to setup the code according to the "Basic RBAC Example" in the playground (simplified it a bit), but i am getting NOT_MEMBER result, not sure whats wrong in my setup, here is the code:

...
devcontext, ok, err := v0.NewDevContext(context.Background(), &apiv0.RequestContext{
        Schema: `
            definition user {}
            definition document {
                relation writer: user
                relation reader: user
                permission edit = writer
                permission view = reader + edit
            }
        `,
        Relationships: []*apiv0.RelationTuple{
            {
                ObjectAndRelation: &apiv0.ObjectAndRelation{
                    Namespace: "document",
                    ObjectId:  "doc-1",
                    Relation:  "reader",
                },
                User: &apiv0.User{
                    UserOneof: &apiv0.User_Userset{
                        Userset: &apiv0.ObjectAndRelation{
                            Namespace: "user",
                            ObjectId:  "tom",
                            Relation:  "...",
                        },
                    },
                },
            },
        },
    })

...

check, err := devcontext.Dispatcher.DispatchCheck(context.Background(), &v1.DispatchCheckRequest{
        Metadata: &dispatch.ResolverMeta{
            AtRevision:     "1.0",
            DepthRemaining: 10,
        },
        ObjectAndRelation: &apiv0.ObjectAndRelation{
            Namespace: "document",
            ObjectId:  "doc-1",
            Relation:  "view",
        },
        Subject: &apiv0.ObjectAndRelation{
            Namespace: "user",
            ObjectId:  "tom",
            Relation:  "...",
        },
    })

[josephschorr]

@stanislavprokopov You're not giving the correct revision (AtRevision) to the Check, so it is using a cache which wasn't yet updated.

I should add I only pointed to the developer API as an example: I wouldn't recommend long term that dispatch nor the devcontext be made usable outside of the binary. Rather, we'd likely expose an explicit library-able interface.

[arashpayan]

I'd like to add another example for the embedded use case. I completely understand that the raison d'être of Zanzibar is that it's distributed+scalable+reliable, but whether intended or not, Zanzibar (and SpiceDB in particular) is a really simple system for modelling permissions and querying them. And if there's even a modest amount of complexity in the permissions, which always creeps into the scope of projects, a Zanzibar based model can be easily augmented to support it. It's much simpler than writing a custom permissions system with all the necessary backend code to support it.

You're probably asking why the distributed and scalable attributes don't matter as much. It's because (as @stanislavprokopov pointed out) these are for applications that will never run on anything more than one server. They're just uninteresting (though critical) business applications that only serve a couple users at a time simultaneously with a max of 100 users registered on the server. I'm talking on the order of 2 queries a minute. Not exactly Google scale. :stuck_out_tongue:

In the environments above, simplicity of deployment and maintenance surpasses any thoughts of scalability. A static Go binary with Spice DB embedded would be incredible.

Of course, I realize that you're trying to launch a profitable company, and this particular use case doesn't lend itself to monetization. The only comparable example I know of would be the way SQLite makes money. https://www.sqlite.org/prosupport.html

Anyway, please don't interpret this comment as a pushy demand/request for support on open source software. I'm just an enthusiastic user sharing a suggestion. Even if this never happens, I'll still be using it for any project of mine that requires anything more than 'admin/not admin' style permissions. :slightly_smiling_face:

[jzelinskie]

I don't think that restricting SpiceDB to only being a networked service is at odds with the Authzed business model -- the more people using the software Authzed builds, the more likely they'll be able to derive value from the paid products.

That being said, I think the biggest reason that we've kept these libraries internal is that we're still breaking APIs regularly as we discover the right abstractions. As we have the need for code sharing in tools outside of SpiceDB, we've been slowly moving things out of internal and into the pkg directory.

One of recent example where we've opened things up is the CLI interface which can be found in pkg/cmd. I think that today you could embed SpiceDB in a binary by using this package to construct a gRPC server then using a gRPC client in the same process to communicate over a unix socket/shared memory/buffered memory. This might not be the ideal UX, but it works today.

I'm willing to spend the rest of this issue discussing the actual desired UX for a library meant to embed SpiceDB. I don't think we need additional convincing that it'd be valuable.

[josephschorr]

If anyone is still interested in this, a lot of work has been done to run SpiceDB directly via WASM, which has made it much more embeddable. Happy to discuss here

[arashpayan]

I'm still very interested in this functionality.

Here's a very rough interface that I think I might be useful. Please read it with a skeptical eye, because I don't have much experience with the internals of SpiceDB and I'm positive that I'm overlooking functionality other folks would want from an embedded SpiceDB.

package spicedb

// The minimum required attributes to start an instance are in the configuration
type Config struct {
    DBURI string
    // one of cockroachdb, postgres, etc.
    DBEngine DBEngineType
    // so the embedding application can control stdout and stderr (maybe this should be in Options?)
    Logger Logger
}

// SpiceDB has tons of options in the CLI, and I'm sure some subset would make sense here, but I'm not confident enough to make any suggestions here.
type Options struct {

}

type Server interface {
    // just returns an instance of an *authzed.Client that we're already familiar with. Though if some other interface is easier to expose for the sake of this implementation, I'd be fine with that too.
    NewClient() *authzed.Client // no token required
    Shutdown(context.Context) error
}

func Start(cfg Config, opts *Options) (Server, error) {
    // implementation
    // - connects to the database
    // - migrates to HEAD everytime
    // - other stuff?
}

[vroldanbet]

@arashpayan I think it should be possible to embed a SpiceDB server into another process. Have you tried that out and concluded it wasn't possible?

[arashpayan]

@vroldanbet No, I haven't even tried it. If any of the SpiceDB folks think that has a chance of success I can take a look at it in April, but my understanding from the above discussion was that there are still internal types that need to be exported. Granted, things may have changed since the WASM changes landed.

[vroldanbet]

@arashpayan I suspect that's the case: I've recently implemented an application that runs a SpiceDB with MemDB embedded - I think it should be possible. Feel free to report anything you might find here!

[arashpayan]

@vroldanbet Thanks. I'll give it a try in April and report back what I find. Do you have any suggestions/tips/code from your application that you can share here to aid in my work?

[vroldanbet]

Yeah, I think you could take a look at the testserver package, used to spin spicedb instances in integration tests. Most of the code used there is in pkg, and for the one that isn't, you should be able to find an exported alternative:

https://github.com/authzed/spicedb/blob/0c25ac283996945af91da101d4865920c371b67c/internal/testserver/server.go#L51-L116

[josephschorr]

@stanislavprokopov @arashpayan Any further guidance needed on this?

[stanislavprokopov]

Well based on the example provided its clearly possible to customize parts of the spicedb server and use custom storage or customise the handlers and some other parts of the server, but based on the current codebase it doesnt look like its still possible to use spicedb as a library to handle permissions checks only, without all the other unrelated stuff, because the critical parts that actually perform these checks are still located in the internal package, correct me if I am wrong.

What I would like to have is a way to do smth like this:

someDataStore := db.NewDatastore()
permissions := spicedb.NewPermissionsService(someDataStore)

if !permissions.Check(CheckParameters, resourceID) {
    // Return 401
}

spicedb is great if you were to build a large project, then surely spinning up one more service for permission handling is ok, but for smth smaller, thats a clear overkill and there is nothing out there that can provide the same fine-grained permissions check.

[ecordell]

Generally you'll want to use the objects in pkg/cmd to configure an in-process SpiceDB; these are the public interfaces to the internal code.

Here's an example of how to build an interface like you're asking for @stanislavprokopov (note that no internal packages are used):

package main

import (
    "context"
    "fmt"
    "time"

    v1 "github.com/authzed/authzed-go/proto/authzed/api/v1"
    "google.golang.org/grpc"

    "github.com/authzed/spicedb/pkg/cmd/datastore"
    "github.com/authzed/spicedb/pkg/cmd/server"
    "github.com/authzed/spicedb/pkg/cmd/util"
)

type PermissionService struct {
    v1.PermissionsServiceClient
    Schema v1.SchemaServiceClient
    close  func()
}

func NewPermissionsService(ds datastore.Config) (*PermissionService, error) {
    ctx, cancel := context.WithCancel(context.Background())
    srv, err := server.NewConfigWithOptions(
        server.WithDatastoreConfig(ds),
        server.WithDispatchMaxDepth(50),
        server.WithMaximumPreconditionCount(10),
        server.WithMaximumUpdatesPerWrite(1000),
        server.WithStreamingAPITimeout(30*time.Second),
        server.WithMaxCaveatContextSize(4096),
        server.WithMaxRelationshipContextSize(1024),
        server.WithGRPCServer(util.GRPCServerConfig{
            Network: util.BufferedNetwork,
            Enabled: true,
        }),
        server.WithSchemaPrefixesRequired(false),
        server.WithGRPCAuthFunc(func(ctx context.Context) (context.Context, error) {
            return ctx, nil
        }),
        server.WithHTTPGateway(util.HTTPServerConfig{HTTPEnabled: false}),
        server.WithDashboardAPI(util.HTTPServerConfig{HTTPEnabled: false}),
        server.WithMetricsAPI(util.HTTPServerConfig{HTTPEnabled: false}),
        server.WithDispatchServer(util.GRPCServerConfig{Enabled: false}),
    ).Complete(ctx)
    if err != nil {
        cancel()
        return nil, err
    }

    go func() {
        if err := srv.Run(ctx); err != nil {
            fmt.Println(err)
        }
    }()

    conn, err := srv.GRPCDialContext(ctx, grpc.WithBlock())
    if err != nil {
        cancel()
        return nil, err
    }

    return &PermissionService{
        PermissionsServiceClient: v1.NewPermissionsServiceClient(conn),
        Schema:                   v1.NewSchemaServiceClient(conn),
        close:                    cancel,
    }, nil
}

func (p *PermissionService) Close() {
    p.close()
}

func main() {
    emptyDS := datastore.NewConfigWithOptionsAndDefaults(
        datastore.WithEngine(datastore.MemoryEngine),
        datastore.WithGCWindow(10*time.Hour),
    )
    permissions, err := NewPermissionsService(*emptyDS)
    if err != nil {
        panic(err)
    }
    defer permissions.Close()

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    _, err = permissions.Schema.WriteSchema(ctx, &v1.WriteSchemaRequest{
        Schema: `
            definition user {}

            definition resource {
                relation writer: user
                relation viewer: user

                permission write = writer
                permission view = viewer + writer
            }
        `,
    })
    if err != nil {
        panic(err)
    }

    _, err = permissions.WriteRelationships(ctx, &v1.WriteRelationshipsRequest{
        Updates: []*v1.RelationshipUpdate{{
            Operation: v1.RelationshipUpdate_OPERATION_TOUCH,
            Relationship: &v1.Relationship{
                Resource: &v1.ObjectReference{ObjectType: "resource", ObjectId: "foo"},
                Relation: "writer",
                Subject:  &v1.SubjectReference{Object: &v1.ObjectReference{ObjectType: "user", ObjectId: "me"}},
            },
        }},
    })
    if err != nil {
        panic(err)
    }

    canView, err := permissions.CheckPermission(ctx, &v1.CheckPermissionRequest{
        Resource:   &v1.ObjectReference{ObjectType: "resource", ObjectId: "foo"},
        Permission: "view",
        Subject:    &v1.SubjectReference{Object: &v1.ObjectReference{ObjectType: "user", ObjectId: "me"}},
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(canView.Permissionship)
}

This example uses the memdb, but you can pass in your own config for an external datastore; everything you can configure via CLI flags can be configured with the packages above.

Note that if you embed spicedb like this with an external store, but don't configure dispatch / expose the dispatch api, you'll miss out on subproblem caching and the distributed computation of check results. But if your problem fits in memory, embedding can make more sense.

[stanislavprokopov]

Thanks for the example, it looks similar to thing that is happening in testserver package, yes, it works but as far as I can tell you are still spinning up the whole spicedb server, including GRPC, metrics api, dashboard api, http gw, etc. And all permission checks are going throughout GRPC calls. I would prefer a simpler solution without all the stuff that's actually not necessary for a permission check.

[arashpayan]

Thanks @ecordell. What you shared looks good to me. I'd probably wrap that all up into a helper package in any project(s) I would need it in (to simplify initialization), but that looks like all the functionality I'd need.

[josephschorr]

whole spicedb server, including GRPC, metrics api, dashboard api, http gw

Actually, most of these are disabled as per this config:

        server.WithHTTPGateway(util.HTTPServerConfig{HTTPEnabled: false}),
        server.WithDashboardAPI(util.HTTPServerConfig{HTTPEnabled: false}),
        server.WithMetricsAPI(util.HTTPServerConfig{HTTPEnabled: false}),
        server.WithDispatchServer(util.GRPCServerConfig{Enabled: false}),

As for gRPC itself, yes, you are calling via gRPC, but that can be done over a buffcon in-memory, so the overhead is minimal:

    server.WithGRPCServer(util.GRPCServerConfig{
            Network: util.BufferedNetwork,
            Enabled: true,
        }),

This is how we ourselves use SpiceDB as an embedded system

[vroldanbet]

I've added an example to github.com/authzed/examples in https://github.com/authzed/examples/pull/10

Closing this, if folks have additional feedback on how doing this could be improved, please open new issues with your requests 🙏