Feature: additional endpoint configuration for RO

[elebiodaslingshot]

Would like to be able to provide a configuration to spicedb that states an endpoint for writing and an endpoint for reading.

This provides the ability for removing bottlenecks in the network and obtaining HA by scaling out on what is a very read intensive application.

[vroldanbet]

@elebiodaslingshot just to clarify, are you referring to different endpoints to separate reads and writes towards the backing datastore? or the SpiceDB API?

[ecordell]

(assuming the answer to @vroldanbet's question is that you want separate SpiceDB api endpoints)

You should be able to achieve this with the operator today: spin up two SpiceDB clusters talking to the same datastore, and configure one of them as read-only.

apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: readwrite
spec:
  config:
    datastoreEngine: postgres
  secretName: spicedb-config
---
apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: readonly
spec:
  config:
    datastoreEngine: postgres
    datastoreReadonly: "true"
  secretName: spicedb-config
---
apiVersion: v1
kind: Secret
metadata:
  name: spicedb-config
stringData:
  datastore_uri: "postgres://<your backend>"
  preshared_key: "averysecretpresharedkey" 

That will result in two clusters, one of which is readonly, and two Services in the cluster that you can route traffic to.

Is that what you were looking for? Did you try it and run into an issue?

[elebiodaslingshot]

@elebiodaslingshot just to clarify, are you referring to different endpoints to separate reads and writes towards the backing datastore? or the SpiceDB API?

yes, seperate reads and writes towards the backing datastore

[elebiodaslingshot]

(assuming the answer to @vroldanbet's question is that you want separate SpiceDB api endpoints)

You should be able to achieve this with the operator today: spin up two SpiceDB clusters talking to the same datastore, and configure one of them as read-only.

apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: readwrite
spec:
  config:
    datastoreEngine: postgres
  secretName: spicedb-config
---
apiVersion: authzed.com/v1alpha1
kind: SpiceDBCluster
metadata:
  name: readonly
spec:
  config:
    datastoreEngine: postgres
    datastoreReadonly: "true"
  secretName: spicedb-config
---
apiVersion: v1
kind: Secret
metadata:
  name: spicedb-config
stringData:
  datastore_uri: "postgres://<your backend>"
  preshared_key: "averysecretpresharedkey" 

That will result in two clusters, one of which is readonly, and two Services in the cluster that you can route traffic to.

Is that what you were looking for? Did you try it and run into an issue?

This is a solution but results in loss of a consolidated api

[vroldanbet]

@elebiodaslingshot I'm afraid supporting this is not trivial, not because it can't be implemented - it would be trivial to add this to SpiceDB - but because it has implications for the consistency model exposed by SpiceDB.

TL;DR: before we can separate read/write traffic, MySQL and PostgreSQL datastore implementations need to be re-engineered to support read-replicas.

See, one of SpiceDB's key features is its tunable consistency model, which clients can define, and leverages ZedTokens to represent a consistent snapshot of the system's state. Correct me if this is not your case though: typically folks want to use separated endpoints in order to send traffic to either a primary or to a cluster of replicas, with the goal of offloading read workloads from the primary for scalability reasons. While this does not break the consistency model by itself, it could depending on the configuration of the backing datastore: a cluster of read replicas will typically lag behind its primary, which can lead to an inconsistent view of the world for SpiceDB and end up defeating the foundations of the service to defeat the "New Enemy Problem". A cluster of read replicas with synchronous replication is possible, but would probably defeat the initial goal of scaling database reads.

Datastores like CockroachDB and Spanner don't have this problem because they have a fundamentally different architectures and have first-class support primitives to expose the underlying MVCC of the database, which SpiceDB leverages to implement its consistency model. We are aware there are options to explore on the PostgreSQL and MySQL side, but the engineering required for adding support for separated endpoints dwarfs in comparison to supporting consistent world-view over lagged replicas.

Needless to say, SpiceDB could support such separation if you are willing to accept that none of the consistency models exposed in our API will honor its semantics and that everything becomes eventually consistent.

[elebiodaslingshot]

This is an excellent response and much appreciated. I have taken a look at Zookies which solve the consistency "somewhat" as it does not guarantee the latest ACL but one newer than the timestamp on the Zookie, correct me if I am wrong. Basing my stuff off this paper: https://www.usenix.org/system/files/atc19-pang.pdf (Page 35 last paragraph)

To your one question yes that is the usecase here to offload from the primary. I think given that our usecase doesnt represent a banking platform eventual consistency is fine especially if we get Zookies implemented across our systems.

Parhaps we should have went with cockroachDB maybe down the road we can switch.

[vroldanbet]

@elebiodaslingshot zookies won't solve the latest ACL nor they are meant for that - it's a causality token. But SpiceDB allows you to ask for the "freshest ACL at the time of the request" with fully_consistent.

That's the thing - you won't be able to rely on zookies if you are offloading traffic to a read-replica, unless SpiceDB is modified to account to replication across primaries and replicas. As I said, we know there are options to explore.

What I'm suggesting is that you can't relly on any of the consistency options we provide in our API the moment you hit a replica, because it could be lagged, and SpiceDB works on the assumption that it's talking to an instance that has the freshest state of the world.

[elebiodaslingshot]

Ok, I see now. I think that answers everything then.

[vroldanbet]

Opened https://github.com/authzed/spicedb/issues/1320 and https://github.com/authzed/spicedb/issues/1321

[elebiodaslingshot]

Appreciate it @vroldanbet 👑