authzed / spicedb

Open Source, Google Zanzibar-inspired permissions database to enable fine-grained authorization for customer applications
https://authzed.com/docs
Apache License 2.0
4.71k stars 250 forks source link

spanner: use stale reads for current_timestamp for optimized revision #1935

Closed ecordell closed 2 weeks ago

ecordell commented 2 weeks ago

The default is to use a strong read which does some coordination with the leader. For optimized revision computation, it should be fine to use the timestamp returned by the local spanner node and skip the extra coordination.

Testing with a single-region spanner saw these queries drop an order of magnitude.

It is probably also fine to use a stale read for the HeadRevision call, but holding off for further testing in case it could potentially cause a new enemy problem.

To test this (since testing this behavior requires a real spanner), I built a modified image with the following now function, and ran it in a stage environment against a (small) spanner:

func (sd *spannerDatastore) now(ctx context.Context) (time.Time, error) {
    go func() {
        start := time.Now()
        var timestamp time.Time
        defer func() {
            log.Ctx(ctx).Warn().Dur("duration", time.Now().Sub(start)).Time("now", timestamp).Str("type", "stale").Msg("now")
        }()
        if err := sd.client.Single().WithTimestampBound(spanner.ExactStaleness(sd.config.followerReadDelay)).Query(ctx, nowStmt).Do(func(r *spanner.Row) error {
            return r.Columns(&timestamp)
        }); err != nil {
            log.Ctx(ctx).Err(err).Msg("failed to stale read current timestamp")
        }
    }()
    start := time.Now()
    var timestamp time.Time
    defer func() {
        log.Ctx(ctx).Warn().Dur("duration", time.Now().Sub(start)).Time("now", timestamp).Str("type", "strong").Msg("now")
    }()
    if err := sd.client.Single().Query(ctx, nowStmt).Do(func(r *spanner.Row) error {
        return r.Columns(&timestamp)
    }); err != nil {
        return time.Time{}, err
    }

    return timestamp, nil
}

To compare the performance of stale vs. strong reads here.

Some logs from the test:

{"level":"warn","requestID":"cpk60tjceiqd0bvvpmug","duration":5.713904,"now":"2024-06-11T14:34:30Z","type":"stale","time":"2024-06-11T14:34:30Z","message":"now"}
{"level":"warn","requestID":"cpk60tjceiqd0bvvpmug","duration":38.547463,"now":"2024-06-11T14:34:30Z","type":"strong","time":"2024-06-11T14:34:30Z","message":"now"}
github-actions[bot] commented 2 weeks ago

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

ecordell commented 2 weeks ago

I have read the CLA Document and I hereby sign the CLA

ecordell commented 2 weeks ago

recheck