georgysavva / scany

Library for scanning data from a database into Go structs and more
MIT License
1.31k stars 68 forks source link

Reduce per-row allocations #130

Closed zolstein closed 7 months ago

zolstein commented 8 months ago

Modify several internal functions to avoid allocating unnecessary memory. This commit employs three strategies:


I used a simple but (hopefully) vaguely representative benchmarking program to measure the impact.

Hyperfine shows an end-to-end runtime reduction of 40%.


# Without change
$ go build main.go && hyperfine --warmup=1 "./main -bench"           12:25:17
Benchmark 1: ./main -bench
  Time (mean ± σ):     15.984 s ±  0.115 s    [User: 93.516 s, System: 20.190 s]
  Range (min … max):   15.825 s … 16.258 s    10 runs

# With change
$ go build main.go && hyperfine --warmup=1 "./main -bench"              12:28:54
Benchmark 1: ./main -bench
  Time (mean ± σ):      9.186 s ±  0.094 s    [User: 60.400 s, System: 8.723 s]
  Range (min … max):    9.020 s …  9.385 s    10 runs

This also reduced allocated objects (when making 1000 queries returning 1000 rows each) by roughly 3 million and allocated space by ~150MB. (I can share the pprof SVGs if you want.)

Here's the (condensed) text: of the benchmark, if you want to replicate.

type Person struct {
    Name string
    Age  sql.Null[int64]
}

type Record struct {
    ID int64
    Person
    CreatedAt time.Time
}

func main() {
    ctx := context.Background()

    numThreads := 10
    numIters := 5000
    var wg sync.WaitGroup
    wg.Add(numThreads)
    for t := 0; t < numThreads; t++ {
        go func(t int) {
            _, err := runQueries(ctx, numIters)
            if err != nil {
                fmt.Fprintf(os.Stderr, "QueryRow failed: %v\n", err)
                os.Exit(1)
            }
            wg.Done()
        }(t)
    }
    wg.Wait()
}

func runQueries(ctx context.Context, numIters int) ([]Record, error) {
    conn, err := pgx.Connect(ctx, "postgres://zolstein:password@localhost:5432/db")
    if err != nil {
        fmt.Fprintf(os.Stderr, "Unable to connect to database: %v\n", err)
        os.Exit(1)
    }
    defer conn.Close(ctx)

    records := make([]Record, 1000)
    for i := 0; i < numIters; i++ {
        records = records[:0]
        rows, err := conn.Query(ctx, "select id, name, age, created_at from people") // Returns 1000 rows
        _ = rows
        if err != nil {
            return nil, err
        }
        err = pgxscan.ScanAll(&records, rows)
        if err != nil {
            return nil, err
        }
        _ = records
    }
    return records, nil
}
codecov[bot] commented 7 months ago

Codecov Report

Attention: Patch coverage is 92.50000% with 3 lines in your changes are missing coverage. Please review.

Project coverage is 80.72%. Comparing base (981a63a) to head (9e763fa).

Files Patch % Lines
dbscan/dbscan.go 87.50% 3 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #130 +/- ## ========================================== + Coverage 80.41% 80.72% +0.30% ========================================== Files 5 5 Lines 531 555 +24 ========================================== + Hits 427 448 +21 - Misses 89 92 +3 Partials 15 15 ``` | [Flag](https://app.codecov.io/gh/georgysavva/scany/pull/130/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Georgy+Savva) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/georgysavva/scany/pull/130/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Georgy+Savva) | `80.72% <92.50%> (+0.30%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Georgy+Savva#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

georgysavva commented 7 months ago

Thank you for this PR. It looks very good!

georgysavva commented 7 months ago

@zolstein here is the new release with the change: https://github.com/georgysavva/scany/releases/tag/v2.1.1