Run and test benchmarks against C# as well

[RReverser]

Description of Changes

Turns out, there are some APIs that are not covered by any of our tests, except benchmarks.

Benchmarks are executed as both actual benchmarks and as integration tests for Rust, but not for C#. This PR fixes that to ensure that both Rust and C# are tested and measured on the CI from now on.

It also makes some adjustments to integration tests themselves, in particular fixes an issue where in integration tests reducers would be called but the response would never be read, so if a reducer panics, the tests would still succeed. Additionally, on failures the SpacetimeDB logs will be read and added to the error message so that you get a useful stacktrace from the module itself.

I also had to adjust some benchmark numbers (in particular, for ia_loop) since they were already very slow on CI for Rust, but become even more so for C#. The new loads are much lower, but should still be very usable for benchmark comparisons.

The only remaining issue is that the benchmark names have changed, and the benchmark comparison tool tries to compare mismatched pairs - I'm not sure what the solution to this is. cc @kazimuth

API and ABI breaking changes

If this is an API or ABI breaking change, please apply the corresponding GitHub label.

Expected complexity level and risk

How complicated do you think these changes are? Grade on a scale from 1 to 5, where 1 is a trivial change, and 5 is a deep-reaching and complex change.

This complexity rating applies not only to the complexity apparent in the diff, but also to its interactions with existing and future code.

If you answered more than a 2, explain what is complex about the PR, and what other components it interacts with in potentially concerning ways.

Testing

Describe any testing you've done, and any testing you'd like your reviewers to do, so that you're confident that all the changes work as expected!

• [x] Write a test you've completed here.
• [ ] Write a test you want a reviewer to do here, so they can check it off when they're satisfied.

[RReverser]

benchmarks please

[github-actions[bot]]

Benchmarking failed. Please check the workflow run for details.

[github-actions[bot]]

Benchmarking failed. Please check the workflow run for details.

[RReverser]

benchmarks please

[github-actions[bot]]

Benchmarking failed. Please check the workflow run for details.

[github-actions[bot]]

Callgrind benchmark results

# Callgrind Benchmark Report These benchmarks were run using [callgrind](https://valgrind.org/docs/manual/cg-manual.html), an instruction-level profiler. They allow comparisons between sqlite (`sqlite`), SpacetimeDB running through a module (`stdb_module`), and the underlying SpacetimeDB data storage engine (`stdb_raw`). Callgrind emulates a CPU to collect the below estimates. Measurement changes larger than five percent are in bold.

In-memory benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6396 | 6399 | -0.05% | 6518 | 6491 | 0.42% | | sqlite | 5579 | 5579 | 0.00% | 6071 | 6047 | 0.40% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76590 | 76593 | -0.00% | 77154 | 77049 | 0.14% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118832 | 119091 | -0.22% | 119526 | 119793 | -0.22% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25083 | 25084 | -0.00% | 25641 | 25578 | 0.25% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24048 | 24051 | -0.01% | 24480 | 24507 | -0.11% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 144695 | 144695 | 0.00% | 146013 | 146159 | -0.10% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 124044 | 124044 | 0.00% | 125180 | 125370 | -0.15% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 131361 | 131361 | 0.00% | 132749 | 132819 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 134494 | 134494 | 0.00% | 136036 | 136086 | -0.04% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 877113 | 878319 | -0.14% | 896207 | 929371 | -3.57% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1027613 | 1026724 | 0.09% | 1061345 | 1056184 | 0.49% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 398320 | 398320 | 0.00% | 414570 | 416162 | -0.38% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 983637 | 983637 | 0.00% | 1021201 | 1024023 | -0.28% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153723 | 153726 | -0.00% | 153809 | 153834 | -0.02% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16748 | 16751 | -0.02% | 16818 | 16843 | -0.15% | | sqlite | u32_u64_str | unique_0 | 1024 | 1068281 | 1067255 | 0.10% | 1071699 | 1070643 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 76261 | 76201 | 0.08% | 77359 | 77187 | 0.22% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27719 | 0.25% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9526 | 0.71% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 20116332 | 20085072 | 0.16% | 20692874 | 20703480 | -0.05% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1286883 | 1285086 | 0.14% | 1330065 | 1359386 | -2.16% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1802182 | 1802182 | 0.00% | 1811346 | 1811420 | -0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 128528 | 128528 | 0.00% | 131416 | 131350 | 0.05% |

On-disk benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6401 | 6404 | -0.05% | 6539 | 6512 | 0.41% | | sqlite | 5621 | 5621 | 0.00% | 6181 | 6157 | 0.39% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76595 | 76598 | -0.00% | 77123 | 77034 | 0.12% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118837 | 119096 | -0.22% | 119523 | 119850 | -0.27% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25085 | 25088 | -0.01% | 25591 | 25518 | 0.29% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24053 | 24056 | -0.01% | 24445 | 24496 | -0.21% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 125965 | 125965 | 0.00% | 127457 | 127535 | -0.06% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 146616 | 146616 | 0.00% | 148338 | 148352 | -0.01% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 136616 | 136616 | 0.00% | 138664 | 138614 | 0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 133457 | 133457 | 0.00% | 135347 | 135329 | 0.01% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 827876 | 826091 | 0.22% | 877400 | 876309 | 0.12% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 979158 | 976812 | 0.24% | 1043858 | 1035968 | 0.76% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 415857 | 415857 | 0.00% | 431361 | 433009 | -0.38% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1021898 | 1021898 | 0.00% | 1058246 | 1062154 | -0.37% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153728 | 153731 | -0.00% | 153794 | 153823 | -0.02% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16753 | 16756 | -0.02% | 16819 | 16844 | -0.15% | | sqlite | u32_u64_str | unique_0 | 1024 | 1071343 | 1070323 | 0.10% | 1075137 | 1074097 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 78033 | 77973 | 0.08% | 79375 | 79267 | 0.14% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27719 | 0.25% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9526 | 0.71% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 19032866 | 18998827 | 0.18% | 19699918 | 19715619 | -0.08% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1239473 | 1237909 | 0.13% | 1312999 | 1313287 | -0.02% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1809743 | 1809743 | 0.00% | 1818487 | 1818309 | 0.01% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 132654 | 132654 | 0.00% | 135622 | 135620 | 0.00% |

[RReverser]

@kazimuth FWIW this is marked as draft because it's not quite ready to look at. I only created a PR because without it our CI doesn't run on branches :)

[kazimuth]

Well, looks good to me so far :)

[RReverser]

benchmarks please

[github-actions[bot]]

Callgrind benchmark results

# Callgrind Benchmark Report These benchmarks were run using [callgrind](https://valgrind.org/docs/manual/cg-manual.html), an instruction-level profiler. They allow comparisons between sqlite (`sqlite`), SpacetimeDB running through a module (`stdb_module`), and the underlying SpacetimeDB data storage engine (`stdb_raw`). Callgrind emulates a CPU to collect the below estimates. Measurement changes larger than five percent are in bold.

In-memory benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6396 | 6399 | -0.05% | 6518 | 6523 | -0.08% | | sqlite | 5579 | 5579 | 0.00% | 6071 | 6019 | 0.86% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76590 | 76593 | -0.00% | 77150 | 76981 | 0.22% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118824 | 119091 | -0.22% | 119542 | 119779 | -0.20% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25081 | 25083 | -0.01% | 25643 | 25627 | 0.06% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24048 | 24051 | -0.01% | 24492 | 24451 | 0.17% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 144695 | 144695 | 0.00% | 146013 | 146261 | -0.17% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 124044 | 124044 | 0.00% | 125180 | 125338 | -0.13% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 131361 | 131361 | 0.00% | 132745 | 132811 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 134494 | 134494 | 0.00% | 136032 | 136100 | -0.05% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 877191 | 875757 | 0.16% | 896277 | 899593 | -0.37% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1026866 | 1023785 | 0.30% | 1090894 | 1064997 | 2.43% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 398320 | 398320 | 0.00% | 414570 | 415178 | -0.15% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 983637 | 983637 | 0.00% | 1021209 | 1020529 | 0.07% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153723 | 153726 | -0.00% | 153809 | 153866 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16748 | 16751 | -0.02% | 16818 | 16875 | -0.34% | | sqlite | u32_u64_str | unique_0 | 1024 | 1068281 | 1067273 | 0.09% | 1071699 | 1070681 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 76261 | 76207 | 0.07% | 77359 | 77445 | -0.11% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 20520384 | 20084936 | 2.17% | 21153292 | 20662240 | 2.38% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1286540 | 1284386 | 0.17% | 1329406 | 1359554 | -2.22% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1802182 | 1802182 | 0.00% | 1811346 | 1811362 | -0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 128528 | 128528 | 0.00% | 131416 | 131492 | -0.06% |

On-disk benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6401 | 6404 | -0.05% | 6539 | 6544 | -0.08% | | sqlite | 5621 | 5621 | 0.00% | 6181 | 6121 | 0.98% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76595 | 76598 | -0.00% | 77119 | 76982 | 0.18% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 119926 | 119096 | 0.70% | 120592 | 119820 | 0.64% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25085 | 25088 | -0.01% | 25611 | 25572 | 0.15% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24053 | 24056 | -0.01% | 24445 | 24400 | 0.18% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 125965 | 125965 | 0.00% | 127457 | 127547 | -0.07% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 146616 | 146616 | 0.00% | 148338 | 148446 | -0.07% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 136616 | 136616 | 0.00% | 138664 | 138724 | -0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 133475 | 133457 | 0.01% | 135397 | 135341 | 0.04% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 827413 | 827011 | 0.05% | 876793 | 880749 | -0.45% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 977898 | 974099 | 0.39% | 1042478 | 1044871 | -0.23% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 415857 | 415857 | 0.00% | 431357 | 431875 | -0.12% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1021916 | 1021898 | 0.00% | 1058268 | 1057614 | 0.06% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153728 | 153731 | -0.00% | 153798 | 153855 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16753 | 16756 | -0.02% | 16819 | 16880 | -0.36% | | sqlite | u32_u64_str | unique_0 | 1024 | 1071361 | 1070323 | 0.10% | 1075167 | 1074097 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 78033 | 77973 | 0.08% | 79375 | 79303 | 0.09% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 19032077 | 19002577 | 0.16% | 19687273 | 19644985 | 0.22% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1239286 | 1237396 | 0.15% | 1311882 | 1309094 | 0.21% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1809743 | 1809761 | -0.00% | 1818487 | 1818485 | 0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 132654 | 132654 | 0.00% | 135622 | 135834 | -0.16% |

[github-actions[bot]]

Benchmarking failed. Please check the workflow run for details.

[github-actions[bot]]

Benchmarking failed. Please check the workflow run for details.

[github-actions[bot]]

Callgrind benchmark in progress...

[github-actions[bot]]

Benchmarking failed. Please check the workflow run for details.

[github-actions[bot]]

Benchmark in progress...

[github-actions[bot]]

Callgrind benchmark results

# Callgrind Benchmark Report These benchmarks were run using [callgrind](https://valgrind.org/docs/manual/cg-manual.html), an instruction-level profiler. They allow comparisons between sqlite (`sqlite`), SpacetimeDB running through a module (`stdb_module`), and the underlying SpacetimeDB data storage engine (`stdb_raw`). Callgrind emulates a CPU to collect the below estimates. Measurement changes larger than five percent are in bold.

In-memory benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6396 | 6399 | -0.05% | 6518 | 6523 | -0.08% | | sqlite | 5579 | 5579 | 0.00% | 6051 | 6019 | 0.53% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76590 | 76593 | -0.00% | 77182 | 77013 | 0.22% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118832 | 119091 | -0.22% | 119698 | 119687 | 0.01% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25084 | 25085 | -0.00% | 25642 | 25617 | 0.10% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24048 | 24051 | -0.01% | 24484 | 24447 | 0.15% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 144695 | 144695 | 0.00% | 146161 | 146257 | -0.07% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 124044 | 124044 | 0.00% | 125232 | 125334 | -0.08% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 131361 | 131361 | 0.00% | 132741 | 132799 | -0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 134494 | 134494 | 0.00% | 136008 | 136104 | -0.07% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 878029 | 877138 | 0.10% | 900015 | 931798 | -3.41% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1025096 | 1024221 | 0.09% | 1057308 | 1065829 | -0.80% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 398320 | 398320 | 0.00% | 415018 | 415186 | -0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 983637 | 983637 | 0.00% | 1022241 | 1020529 | 0.17% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153723 | 153726 | -0.00% | 153809 | 153866 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16748 | 16751 | -0.02% | 16818 | 16875 | -0.34% | | sqlite | u32_u64_str | unique_0 | 1024 | 1068281 | 1067255 | 0.10% | 1071735 | 1070663 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 76261 | 76207 | 0.07% | 77359 | 77445 | -0.11% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 20540888 | 20490259 | 0.25% | 21224026 | 21136189 | 0.42% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1286321 | 1284538 | 0.14% | 1364203 | 1325984 | 2.88% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1802182 | 1802182 | 0.00% | 1811346 | 1811362 | -0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 128528 | 128528 | 0.00% | 131464 | 131492 | -0.02% |

On-disk benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6401 | 6404 | -0.05% | 6539 | 6544 | -0.08% | | sqlite | 5621 | 5621 | 0.00% | 6161 | 6121 | 0.65% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76595 | 76598 | -0.00% | 77151 | 76978 | 0.22% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118837 | 119096 | -0.22% | 119547 | 119884 | -0.28% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25085 | 25088 | -0.01% | 25611 | 25572 | 0.15% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24053 | 24056 | -0.01% | 24449 | 24404 | 0.18% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 125965 | 125965 | 0.00% | 127553 | 127543 | 0.01% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 146616 | 146616 | 0.00% | 148426 | 148450 | -0.02% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 136616 | 136616 | 0.00% | 138660 | 138724 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 133457 | 133457 | 0.00% | 135335 | 135341 | -0.00% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 828309 | 826099 | 0.27% | 881031 | 879673 | 0.15% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 977465 | 976494 | 0.10% | 1040285 | 1047160 | -0.66% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 415857 | 415857 | 0.00% | 431809 | 431867 | -0.01% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1021898 | 1021898 | 0.00% | 1059350 | 1057622 | 0.16% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153728 | 153731 | -0.00% | 153794 | 153855 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16753 | 16756 | -0.02% | 16819 | 16880 | -0.36% | | sqlite | u32_u64_str | unique_0 | 1024 | 1071343 | 1070323 | 0.10% | 1075181 | 1074097 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 78033 | 77973 | 0.08% | 79431 | 79303 | 0.16% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 19032967 | 19001267 | 0.17% | 19754971 | 19640773 | 0.58% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1239715 | 1238283 | 0.12% | 1316711 | 1309651 | 0.54% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1809743 | 1809743 | 0.00% | 1818475 | 1818455 | 0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 132654 | 132654 | 0.00% | 135602 | 135834 | -0.17% |

[github-actions[bot]]

Benchmark in progress...

[github-actions[bot]]

Callgrind benchmark results

# Callgrind Benchmark Report These benchmarks were run using [callgrind](https://valgrind.org/docs/manual/cg-manual.html), an instruction-level profiler. They allow comparisons between sqlite (`sqlite`), SpacetimeDB running through a module (`stdb_module`), and the underlying SpacetimeDB data storage engine (`stdb_raw`). Callgrind emulates a CPU to collect the below estimates. Measurement changes larger than five percent are in bold.

In-memory benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6396 | 6399 | -0.05% | 6518 | 6523 | -0.08% | | sqlite | 5579 | 5579 | 0.00% | 6051 | 6019 | 0.53% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76590 | 76593 | -0.00% | 77178 | 77013 | 0.21% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 119921 | 119091 | 0.70% | 120691 | 119687 | 0.84% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25098 | 25085 | 0.05% | 25660 | 25617 | 0.17% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24048 | 24051 | -0.01% | 24480 | 24447 | 0.13% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 144695 | 144695 | 0.00% | 146161 | 146257 | -0.07% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 124044 | 124044 | 0.00% | 125232 | 125334 | -0.08% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 131361 | 131361 | 0.00% | 132733 | 132799 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 134494 | 134494 | 0.00% | 136008 | 136104 | -0.07% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 877034 | 877138 | -0.01% | 898980 | 931798 | -3.52% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1026852 | 1024221 | 0.26% | 1058448 | 1065829 | -0.69% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 398320 | 398320 | 0.00% | 415014 | 415186 | -0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 983637 | 983637 | 0.00% | 1022249 | 1020529 | 0.17% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153723 | 153726 | -0.00% | 153809 | 153866 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16748 | 16751 | -0.02% | 16818 | 16875 | -0.34% | | sqlite | u32_u64_str | unique_0 | 1024 | 1068281 | 1067255 | 0.10% | 1071735 | 1070663 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 76261 | 76207 | 0.07% | 77359 | 77445 | -0.11% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 20114262 | 20490259 | -1.84% | 20759616 | 21136189 | -1.78% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1286351 | 1284538 | 0.14% | 1364221 | 1325984 | 2.88% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1802200 | 1802182 | 0.00% | 1811372 | 1811362 | 0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 128528 | 128528 | 0.00% | 131464 | 131492 | -0.02% |

On-disk benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6401 | 6404 | -0.05% | 6539 | 6544 | -0.08% | | sqlite | 5621 | 5621 | 0.00% | 6161 | 6121 | 0.65% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76595 | 76598 | -0.00% | 77147 | 76978 | 0.22% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118837 | 119096 | -0.22% | 119583 | 119884 | -0.25% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25085 | 25088 | -0.01% | 25603 | 25572 | 0.12% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24053 | 24056 | -0.01% | 24445 | 24404 | 0.17% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 125965 | 125965 | 0.00% | 127553 | 127543 | 0.01% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 146616 | 146616 | 0.00% | 148422 | 148450 | -0.02% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 136616 | 136616 | 0.00% | 138660 | 138724 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 133457 | 133457 | 0.00% | 135335 | 135341 | -0.00% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 827542 | 826099 | 0.17% | 880298 | 879673 | 0.07% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 977242 | 976494 | 0.08% | 1040320 | 1047160 | -0.65% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 415875 | 415857 | 0.00% | 431831 | 431867 | -0.01% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1021898 | 1021898 | 0.00% | 1059346 | 1057622 | 0.16% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153728 | 153731 | -0.00% | 153794 | 153855 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16753 | 16756 | -0.02% | 16819 | 16880 | -0.36% | | sqlite | u32_u64_str | unique_0 | 1024 | 1071343 | 1070323 | 0.10% | 1075181 | 1074097 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 78033 | 77973 | 0.08% | 79431 | 79303 | 0.16% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 19033443 | 19001267 | 0.17% | 19752631 | 19640773 | 0.57% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1240028 | 1238283 | 0.14% | 1316432 | 1309651 | 0.52% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1809743 | 1809743 | 0.00% | 1818475 | 1818455 | 0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 132654 | 132654 | 0.00% | 135602 | 135834 | -0.17% |

[github-actions[bot]]

Benchmark in progress...

[github-actions[bot]]

Callgrind benchmark results

# Callgrind Benchmark Report These benchmarks were run using [callgrind](https://valgrind.org/docs/manual/cg-manual.html), an instruction-level profiler. They allow comparisons between sqlite (`sqlite`), SpacetimeDB running through a module (`stdb_module`), and the underlying SpacetimeDB data storage engine (`stdb_raw`). Callgrind emulates a CPU to collect the below estimates. Measurement changes larger than five percent are in bold.

In-memory benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6396 | 6399 | -0.05% | 6518 | 6523 | -0.08% | | sqlite | 5579 | 5579 | 0.00% | 6051 | 6019 | 0.53% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76590 | 76593 | -0.00% | 77186 | 77013 | 0.22% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 119921 | 119091 | 0.70% | 120691 | 119687 | 0.84% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25080 | 25085 | -0.02% | 25662 | 25617 | 0.18% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24048 | 24051 | -0.01% | 24480 | 24447 | 0.13% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 144695 | 144695 | 0.00% | 146161 | 146257 | -0.07% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 124044 | 124044 | 0.00% | 125232 | 125334 | -0.08% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 131361 | 131361 | 0.00% | 132741 | 132799 | -0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 134494 | 134494 | 0.00% | 135996 | 136104 | -0.08% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 878335 | 877138 | 0.14% | 900341 | 931798 | -3.38% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1027999 | 1024221 | 0.37% | 1060039 | 1065829 | -0.54% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 398320 | 398320 | 0.00% | 415018 | 415186 | -0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 983637 | 983637 | 0.00% | 1022257 | 1020529 | 0.17% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153723 | 153726 | -0.00% | 153809 | 153866 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16748 | 16751 | -0.02% | 16822 | 16875 | -0.31% | | sqlite | u32_u64_str | unique_0 | 1024 | 1068281 | 1067255 | 0.10% | 1071735 | 1070663 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 76261 | 76207 | 0.07% | 77359 | 77445 | -0.11% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 20104470 | 20490259 | -1.88% | 20733720 | 21136189 | -1.90% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1286440 | 1284538 | 0.15% | 1333214 | 1325984 | 0.55% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1802182 | 1802182 | 0.00% | 1811346 | 1811362 | -0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 128528 | 128528 | 0.00% | 131464 | 131492 | -0.02% |

On-disk benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6401 | 6404 | -0.05% | 6539 | 6544 | -0.08% | | sqlite | 5621 | 5621 | 0.00% | 6161 | 6121 | 0.65% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76595 | 76598 | -0.00% | 77159 | 76978 | 0.24% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118837 | 119096 | -0.22% | 119547 | 119884 | -0.28% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25086 | 25088 | -0.01% | 25624 | 25572 | 0.20% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24053 | 24056 | -0.01% | 24441 | 24404 | 0.15% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 125965 | 125965 | 0.00% | 127553 | 127543 | 0.01% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 146616 | 146616 | 0.00% | 148426 | 148450 | -0.02% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 136616 | 136616 | 0.00% | 138660 | 138724 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 133457 | 133457 | 0.00% | 135335 | 135341 | -0.00% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 826674 | 826099 | 0.07% | 848468 | 879673 | -3.55% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 976905 | 976494 | 0.04% | 1039623 | 1047160 | -0.72% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 415857 | 415857 | 0.00% | 431801 | 431867 | -0.02% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1021898 | 1021898 | 0.00% | 1059354 | 1057622 | 0.16% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153728 | 153731 | -0.00% | 153798 | 153855 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16753 | 16756 | -0.02% | 16819 | 16880 | -0.36% | | sqlite | u32_u64_str | unique_0 | 1024 | 1071343 | 1070323 | 0.10% | 1075181 | 1074097 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 78033 | 77973 | 0.08% | 79431 | 79303 | 0.16% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 19031242 | 19001267 | 0.16% | 19741242 | 19640773 | 0.51% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1239404 | 1238283 | 0.09% | 1317528 | 1309651 | 0.60% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1809743 | 1809743 | 0.00% | 1818475 | 1818455 | 0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 132654 | 132654 | 0.00% | 135602 | 135834 | -0.17% |

[github-actions[bot]]

Criterion benchmark results

# Criterion benchmark report **YOU SHOULD PROBABLY IGNORE THESE RESULTS.** Criterion is a wall time based benchmarking system that is extremely noisy when run on CI. We collect these results for longitudinal analysis, but they are not reliable for comparing individual PRs. Go look at the callgrind report instead. ## empty | db | on disk | new latency | old latency | new throughput | old throughput | |----------|---------|---------------|---------------|----------------|----------------| | sqlite | 💿 | - | 412.2±2.17ns | - | - | | sqlite | 🧠 | - | 404.0±1.61ns | - | - | | stdb_raw | 💿 | 773.5±1.19ns | 779.0±0.89ns | - | - | | stdb_raw | 🧠 | 772.2±0.99ns | 778.9±1.13ns | - | - | ## insert_1 | db | on disk | schema | indices | preload | new latency | old latency | new throughput | old throughput | |----|---------|--------|---------|---------|-------------|-------------|----------------|----------------| ## insert_bulk | db | on disk | schema | indices | preload | count | new latency | old latency | new throughput | old throughput | |----------|---------|-------------|-------------------|---------|-------|-----------------|-----------------|----------------|----------------| | sqlite | 💿 | u32_u64_str | btree_each_column | 2048 | 256 | - | 585.8±0.61µs | - | 1706 tx/sec | | sqlite | 💿 | u32_u64_str | unique_0 | 2048 | 256 | - | 152.0±0.63µs | - | 6.4 Ktx/sec | | sqlite | 💿 | u32_u64_u64 | btree_each_column | 2048 | 256 | - | 471.0±0.44µs | - | 2.1 Ktx/sec | | sqlite | 💿 | u32_u64_u64 | unique_0 | 2048 | 256 | - | 137.8±0.61µs | - | 7.1 Ktx/sec | | sqlite | 🧠 | u32_u64_str | btree_each_column | 2048 | 256 | - | 446.5±0.37µs | - | 2.2 Ktx/sec | | sqlite | 🧠 | u32_u64_str | unique_0 | 2048 | 256 | - | 121.3±1.06µs | - | 8.0 Ktx/sec | | sqlite | 🧠 | u32_u64_u64 | btree_each_column | 2048 | 256 | - | 368.8±0.77µs | - | 2.6 Ktx/sec | | sqlite | 🧠 | u32_u64_u64 | unique_0 | 2048 | 256 | - | 105.8±0.61µs | - | 9.2 Ktx/sec | | stdb_raw | 💿 | u32_u64_str | btree_each_column | 2048 | 256 | 482.5±22.55µs | 587.6±21.64µs | 2.0 Ktx/sec | 1701 tx/sec | | stdb_raw | 💿 | u32_u64_str | unique_0 | 2048 | 256 | 486.2±31.01µs | 467.3±45.28µs | 2.0 Ktx/sec | 2.1 Ktx/sec | | stdb_raw | 💿 | u32_u64_u64 | btree_each_column | 2048 | 256 | 380.5±5.84µs | 378.9±5.66µs | 2.6 Ktx/sec | 2.6 Ktx/sec | | stdb_raw | 💿 | u32_u64_u64 | unique_0 | 2048 | 256 | 350.9±9.04µs | 299.3±7.61µs | 2.8 Ktx/sec | 3.3 Ktx/sec | | stdb_raw | 🧠 | u32_u64_str | btree_each_column | 2048 | 256 | 300.0±0.57µs | 294.9±0.15µs | 3.3 Ktx/sec | 3.3 Ktx/sec | | stdb_raw | 🧠 | u32_u64_str | unique_0 | 2048 | 256 | 231.6±0.23µs | 229.6±0.43µs | 4.2 Ktx/sec | 4.3 Ktx/sec | | stdb_raw | 🧠 | u32_u64_u64 | btree_each_column | 2048 | 256 | 243.0±0.34µs | 232.6±0.27µs | 4.0 Ktx/sec | 4.2 Ktx/sec | | stdb_raw | 🧠 | u32_u64_u64 | unique_0 | 2048 | 256 | 214.7±0.08µs | 210.0±0.13µs | 4.5 Ktx/sec | 4.7 Ktx/sec | ## iterate | db | on disk | schema | indices | new latency | old latency | new throughput | old throughput | |----------|---------|-------------|----------|--------------|---------------|----------------|----------------| | sqlite | 💿 | u32_u64_str | unique_0 | - | 23.5±0.13µs | - | 41.5 Ktx/sec | | sqlite | 💿 | u32_u64_u64 | unique_0 | - | 21.5±0.24µs | - | 45.3 Ktx/sec | | sqlite | 🧠 | u32_u64_str | unique_0 | - | 20.8±0.18µs | - | 46.8 Ktx/sec | | sqlite | 🧠 | u32_u64_u64 | unique_0 | - | 19.1±0.39µs | - | 51.1 Ktx/sec | | stdb_raw | 💿 | u32_u64_str | unique_0 | 4.9±0.00µs | 4.0±0.00µs | 200.0 Ktx/sec | 246.4 Ktx/sec | | stdb_raw | 💿 | u32_u64_u64 | unique_0 | 4.8±0.00µs | 3.9±0.00µs | 204.1 Ktx/sec | 253.0 Ktx/sec | | stdb_raw | 🧠 | u32_u64_str | unique_0 | 4.9±0.00µs | 3.9±0.00µs | 199.7 Ktx/sec | 247.3 Ktx/sec | | stdb_raw | 🧠 | u32_u64_u64 | unique_0 | 4.8±0.00µs | 3.9±0.00µs | 204.1 Ktx/sec | 253.2 Ktx/sec | ## find_unique | db | on disk | key type | preload | new latency | old latency | new throughput | old throughput | |----|---------|----------|---------|-------------|-------------|----------------|----------------| ## filter | db | on disk | key type | index strategy | load | count | new latency | old latency | new throughput | old throughput | |----------|---------|----------|----------------|------|-------|--------------|---------------|----------------|----------------| | sqlite | 💿 | string | index | 2048 | 256 | - | 69.5±0.34µs | - | 14.1 Ktx/sec | | sqlite | 💿 | u64 | index | 2048 | 256 | - | 66.0±0.38µs | - | 14.8 Ktx/sec | | sqlite | 🧠 | string | index | 2048 | 256 | - | 64.8±0.16µs | - | 15.1 Ktx/sec | | sqlite | 🧠 | u64 | index | 2048 | 256 | - | 60.1±0.16µs | - | 16.2 Ktx/sec | | stdb_raw | 💿 | string | index | 2048 | 256 | 4.9±0.00µs | 5.0±0.00µs | 198.1 Ktx/sec | 195.6 Ktx/sec | | stdb_raw | 💿 | u64 | index | 2048 | 256 | 4.9±0.01µs | 4.9±0.00µs | 201.0 Ktx/sec | 198.7 Ktx/sec | | stdb_raw | 🧠 | string | index | 2048 | 256 | 4.9±0.00µs | 5.0±0.00µs | 198.4 Ktx/sec | 195.7 Ktx/sec | | stdb_raw | 🧠 | u64 | index | 2048 | 256 | 4.8±0.00µs | 4.9±0.00µs | 201.4 Ktx/sec | 198.6 Ktx/sec | ## serialize | schema | format | count | new latency | old latency | new throughput | old throughput | |-------------|---------------------------|-------|-----------------|-----------------|----------------|----------------| | u32_u64_str | bflatn_to_bsatn_fast_path | 100 | 3.5±0.00µs | 3.6±0.01µs | 27.1 Mtx/sec | 26.8 Mtx/sec | | u32_u64_str | bflatn_to_bsatn_slow_path | 100 | 3.6±0.03µs | 3.5±0.00µs | 26.5 Mtx/sec | 26.9 Mtx/sec | | u32_u64_str | bsatn | 100 | 15.4±0.09ns | 15.5±0.10ns | 6.0 Gtx/sec | 6.0 Gtx/sec | | u32_u64_str | bsatn | 100 | 2.4±0.07µs | 2.3±0.02µs | 39.2 Mtx/sec | 40.8 Mtx/sec | | u32_u64_str | json | 100 | 10.4±0.43µs | 8.4±0.05µs | 9.2 Mtx/sec | 11.3 Mtx/sec | | u32_u64_str | json | 100 | 4.8±0.03µs | 4.9±0.05µs | 19.9 Mtx/sec | 19.4 Mtx/sec | | u32_u64_str | product_value | 100 | 1073.6±4.03ns | 1017.7±0.84ns | 88.8 Mtx/sec | 93.7 Mtx/sec | | u32_u64_u64 | bflatn_to_bsatn_fast_path | 100 | 998.0±7.24ns | 950.0±14.12ns | 95.6 Mtx/sec | 100.4 Mtx/sec | | u32_u64_u64 | bflatn_to_bsatn_slow_path | 100 | 2.8±0.00µs | 2.8±0.05µs | 34.0 Mtx/sec | 34.3 Mtx/sec | | u32_u64_u64 | bsatn | 100 | 14.8±0.01ns | 14.8±0.00ns | 6.3 Gtx/sec | 6.3 Gtx/sec | | u32_u64_u64 | bsatn | 100 | 1575.3±26.14ns | 1624.3±27.88ns | 60.5 Mtx/sec | 58.7 Mtx/sec | | u32_u64_u64 | json | 100 | 3.1±0.03µs | 3.3±0.02µs | 30.8 Mtx/sec | 29.1 Mtx/sec | | u32_u64_u64 | json | 100 | 5.6±0.06µs | 6.2±0.05µs | 17.1 Mtx/sec | 15.4 Mtx/sec | | u32_u64_u64 | product_value | 100 | 1073.4±1.93ns | 1015.7±0.52ns | 88.9 Mtx/sec | 93.9 Mtx/sec | | u64_u64_u32 | bflatn_to_bsatn_fast_path | 100 | 750.2±1.46ns | 738.8±0.52ns | 127.1 Mtx/sec | 129.1 Mtx/sec | | u64_u64_u32 | bflatn_to_bsatn_slow_path | 100 | 2.8±0.01µs | 2.8±0.00µs | 34.0 Mtx/sec | 34.3 Mtx/sec | | u64_u64_u32 | bsatn | 100 | 1064.5±1.24ns | 704.4±1.34ns | 89.6 Mtx/sec | 135.4 Mtx/sec | | u64_u64_u32 | bsatn | 100 | 1616.4±25.68ns | 1636.2±24.71ns | 59.0 Mtx/sec | 58.3 Mtx/sec | | u64_u64_u32 | json | 100 | 3.5±0.03µs | 3.4±0.03µs | 27.5 Mtx/sec | 28.1 Mtx/sec | | u64_u64_u32 | json | 100 | 5.7±0.09µs | 6.2±0.23µs | 16.8 Mtx/sec | 15.3 Mtx/sec | | u64_u64_u32 | product_value | 100 | 1072.3±0.87ns | 1015.4±0.47ns | 88.9 Mtx/sec | 93.9 Mtx/sec | ## stdb_module_large_arguments | arg size | new latency | old latency | new throughput | old throughput | |----------|-------------|----------------|----------------|----------------| | 64KiB | - | 101.4±7.67µs | - | - | ## stdb_module_print_bulk | line count | new latency | old latency | new throughput | old throughput | |------------|-------------|----------------|----------------|----------------| | 1 | - | 52.2±5.07µs | - | - | | 100 | - | 596.5±9.84µs | - | - | | 1000 | - | 3.8±0.78ms | - | - | ## remaining | name | new latency | old latency | new throughput | old throughput | |------------------------------------------------------------------------------|-------------------|------------------|----------------|----------------| | special/db_game/circles/load=10 | - | 35.7±2.65ms | - | - | | special/db_game/circles/load=100 | - | 37.0±6.20ms | - | - | | special/db_game/csharp/circles/load=10 | 2.7±0.01s | - | - | - | | special/db_game/csharp/circles/load=100 | 2.7±0.01s | - | - | - | | special/db_game/csharp/ia_loop/load=10 | 360.8±4.56ms | - | - | - | | special/db_game/csharp/ia_loop/load=100 | 940.6±4.84ms | - | - | - | | special/db_game/ia_loop/load=500 | - | 155.7±3.43ms | - | - | | special/db_game/ia_loop/load=5000 | - | 5.7±0.03s | - | - | | special/db_game/rust/circles/load=10 | 49.9±6.18ms | - | - | - | | special/db_game/rust/circles/load=100 | 50.5±4.94ms | - | - | - | | special/db_game/rust/ia_loop/load=10 | 9.3±0.58ms | - | - | - | | special/db_game/rust/ia_loop/load=100 | 11.8±0.27ms | - | - | - | | special/stdb_module/csharp/large_arguments/64KiB | 2.5±0.11ms | - | - | - | | special/stdb_module/csharp/print_bulk/lines=1 | 165.6±7.76µs | - | - | - | | special/stdb_module/csharp/print_bulk/lines=100 | 1918.0±189.98µs | - | - | - | | special/stdb_module/csharp/print_bulk/lines=1000 | 12.2±1.91ms | - | - | - | | special/stdb_module/rust/large_arguments/64KiB | 95.7±9.60µs | - | - | - | | special/stdb_module/rust/print_bulk/lines=1 | 52.4±5.45µs | - | - | - | | special/stdb_module/rust/print_bulk/lines=100 | 594.4±4.52µs | - | - | - | | special/stdb_module/rust/print_bulk/lines=1000 | 5.3±0.04ms | - | - | - | | sqlite/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | - | 54.8±0.07µs | - | 17.8 Ktx/sec | | sqlite/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | - | 46.7±0.24µs | - | 20.9 Ktx/sec | | sqlite/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | - | 39.3±0.22µs | - | 24.9 Ktx/sec | | sqlite/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | - | 35.2±0.14µs | - | 27.8 Ktx/sec | | stdb_module/csharp/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 11.8±0.32ms | - | 84 tx/sec | - | | stdb_module/csharp/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 9.6±0.55ms | - | 104 tx/sec | - | | stdb_module/rust/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 1250.1±11.15µs | - | 799 tx/sec | - | | stdb_module/rust/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 986.9±15.43µs | - | 1013 tx/sec | - | | stdb_module/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | - | 1275.2±11.74µs | - | 784 tx/sec | | stdb_module/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | - | 990.9±9.88µs | - | 1009 tx/sec | | stdb_raw/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 653.0±22.41µs | 629.0±26.03µs | 1531 tx/sec | 1589 tx/sec | | stdb_raw/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 474.5±19.95µs | 445.2±4.53µs | 2.1 Ktx/sec | 2.2 Ktx/sec | | stdb_raw/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 364.4±0.41µs | 368.0±1.39µs | 2.7 Ktx/sec | 2.7 Ktx/sec | | stdb_raw/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 330.6±0.26µs | 327.3±0.85µs | 3.0 Ktx/sec | 3.0 Ktx/sec |

[github-actions[bot]]

Callgrind benchmark results

# Callgrind Benchmark Report These benchmarks were run using [callgrind](https://valgrind.org/docs/manual/cg-manual.html), an instruction-level profiler. They allow comparisons between sqlite (`sqlite`), SpacetimeDB running through a module (`stdb_module`), and the underlying SpacetimeDB data storage engine (`stdb_raw`). Callgrind emulates a CPU to collect the below estimates. Measurement changes larger than five percent are in bold.

In-memory benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6396 | 6399 | -0.05% | 6518 | 6523 | -0.08% | | sqlite | 5579 | 5579 | 0.00% | 6051 | 6019 | 0.53% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76590 | 76593 | -0.00% | 77178 | 77013 | 0.21% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118832 | 119091 | -0.22% | 119538 | 119687 | -0.12% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25080 | 25085 | -0.02% | 25690 | 25617 | 0.28% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24048 | 24051 | -0.01% | 24488 | 24447 | 0.17% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 144695 | 144695 | 0.00% | 146161 | 146257 | -0.07% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 124044 | 124044 | 0.00% | 125236 | 125334 | -0.08% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 131361 | 131361 | 0.00% | 132737 | 132799 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 134494 | 134494 | 0.00% | 136008 | 136104 | -0.07% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 878454 | 877138 | 0.15% | 931272 | 931798 | -0.06% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1025070 | 1024221 | 0.08% | 1057042 | 1065829 | -0.82% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 398320 | 398320 | 0.00% | 415014 | 415186 | -0.04% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 983637 | 983637 | 0.00% | 1022265 | 1020529 | 0.17% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153723 | 153726 | -0.00% | 153809 | 153866 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16748 | 16751 | -0.02% | 16818 | 16875 | -0.34% | | sqlite | u32_u64_str | unique_0 | 1024 | 1068281 | 1067255 | 0.10% | 1071735 | 1070663 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 76261 | 76207 | 0.07% | 77359 | 77445 | -0.11% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 20113042 | 20490259 | -1.84% | 20758694 | 21136189 | -1.79% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1286383 | 1284538 | 0.14% | 1333137 | 1325984 | 0.54% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1802200 | 1802182 | 0.00% | 1811372 | 1811362 | 0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 128528 | 128528 | 0.00% | 131464 | 131492 | -0.02% |

On-disk benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6401 | 6404 | -0.05% | 6539 | 6544 | -0.08% | | sqlite | 5621 | 5621 | 0.00% | 6161 | 6121 | 0.65% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76595 | 76598 | -0.00% | 77143 | 76978 | 0.21% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 118837 | 119096 | -0.22% | 119583 | 119884 | -0.25% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25475 | 25088 | 1.54% | 26129 | 25572 | 2.18% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24053 | 24056 | -0.01% | 24445 | 24404 | 0.17% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 125965 | 125965 | 0.00% | 127553 | 127543 | 0.01% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 146616 | 146616 | 0.00% | 148426 | 148450 | -0.02% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 136616 | 136616 | 0.00% | 138660 | 138724 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 133457 | 133457 | 0.00% | 135335 | 135341 | -0.00% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 826862 | 826099 | 0.09% | 879458 | 879673 | -0.02% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 977123 | 976494 | 0.06% | 1040351 | 1047160 | -0.65% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 415857 | 415857 | 0.00% | 431801 | 431867 | -0.02% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1021898 | 1021898 | 0.00% | 1059350 | 1057622 | 0.16% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153728 | 153731 | -0.00% | 153798 | 153855 | -0.04% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16753 | 16756 | -0.02% | 16819 | 16880 | -0.36% | | sqlite | u32_u64_str | unique_0 | 1024 | 1071343 | 1070323 | 0.10% | 1075181 | 1074097 | 0.10% | | sqlite | u32_u64_str | unique_0 | 64 | 78033 | 77973 | 0.08% | 79431 | 79303 | 0.16% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50316 | 50282 | 0.07% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27787 | 27821 | -0.12% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9594 | 9628 | -0.35% | | 16 | json | 12188 | 12188 | 0.00% | 14228 | 14194 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 19029789 | 19001267 | 0.15% | 19739015 | 19640773 | 0.50% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1239254 | 1238283 | 0.08% | 1316814 | 1309651 | 0.55% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1809743 | 1809743 | 0.00% | 1818475 | 1818455 | 0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 132654 | 132654 | 0.00% | 135602 | 135834 | -0.17% |

[RReverser]

benchmarks please

[github-actions[bot]]

Callgrind benchmark in progress...

[github-actions[bot]]

Benchmark in progress...

[RReverser]

The failure here is the one that will be fixed by #1987.

[github-actions[bot]]

Criterion benchmark results

# Criterion benchmark report **YOU SHOULD PROBABLY IGNORE THESE RESULTS.** Criterion is a wall time based benchmarking system that is extremely noisy when run on CI. We collect these results for longitudinal analysis, but they are not reliable for comparing individual PRs. Go look at the callgrind report instead. ## empty | db | on disk | new latency | old latency | new throughput | old throughput | |----------|---------|---------------|---------------|----------------|----------------| | sqlite | 💿 | - | 421.3±1.80ns | - | - | | sqlite | 🧠 | - | 413.6±1.18ns | - | - | | stdb_raw | 💿 | 773.1±0.78ns | 779.3±3.76ns | - | - | | stdb_raw | 🧠 | 775.9±0.85ns | 777.7±1.16ns | - | - | ## insert_1 | db | on disk | schema | indices | preload | new latency | old latency | new throughput | old throughput | |----|---------|--------|---------|---------|-------------|-------------|----------------|----------------| ## insert_bulk | db | on disk | schema | indices | preload | count | new latency | old latency | new throughput | old throughput | |----------|---------|-------------|-------------------|---------|-------|-----------------|-----------------|----------------|----------------| | sqlite | 💿 | u32_u64_str | btree_each_column | 2048 | 256 | - | 584.6±0.45µs | - | 1710 tx/sec | | sqlite | 💿 | u32_u64_str | unique_0 | 2048 | 256 | - | 150.0±0.52µs | - | 6.5 Ktx/sec | | sqlite | 💿 | u32_u64_u64 | btree_each_column | 2048 | 256 | - | 480.0±42.66µs | - | 2.0 Ktx/sec | | sqlite | 💿 | u32_u64_u64 | unique_0 | 2048 | 256 | - | 136.6±1.01µs | - | 7.1 Ktx/sec | | sqlite | 🧠 | u32_u64_str | btree_each_column | 2048 | 256 | - | 447.6±3.58µs | - | 2.2 Ktx/sec | | sqlite | 🧠 | u32_u64_str | unique_0 | 2048 | 256 | - | 126.9±0.36µs | - | 7.7 Ktx/sec | | sqlite | 🧠 | u32_u64_u64 | btree_each_column | 2048 | 256 | - | 368.0±0.24µs | - | 2.7 Ktx/sec | | sqlite | 🧠 | u32_u64_u64 | unique_0 | 2048 | 256 | - | 104.6±0.54µs | - | 9.3 Ktx/sec | | stdb_raw | 💿 | u32_u64_str | btree_each_column | 2048 | 256 | 591.9±26.82µs | 601.7±20.50µs | 1689 tx/sec | 1661 tx/sec | | stdb_raw | 💿 | u32_u64_str | unique_0 | 2048 | 256 | 485.7±25.34µs | 473.7±19.44µs | 2.0 Ktx/sec | 2.1 Ktx/sec | | stdb_raw | 💿 | u32_u64_u64 | btree_each_column | 2048 | 256 | 354.8±26.21µs | 376.0±9.37µs | 2.8 Ktx/sec | 2.6 Ktx/sec | | stdb_raw | 💿 | u32_u64_u64 | unique_0 | 2048 | 256 | 337.8±21.79µs | 343.8±8.88µs | 2.9 Ktx/sec | 2.8 Ktx/sec | | stdb_raw | 🧠 | u32_u64_str | btree_each_column | 2048 | 256 | 299.7±0.25µs | 297.9±0.21µs | 3.3 Ktx/sec | 3.3 Ktx/sec | | stdb_raw | 🧠 | u32_u64_str | unique_0 | 2048 | 256 | 231.9±0.33µs | 228.8±0.41µs | 4.2 Ktx/sec | 4.3 Ktx/sec | | stdb_raw | 🧠 | u32_u64_u64 | btree_each_column | 2048 | 256 | 239.2±0.10µs | 239.1±0.20µs | 4.1 Ktx/sec | 4.1 Ktx/sec | | stdb_raw | 🧠 | u32_u64_u64 | unique_0 | 2048 | 256 | 212.5±0.39µs | 207.0±0.25µs | 4.6 Ktx/sec | 4.7 Ktx/sec | ## iterate | db | on disk | schema | indices | new latency | old latency | new throughput | old throughput | |----------|---------|-------------|----------|--------------|---------------|----------------|----------------| | sqlite | 💿 | u32_u64_str | unique_0 | - | 23.4±0.11µs | - | 41.8 Ktx/sec | | sqlite | 💿 | u32_u64_u64 | unique_0 | - | 22.1±0.06µs | - | 44.2 Ktx/sec | | sqlite | 🧠 | u32_u64_str | unique_0 | - | 20.9±0.08µs | - | 46.6 Ktx/sec | | sqlite | 🧠 | u32_u64_u64 | unique_0 | - | 19.5±0.08µs | - | 50.2 Ktx/sec | | stdb_raw | 💿 | u32_u64_str | unique_0 | 4.9±0.00µs | 4.0±0.00µs | 199.6 Ktx/sec | 245.9 Ktx/sec | | stdb_raw | 💿 | u32_u64_u64 | unique_0 | 4.8±0.00µs | 3.9±0.00µs | 203.7 Ktx/sec | 251.8 Ktx/sec | | stdb_raw | 🧠 | u32_u64_str | unique_0 | 4.9±0.00µs | 4.0±0.00µs | 199.2 Ktx/sec | 246.0 Ktx/sec | | stdb_raw | 🧠 | u32_u64_u64 | unique_0 | 4.8±0.00µs | 3.9±0.00µs | 203.8 Ktx/sec | 251.8 Ktx/sec | ## find_unique | db | on disk | key type | preload | new latency | old latency | new throughput | old throughput | |----|---------|----------|---------|-------------|-------------|----------------|----------------| ## filter | db | on disk | key type | index strategy | load | count | new latency | old latency | new throughput | old throughput | |----------|---------|----------|----------------|------|-------|--------------|---------------|----------------|----------------| | sqlite | 💿 | string | index | 2048 | 256 | - | 70.4±0.35µs | - | 13.9 Ktx/sec | | sqlite | 💿 | u64 | index | 2048 | 256 | - | 66.9±0.12µs | - | 14.6 Ktx/sec | | sqlite | 🧠 | string | index | 2048 | 256 | - | 66.0±0.16µs | - | 14.8 Ktx/sec | | sqlite | 🧠 | u64 | index | 2048 | 256 | - | 60.0±0.28µs | - | 16.3 Ktx/sec | | stdb_raw | 💿 | string | index | 2048 | 256 | 4.9±0.00µs | 5.0±0.00µs | 197.4 Ktx/sec | 195.7 Ktx/sec | | stdb_raw | 💿 | u64 | index | 2048 | 256 | 4.9±0.00µs | 4.9±0.00µs | 201.1 Ktx/sec | 198.8 Ktx/sec | | stdb_raw | 🧠 | string | index | 2048 | 256 | 4.9±0.00µs | 5.0±0.00µs | 197.5 Ktx/sec | 195.5 Ktx/sec | | stdb_raw | 🧠 | u64 | index | 2048 | 256 | 4.8±0.00µs | 4.9±0.00µs | 201.4 Ktx/sec | 198.7 Ktx/sec | ## serialize | schema | format | count | new latency | old latency | new throughput | old throughput | |-------------|---------------------------|-------|-----------------|-----------------|----------------|----------------| | u32_u64_str | bflatn_to_bsatn_fast_path | 100 | 3.6±0.00µs | 3.9±0.01µs | 26.6 Mtx/sec | 24.7 Mtx/sec | | u32_u64_str | bflatn_to_bsatn_slow_path | 100 | 2.9±0.02µs | 3.5±0.01µs | 32.8 Mtx/sec | 27.0 Mtx/sec | | u32_u64_str | bsatn | 100 | 16.0±0.04ns | 15.6±0.17ns | 5.8 Gtx/sec | 6.0 Gtx/sec | | u32_u64_str | bsatn | 100 | 2.3±0.01µs | 2.4±0.00µs | 41.3 Mtx/sec | 40.4 Mtx/sec | | u32_u64_str | json | 100 | 5.0±0.05µs | 5.0±0.03µs | 19.0 Mtx/sec | 19.2 Mtx/sec | | u32_u64_str | json | 100 | 8.0±0.10µs | 9.6±0.02µs | 11.9 Mtx/sec | 10.0 Mtx/sec | | u32_u64_str | product_value | 100 | 1019.7±9.21ns | 1019.1±1.23ns | 93.5 Mtx/sec | 93.6 Mtx/sec | | u32_u64_u64 | bflatn_to_bsatn_fast_path | 100 | 972.9±5.57ns | 1010.7±8.08ns | 98.0 Mtx/sec | 94.4 Mtx/sec | | u32_u64_u64 | bflatn_to_bsatn_slow_path | 100 | 2.4±0.00µs | 2.8±0.00µs | 39.5 Mtx/sec | 34.5 Mtx/sec | | u32_u64_u64 | bsatn | 100 | 15.6±0.01ns | 14.8±0.01ns | 6.0 Gtx/sec | 6.3 Gtx/sec | | u32_u64_u64 | bsatn | 100 | 1629.1±30.64ns | 1826.3±19.44ns | 58.5 Mtx/sec | 52.2 Mtx/sec | | u32_u64_u64 | json | 100 | 3.2±0.06µs | 3.4±0.04µs | 30.1 Mtx/sec | 27.8 Mtx/sec | | u32_u64_u64 | json | 100 | 5.8±0.12µs | 5.9±0.08µs | 16.4 Mtx/sec | 16.3 Mtx/sec | | u32_u64_u64 | product_value | 100 | 1015.8±0.57ns | 1015.7±0.71ns | 93.9 Mtx/sec | 93.9 Mtx/sec | | u64_u64_u32 | bflatn_to_bsatn_fast_path | 100 | 742.3±4.33ns | 753.0±0.32ns | 128.5 Mtx/sec | 126.6 Mtx/sec | | u64_u64_u32 | bflatn_to_bsatn_slow_path | 100 | 2.4±0.04µs | 2.8±0.00µs | 39.4 Mtx/sec | 34.3 Mtx/sec | | u64_u64_u32 | bsatn | 100 | 1098.7±0.64ns | 923.9±3.27ns | 86.8 Mtx/sec | 103.2 Mtx/sec | | u64_u64_u32 | bsatn | 100 | 1606.5±74.45ns | 1715.6±19.68ns | 59.4 Mtx/sec | 55.6 Mtx/sec | | u64_u64_u32 | json | 100 | 3.5±0.06µs | 3.3±0.17µs | 27.6 Mtx/sec | 29.0 Mtx/sec | | u64_u64_u32 | json | 100 | 5.9±0.07µs | 5.8±0.01µs | 16.3 Mtx/sec | 16.4 Mtx/sec | | u64_u64_u32 | product_value | 100 | 1015.1±1.19ns | 1014.6±0.68ns | 94.0 Mtx/sec | 94.0 Mtx/sec | ## stdb_module_large_arguments | arg size | new latency | old latency | new throughput | old throughput | |----------|-------------|-----------------|----------------|----------------| | 64KiB | - | 102.2±10.40µs | - | - | ## stdb_module_print_bulk | line count | new latency | old latency | new throughput | old throughput | |------------|-------------|-----------------|----------------|----------------| | 1 | - | 52.4±6.81µs | - | - | | 100 | - | 598.3±12.12µs | - | - | | 1000 | - | 4.7±0.80ms | - | - | ## remaining | name | new latency | old latency | new throughput | old throughput | |------------------------------------------------------------------------------|-------------------|------------------|----------------|----------------| | special/db_game/circles/load=10 | - | 51.3±2.38ms | - | - | | special/db_game/circles/load=100 | - | 36.1±1.22ms | - | - | | special/db_game/csharp/circles/load=10 | 2.7±0.01s | - | - | - | | special/db_game/csharp/circles/load=100 | 2.7±0.00s | - | - | - | | special/db_game/csharp/ia_loop/load=10 | 356.5±1.52ms | - | - | - | | special/db_game/csharp/ia_loop/load=100 | 931.5±6.14ms | - | - | - | | special/db_game/ia_loop/load=500 | - | 148.1±0.81ms | - | - | | special/db_game/ia_loop/load=5000 | - | 5.3±0.05s | - | - | | special/db_game/rust/circles/load=10 | 43.4±1.85ms | - | - | - | | special/db_game/rust/circles/load=100 | 50.1±4.56ms | - | - | - | | special/db_game/rust/ia_loop/load=10 | 9.4±0.39ms | - | - | - | | special/db_game/rust/ia_loop/load=100 | 11.9±0.62ms | - | - | - | | special/stdb_module/csharp/large_arguments/64KiB | 2.1±0.43ms | - | - | - | | special/stdb_module/csharp/print_bulk/lines=1 | 170.3±8.60µs | - | - | - | | special/stdb_module/csharp/print_bulk/lines=100 | 1883.1±242.80µs | - | - | - | | special/stdb_module/csharp/print_bulk/lines=1000 | 12.0±1.84ms | - | - | - | | special/stdb_module/rust/large_arguments/64KiB | 99.8±7.46µs | - | - | - | | special/stdb_module/rust/print_bulk/lines=1 | 54.3±4.68µs | - | - | - | | special/stdb_module/rust/print_bulk/lines=100 | 605.3±11.50µs | - | - | - | | special/stdb_module/rust/print_bulk/lines=1000 | 3.6±0.33ms | - | - | - | | sqlite/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | - | 54.3±0.20µs | - | 18.0 Ktx/sec | | sqlite/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | - | 46.3±0.09µs | - | 21.1 Ktx/sec | | sqlite/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | - | 39.6±0.13µs | - | 24.7 Ktx/sec | | sqlite/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | - | 34.7±0.11µs | - | 28.1 Ktx/sec | | stdb_module/csharp/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 11.9±0.07ms | - | 84 tx/sec | - | | stdb_module/csharp/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 9.9±0.66ms | - | 100 tx/sec | - | | stdb_module/rust/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 1250.3±13.71µs | - | 799 tx/sec | - | | stdb_module/rust/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 994.9±18.13µs | - | 1005 tx/sec | - | | stdb_module/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | - | 1242.6±13.61µs | - | 804 tx/sec | | stdb_module/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | - | 998.1±5.79µs | - | 1001 tx/sec | | stdb_raw/💿/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 633.0±18.61µs | 615.3±41.19µs | 1579 tx/sec | 1625 tx/sec | | stdb_raw/💿/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 428.2±6.87µs | 480.8±7.76µs | 2.3 Ktx/sec | 2.0 Ktx/sec | | stdb_raw/🧠/update_bulk/u32_u64_str/unique_0/load=2048/count=256 | 374.4±0.17µs | 363.1±0.20µs | 2.6 Ktx/sec | 2.7 Ktx/sec | | stdb_raw/🧠/update_bulk/u32_u64_u64/unique_0/load=2048/count=256 | 335.7±0.27µs | 323.5±0.46µs | 2.9 Ktx/sec | 3.0 Ktx/sec |

[github-actions[bot]]

Callgrind benchmark results

# Callgrind Benchmark Report These benchmarks were run using [callgrind](https://valgrind.org/docs/manual/cg-manual.html), an instruction-level profiler. They allow comparisons between sqlite (`sqlite`), SpacetimeDB running through a module (`stdb_module`), and the underlying SpacetimeDB data storage engine (`stdb_raw`). Callgrind emulates a CPU to collect the below estimates. Measurement changes larger than five percent are in bold.

In-memory benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | 6397 | 6397 | 0.00% | 6519 | 6527 | -0.12% | | sqlite | 5579 | 5589 | -0.18% | 5941 | 6117 | -2.88% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76591 | 76591 | 0.00% | 77039 | 77031 | 0.01% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 119089 | 120178 | -0.91% | 119855 | 120932 | -0.89% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25082 | 25081 | 0.00% | 25604 | 25735 | -0.51% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24049 | 24049 | 0.00% | 24445 | 24609 | -0.67% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 144695 | 144695 | 0.00% | 146185 | 146181 | 0.00% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 124044 | 124044 | 0.00% | 125192 | 125272 | -0.06% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 131361 | 131361 | 0.00% | 132729 | 132857 | -0.10% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 134494 | 134494 | 0.00% | 136124 | 136176 | -0.04% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 878645 | 878993 | -0.04% | 898727 | 934437 | -3.82% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1025919 | 1031446 | -0.54% | 1062275 | 1098746 | -3.32% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 398320 | 398320 | 0.00% | 414892 | 414994 | -0.02% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 983637 | 983637 | 0.00% | 1022751 | 1021691 | 0.10% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153724 | 153724 | 0.00% | 153840 | 153866 | -0.02% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16749 | 16749 | 0.00% | 16849 | 16871 | -0.13% | | sqlite | u32_u64_str | unique_0 | 1024 | 1068275 | 1067255 | 0.10% | 1071515 | 1070691 | 0.08% | | sqlite | u32_u64_str | unique_0 | 64 | 76261 | 76201 | 0.08% | 77287 | 77335 | -0.06% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50218 | 50180 | 0.08% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27719 | 27719 | 0.00% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9526 | 9526 | 0.00% | | 16 | json | 12188 | 12188 | 0.00% | 14126 | 14092 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 20119200 | 20120482 | -0.01% | 20640060 | 20703592 | -0.31% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1287320 | 1287068 | 0.02% | 1326058 | 1329642 | -0.27% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1802182 | 1802182 | 0.00% | 1811458 | 1811474 | -0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 128528 | 128528 | 0.00% | 131398 | 131416 | -0.01% |

On-disk benchmarks

### callgrind: empty transaction | db | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | 6402 | 6402 | 0.00% | 6536 | 6536 | 0.00% | | sqlite | 5621 | 5621 | 0.00% | 6047 | 6175 | -2.07% | ### callgrind: filter | db | schema | indices | count | preload | _column | data_type | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|---------|-----------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 76596 | 76596 | 0.00% | 77024 | 77012 | 0.02% | | stdb_raw | u32_u64_str | no_index | 64 | 128 | 2 | string | 120183 | 119094 | 0.91% | 121009 | 119764 | 1.04% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 25086 | 25086 | 0.00% | 25572 | 25728 | -0.61% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 24054 | 24054 | 0.00% | 24422 | 24582 | -0.65% | | sqlite | u32_u64_str | no_index | 64 | 128 | 1 | u64 | 125965 | 125971 | -0.00% | 127521 | 127555 | -0.03% | | sqlite | u32_u64_str | no_index | 64 | 128 | 2 | string | 146616 | 146616 | 0.00% | 148418 | 148458 | -0.03% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 2 | string | 136616 | 136616 | 0.00% | 138668 | 138824 | -0.11% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1 | u64 | 133457 | 133457 | 0.00% | 135319 | 135343 | -0.02% | ### callgrind: insert bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|-------------------|-------|---------|----------------------|--------------------------|--------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 827614 | 828159 | -0.07% | 877486 | 882163 | -0.53% | | stdb_raw | u32_u64_str | btree_each_column | 64 | 128 | 977349 | 978051 | -0.07% | 1040467 | 1042435 | -0.19% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 415857 | 415857 | 0.00% | 431631 | 431843 | -0.05% | | sqlite | u32_u64_str | btree_each_column | 64 | 128 | 1021898 | 1021898 | 0.00% | 1059450 | 1059298 | 0.01% | ### callgrind: iterate | db | schema | indices | count | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 153729 | 153729 | 0.00% | 153825 | 153851 | -0.02% | | stdb_raw | u32_u64_str | unique_0 | 64 | 16754 | 16754 | 0.00% | 16850 | 16872 | -0.13% | | sqlite | u32_u64_str | unique_0 | 1024 | 1071343 | 1070323 | 0.10% | 1075045 | 1074105 | 0.09% | | sqlite | u32_u64_str | unique_0 | 64 | 78033 | 77973 | 0.08% | 79275 | 79263 | 0.02% | ### callgrind: serialize_product_value | count | format | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |-------|--------|----------------------|--------------------------|-------|------------------|----------------------|----------| | 64 | json | 47528 | 47528 | 0.00% | 50218 | 50180 | 0.08% | | 64 | bsatn | 25509 | 25509 | 0.00% | 27719 | 27719 | 0.00% | | 16 | bsatn | 8200 | 8200 | 0.00% | 9526 | 9526 | 0.00% | | 16 | json | 12188 | 12188 | 0.00% | 14126 | 14092 | 0.24% | ### callgrind: update bulk | db | schema | indices | count | preload | total reads + writes | old total reads + writes | Δrw | estimated cycles | old estimated cycles | Δcycles | |----------|-------------|----------|-------|---------|----------------------|--------------------------|-------|------------------|----------------------|----------| | stdb_raw | u32_u64_str | unique_0 | 1024 | 1024 | 19037270 | 19035674 | 0.01% | 19625194 | 19691388 | -0.34% | | stdb_raw | u32_u64_str | unique_0 | 64 | 128 | 1240913 | 1239755 | 0.09% | 1308873 | 1312385 | -0.27% | | sqlite | u32_u64_str | unique_0 | 1024 | 1024 | 1809743 | 1809743 | 0.00% | 1818367 | 1818423 | -0.00% | | sqlite | u32_u64_str | unique_0 | 64 | 128 | 132654 | 132654 | 0.00% | 135556 | 135606 | -0.04% |

[gefjon]

Internal tests failures appear to be spurious/unrelated, but let's wait to merge until they're fixed just in case.

[RReverser]

There are some regexes here that need to be updated with new strings for this, to get the auto-generated comments fixed. This is, unfortunately, kind of a pain to test, so I can do it if needed.

Yeah if you could, please do.

Internal tests failures appear to be spurious/unrelated, but let's wait to merge until they're fixed just in case.

SGTM.

[bfops]

I hit the "Update branch" button because I believe it will make the internal tests pass.

[RReverser]

It did, thanks.