search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.72k stars 3.75k forks source link

ui: gzip-compressed assets incur a hefty `init` latency #126916

Open srosenberg opened 1 month ago

srosenberg commented 1 month ago

ui.Assets are unpacked during init,

//go:embed assets.tar.gz
var assets []byte

func init() {
        fs, err := targz.AsFS(bytes.NewBuffer(assets))
        if err != nil {
                panic(err)
        }
        ui.Assets = fs
        ui.HaveUI = true
}

At this time, their gzip-compressed size is ~11MB,

ls -lh  /private/var/tmp/_bazel_srosenberg/567ebe324bb100ad2d71e73476e77acb/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/ui/distccl/assets.tar.gz
-r-xr-xr-x  1 srosenberg  wheel    11M Jul 10 00:00 /private/var/tmp/_bazel_srosenberg/567ebe324bb100ad2d71e73476e77acb/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/ui/distccl/assets.tar.gz

The init latency on m1pro is ~139ms,

GODEBUG=inittrace=1 ./cockroach version 2>&1|grep init|grep pkg/ui |awk '{print $5}' |tr -d '@'|awk '{sum += $0} END {print sum}'
139.02

which is > 50% of the total init latency for the entire binary,

GODEBUG=inittrace=1 ./cockroach version 2>&1|grep init|awk '{print $5}' |tr -d '@'|awk '{sum += $0} END {print sum}'
237.397

Jira issue: CRDB-40183

srosenberg commented 1 month ago

An initial discussion thread is in slack [1]. After @RaduBerinde's suggestion to investigate a "faster unzip library", and "something that uses all CPUs", a new rabbit hole was waiting to be discovered...

pgzip

The first attempt was to replace gzip with pgzip, in order to take advantage of all available CPU cores. This yielded only a small speed-up, despite trying a number of custom parameters. Thus, to reduce the init latency, we must find a faster decompressor for ~11MB (of icons and minified js).

zstd

Thanks to the native Go implementations [2], switching between gzip, pgzip, and zstd is a piece of cake. Surprisingly, zstd was able to compress it down to ~4MB!?

ls -lh  /private/var/tmp/_bazel_srosenberg/567ebe324bb100ad2d71e73476e77acb/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/ui/distccl/assets.tar.gz
-r-xr-xr-x  1 srosenberg  wheel   4.0M Jul 10 00:22 /private/var/tmp/_bazel_srosenberg/567ebe324bb100ad2d71e73476e77acb/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/ui/distccl/assets.tar.gz

The reader might be wondering if that's a typo. It isn't. Both gzip and zstd are LZ77 variants, i.e., dictionary-based encoders with Huffman coding. In addition, zstd uses Finite State Entropy (FSE); i.e., its entropy encoding beats gzip. Except, disabling entropy encoder (WithNoEntropyCompression(true)) still results in ~4MB. Something doesn't add up...

After some trial and error, and spelunking in the zstd sources, we have our answer–gzip maximum window size is limited to 32KB, whereas zstd can go as high as 512MB; the default is conservatively set at 8MB. Since increasing the window was instrumental in reducing the size, it suggests that there might be fairly large, duplicated UI assets (more on that later).

Thus, having reduced the size of the packed UI assets to 4MB, and using zstd's fast decompressor, the init time becomes fairly reasonable. The init latency on m1pro is now ~38ms,

GODEBUG=inittrace=1 ./cockroach version 2>&1|grep init|grep pkg/ui |awk '{print $5}' |tr -d '@'|awk '{sum += $0} END {print sum}'
38.019

whereas the total latency is ~128ms,

GODEBUG=inittrace=1 ./cockroach version 2>&1|grep init|awk '{print $5}' |tr -d '@'|awk '{sum += $0} END {print sum}'
128.886

Cockroach Logo with a Gradient

Looking at assets.tar.gz, we observe that bundle.js weighs in at a whopping ~22MB. Armed with a plain vanilla LZ77.go and a large window, we find a number of interesting offsets, where something large is being duplicated,

cat /tmp/offsets|awk '{print $2}' |sort -urn |head -25
355300
355024
354725
351613
349835
349834
349831
349804
349733

The top entry, 355300 maps to the file offset of 11429984. Let's see what's there,

tail -c +11429984 bundle.js |less

reveals a base-64 encoded image with the following prefix,

"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP

In fact, all of the above window sizes map to the same base-64 encoded image. What is this mysterious image, and why is it repeated several times? After much spelunking and reverse engineering an SVG, the mysterious image is a rectangular gradient 262KB in size 🤯

cockroach_logo_svg_screenshot

Indeed, the image is used as a mask to apply gradient to an otherwise dark cockroach logo, denoted by the SVG path in [3]. This begs a question–is this technique sound? Why couldn't we use a smaller image to get the same gradient effect? My guess is this code was generated by some third-party tool, perhaps Figma? 😉 It appears to have landed in master as of [4]. Finally, we have multiple styles of the same logo, hence the duplication of the same image. These styled icons landed as of [5]. Searching for the image prefix, we get the following hits,

grep -r -o iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP pkg/ui
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachdbDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachdbLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/src/components/icon/cockroachLabsLockupIcon.tsx:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachdbDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachdbLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP

Conclusion

zstd is a clear winner. (Always prefer it over gzip.) 4MB is a lot more manageable than 11MB, bringing down the total initialization time to ~128ms on m1pro. Meanwhile, we should be careful with merging large assets, considering its side-effects of ballooning the size of the cockroach binary and increasing the startup time. The incoming PR will now fail the build if assets.tar.gz ever reaches 8MB again.

[1] https://cockroachlabs.slack.com/archives/C4X2J0RH6/p1719953997816919 [2] https://github.com/klauspost/compress [3] https://github.com/cockroachdb/cockroach/blob/0b918d1dc3a9ce1f04975202be3b04ec375a816e/pkg/ui/workspaces/db-console/src/components/icon/cockroachLabsLockupIcon.tsx#L48-L51 [4] https://github.com/cockroachdb/cockroach/pull/57130 [5] https://github.com/cockroachdb/ui/pull/441

srosenberg commented 1 month ago

bincheck verification should alert us next time. Note that another side-effect of unpacking the UI assets is a fairly large memory footprint: ~265MB vs. ~67MB (without).