Open srosenberg opened 1 month ago
An initial discussion thread is in slack [1]. After @RaduBerinde's suggestion to investigate a "faster unzip library", and "something that uses all CPUs", a new rabbit hole was waiting to be discovered...
pgzip
The first attempt was to replace gzip
with pgzip
, in order to take advantage of all available CPU cores. This yielded only a small speed-up, despite trying a number of custom parameters. Thus, to reduce the init
latency, we must find a faster decompressor for ~11MB (of icons and minified js).
zstd
Thanks to the native Go implementations [2], switching between gzip
, pgzip
, and zstd
is a piece of cake. Surprisingly, zstd
was able to compress it down to ~4MB
!?
ls -lh /private/var/tmp/_bazel_srosenberg/567ebe324bb100ad2d71e73476e77acb/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/ui/distccl/assets.tar.gz
-r-xr-xr-x 1 srosenberg wheel 4.0M Jul 10 00:22 /private/var/tmp/_bazel_srosenberg/567ebe324bb100ad2d71e73476e77acb/execroot/com_github_cockroachdb_cockroach/bazel-out/darwin_arm64-fastbuild/bin/pkg/ui/distccl/assets.tar.gz
The reader might be wondering if that's a typo. It isn't. Both gzip
and zstd
are LZ77
variants, i.e., dictionary-based encoders with Huffman coding. In addition, zstd
uses Finite State Entropy (FSE); i.e., its entropy encoding beats gzip
. Except, disabling entropy encoder (WithNoEntropyCompression(true)
) still results in ~4MB
. Something doesn't add up...
After some trial and error, and spelunking in the zstd
sources, we have our answer–gzip
maximum window size is limited to 32KB
, whereas zstd
can go as high as 512MB
; the default is conservatively set at 8MB
. Since increasing the window was instrumental in reducing the size, it suggests that there might be fairly large, duplicated UI assets (more on that later).
Thus, having reduced the size of the packed UI assets to 4MB
, and using zstd
's fast decompressor, the init
time becomes fairly reasonable. The init latency on m1pro
is now ~38ms
,
GODEBUG=inittrace=1 ./cockroach version 2>&1|grep init|grep pkg/ui |awk '{print $5}' |tr -d '@'|awk '{sum += $0} END {print sum}'
38.019
whereas the total latency is ~128ms
,
GODEBUG=inittrace=1 ./cockroach version 2>&1|grep init|awk '{print $5}' |tr -d '@'|awk '{sum += $0} END {print sum}'
128.886
Looking at assets.tar.gz
, we observe that bundle.js
weighs in at a whopping ~22MB
. Armed with a plain vanilla LZ77.go
and a large window, we find a number of interesting offsets, where something large is being duplicated,
cat /tmp/offsets|awk '{print $2}' |sort -urn |head -25
355300
355024
354725
351613
349835
349834
349831
349804
349733
The top entry, 355300
maps to the file offset of 11429984
. Let's see what's there,
tail -c +11429984 bundle.js |less
reveals a base-64 encoded image with the following prefix,
"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
In fact, all of the above window sizes map to the same base-64 encoded image. What is this mysterious image, and why is it repeated several times? After much spelunking and reverse engineering an SVG, the mysterious image is a rectangular gradient 262KB
in size 🤯
Indeed, the image is used as a mask to apply gradient to an otherwise dark cockroach logo, denoted by the SVG path in [3]. This begs a question–is this technique sound? Why couldn't we use a smaller image to get the same gradient effect? My guess is this code was generated by some third-party tool, perhaps Figma? 😉 It appears to have landed in master as of [4]. Finally, we have multiple styles of the same logo, hence the duplication of the same image. These styled icons landed as of [5]. Searching for the image prefix, we get the following hits,
grep -r -o iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP pkg/ui
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachdbDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachdbLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/db-console/src/components/icon/cockroachLabsLockupIcon.tsx:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachdbDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachdbLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachMarkLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudDarkFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachLabsLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
pkg/ui/workspaces/cluster-ui/node_modules/@cockroachlabs/icons/dist/components/CockroachcloudLightFull.js:iVBORw0KGgoAAAANSUhEUgAAA2EAAAQ4CAIAAAAPSEP
zstd
is a clear winner. (Always prefer it over gzip
.) 4MB
is a lot more manageable than 11MB
, bringing down the total initialization time to ~128ms
on m1pro
. Meanwhile, we should be careful with merging large assets, considering its side-effects of ballooning the size of the cockroach binary and increasing the startup time. The incoming PR will now fail the build if assets.tar.gz
ever reaches 8MB
again.
[1] https://cockroachlabs.slack.com/archives/C4X2J0RH6/p1719953997816919 [2] https://github.com/klauspost/compress [3] https://github.com/cockroachdb/cockroach/blob/0b918d1dc3a9ce1f04975202be3b04ec375a816e/pkg/ui/workspaces/db-console/src/components/icon/cockroachLabsLockupIcon.tsx#L48-L51 [4] https://github.com/cockroachdb/cockroach/pull/57130 [5] https://github.com/cockroachdb/ui/pull/441
bincheck verification should alert us next time. Note that another side-effect of unpacking the UI assets is a fairly large memory footprint: ~265MB
vs. ~67MB
(without).
ui.Assets
are unpacked duringinit
,At this time, their gzip-compressed size is ~
11MB
,The
init
latency onm1pro
is ~139ms
,which is > 50% of the total
init
latency for the entire binary,Jira issue: CRDB-40183