Feature: Smaller surrealdb.wasm binary with more feature flags

[nickchomey]

Is your feature request related to a problem?

Surrealdb.wasm is 14MB, which is quite large. It can be compressed with brotli or zstd down to about 2.5-3.5MB (depending on compression level), which is more palatable. But still large, especially if you want to make surrealdb at the core of your webapp and it therefore needs to download and load as quickly as possible. This is even moreso the case for users who have slow and/or expensive bandwidth.

Also, it isn't guaranteed that compression will always happen. Literally right now https://surrealist.app is not compressing the wasm, so it takes 5+ seconds to download and 10+ seconds for the whole site to be ready. And it was 30 seconds on my phone!

Describe the solution

SurrealDB is massively comprehensive, and it is highly unlikely that every application will make use of its full range of features. This is especially the case for when Surrealdb is to be run in a browser - particularly on mobile devices with limited storage and compute capability.

For example, I very much doubt that I'll want to offer my users the ability to do any of the machine learning or vector search stuff - I will reserve such things for my backend servers. In fact, I might even want to leave graph capabilities out and simply use Surrealdb.wasm as a document/kv store with full text search and live queries (something that isn't all that possible with existing javascript-based indexeddb libraries)

So, I wonder if there could be a way to offer some build config flags that allow us to only include particular features/modules in the binary? I'm sure some features simply cannot be extricated from the database, but surely some of them might be?

Alternative methods

Brotli and zstd (which is mostly supported by browsers now) can get it down to a palatable 2.5-3.5MB. But if we can "tree-shake" the binary, that would be ideal.

And shared dictionary compression is in early phases of testing, which will make static asset sizes nearly inconsequential. But it could be a while til this is ubiquitous.

SurrealDB version

all versions

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[kearfy]

Hey @nickchomey, making the bundle smaller is quite complex, unfortunately! It is a challenge that we need to take on in the core surrealdb repo, however the bundle will always stay at least a couple megabytes, simply because you're running SurrealDB entirely in your browser. Allthough I cannot provide a timeline on this, rest assured that this issue is definitely on our radar. We'll definitely take your suggestion into account. Thanks!

[nickchomey]

Thanks for the consideration!

And I suppose that I'm not really describing "tree-shaking" so much as just something akin to ES Modules - pick and choose which things to import to begin with.

I see now that in cargo.toml, there's a features section that you can select from with wasm-pack

[features]
kv-indxdb = ["surrealdb/kv-indxdb"]
kv-mem = ["surrealdb/kv-mem"]
stack-traces = ["dep:console_error_panic_hook"]

"compile": "wasm-pack build --release --target deno --out-name index --out-dir compiled/embedded --no-default-features --features kv-indxdb,kv-mem",

As with js libraries with ES Modules, I'm sure there would need to be a surrealdb-core module (and it might need to include a lot of the functionality), but hopefully modules/features for some things like surrealml, vectors, graph, full-text, etc... could be added somehow.

[nickchomey]

As it turns out, this exact mechanism was already proposed a while ago in the main surrealdb repo https://github.com/surrealdb/surrealdb/issues/2312

But no one there talked about wasm binary size, which is surely one of the strongest arguments for doing this, with offline-/local-first web apps (https://localfirstweb.dev/) quickly becoming the next big thing in web dev.

[limcheekin]

+1