Deprecate JSON in IPC

[wyhaya]

Discussed in https://github.com/tauri-apps/tauri/discussions/7699

^{Originally posted by **wyhaya** August 26, 2023} Currently in Tauri's ipc it is passed in `JSON` format, which can cause some problems. ### Number overflow Since JSON doesn't support `BigInt`, JavaScript's `Number` doesn't represent numbers in Rust well, e.g. `u64`,`u128` etc. ```Rust // This will get an unexpected result in JavaScript #[command] fn number() -> u64 { u64::MAX } ``` u64::MAX = `18446744073709551615`, But in JavaScript we get `18446744073709552000`, To pass it safely, we have to convert it to String, which is not a good solution. ## Binary data JSON doesn't support passing Bytes, currently Tauri converts it to a `JSON Array`, which I think adds some overhead. For example if I want to pass a Vec as binary data, I convert it to `String` via `base64` and then decode it to `Uint8Array` in JavaScript, and even then it's faster than Array. ```Rust use base64::{engine::general_purpose, Engine as _}; #[command] async fn test() -> Result { let bytes = vec![...] ; Ok(general_purpose::STANDARD_NO_PAD.encode(bytes)) } ``` ## Tauri 2.0.0-alpha.11 I've noticed that the new version of tauri supports passing binary data directly to the corresponding `ArrayBuffer`, which is great, now we have unlimited possibilities. https://github.com/tauri-apps/tauri/blob/af3268a4be69e2dee72ba867ee54bc36d12712e6/examples/commands/main.rs#L220-L224 But does this mean we have to manually convert `struct` to `Vec` and parse `ArrayBuffer` in Javascript? If we only have a single value, this is perfectly fine, if we have multiple values, we have to manually serialise and deserialise? Consider the following code: ```Rust struct Data { xl: Vec, lg: Vec } // How do I respond to JavaScript with a `Response` and parse it in JavaScript? ``` ## My thoughts JSON is great in most cases, but in some cases it doesn't meet our needs, and passing `Response` directly is very efficient, but not convenient for developers. We can write a new `RJON`(Rust JavaScript Object Notation) to replace JSON. 1. It is based on binary instead of text and will be faster in serialisation and deserialisation. 2. It supports more types in JavaScript and is safer without worrying about overflow. 3. It supports Bytes directly, without converting them to Array. 4. More. e.g `Datetime` `UUID` A simple example: ```Rust pub enum Value { Null, Bool(bool), I8(i8), U8(u8), I16(i16), U16(u16), I32(i32), U32(u32), F32(f32), I64(i64), U64(u64), F64(f64), String(String), Bytes(Bytes), Array(Vec), Map(HashMap), } ``` ```JavaScript export type Value = | null | boolean | Number | BigInt | string | Uint8Array | Value[] | { [key: string]: Value } ``` We can now pass these values better ```Rust struct Data { a: i32, // -> Number b: u64, // -> BigInt c: f32, // -> Number d: bool, // -> Boolean f: Vec, // -> Array g: Bytes, // -> Uint8Array } ``` ## Bad We have to implement `RJON` serialisation and deserialisation in both Rust and JavaScript, which adds some work, and adds some size (which should be small) since JSON is built-in in JavaScript and `RJON` is not. ## Other It would also be nice to allow `custom` IPC to bypass JSON. ```Rust // Use Tauri default IPC builder.setup_ipc(|val| { None }); // or // Use custom IPC format builder.setup_ipc(|val| { let res = Response::new(to_bytes(&val)); Some(res) }); ```

[Brendonovich]

I know @JonasKruckenberg and @oscartbeaumont have both had thoughts about this, with Jonas' approach using tauri-bindgen and Oscar's using Specta. Will leave it to them to go into detail about their approaches if they want.

[oscartbeaumont]

Yeah, I really want raw byte-level IPC for some long-term plans on rspc. I am hoping to eventually be able to do file upload/download and have been eyeing potential formats so BigInt-style numbers and Date's are supported.

I am curious if a custom IPC hook @wyhaya proposes is actually required. Hopefully, Tauri will support async custom URI handlers in the near future, and in my limited knowledge of the IPC internals, they sound very similar but I could very much be wrong.

@Brendonovich and I have been discussing the idea of doing Specta serialisation/deserialisation to be able to deal with these similar issues for rspc (and potentially tauri-specta) and I would be curious if there is some room here for collaborating or creating a shared standard if there is not one out there that would work for our usecase.

[lucasfernog]

@wyhaya we do have a way to implement a custom IPC: https://docs.rs/tauri/1.4.1/tauri/struct.Builder.html#method.invoke_system

and yes with the Response struct you need to convert your data to Vec somehow. serde_json::to_vec is the existing solution, though custom ones could be implemented. This is a limitation we have until RSON or something like that is implemented, for instance I had to split the command to read response body in order to take advantage of Response without serialization: https://github.com/tauri-apps/plugins-workspace/blob/903361100cd2dfe1cb5f4027c9a8b53f9d09a612/plugins/http/src/commands.rs#L167

[JonasKruckenberg]

It's also worth noting that tauri-bindgen fully sidesteps tauris ipc in order to send binary data (using a custom encoding that can optimize more that the regular tauri ipc bc we have more type info available)

Other tools like specta could theoretically use the same technique.

So in essence: The custom ipc method already exists it's called custom_uri_scheme_protocol 😉

[oscartbeaumont]

custom_uri_scheme_protocol hasn't really been able to satisfy the requirements for rspc and Spacedrive in its current form due to it being synchronous but it's awesome to see the Wry PR merge for async handler. Can't wait to have them in Tauri!

We recently had to move Spacedrive's media serving off of custom_uri_scheme_protocol to a localhost Axum server because it was causing freezing within our inspector view as we execute asynchronous database queries and IO operations (potentially to really slow devices like NAS's) within it that ended up requiring a block_on. I suspect it is blocking the main thread causing the UI to lock up and the app to sometimes crash which seems super logical but nonetheless was an issue.

Do you know if anyone has done benchmarking between tauri::ipc::Response::new and the custom URI handlers? It seems very inefficient for tauri::ipc::Response to serialize it to a serde_json::Value which it looks like is what it does, but then again in practice I'm sure it's probably not that bad. I could conduct some benchmarks here if none exist because I would be really curious of the results.

Relying on custom URI protocols and leaving this to the ecosystem does seem like a pretty solid solution, especially given tauri-bindgen is an officially maintained solution.