Memory benchmark might be incorrect: Tauri might consume more RAM than Electron

[jviotti]

Hey there! I'm researching the use of native web views in comparison Chromium (with Electron) and faced some unexpected results in terms of memory usage. The current memory benchmark (https://tauri.app/v1/references/benchmarks/) finds Electron to have a ~500 MB memory usage on Linux, being 2x from the one from Tauri.

The problem might be related to failing to account for shared memory, given that Chromium uses the same executable for multiple of its processes, with a significant part of them being read-only and thus actually shared. If we take these into account, the results are very different:

postman.com	macOS 12.6.1	Ubuntu 22.04.1	Windows 10
Tauri	421 MB	581 MB	399 MB
Safari	471 MB	:x:	:x:
Electron	337 MB	240 MB	318 MB
Chrome	381 MB	370 MB

vscode.dev	macOS 12.6.1	Ubuntu 22.04.1	Windows 10
Tauri	429 MB	572 MB	370 MB
Electron	332 MB	222 MB	312 MB

It appears the WebKit webview uses more memory for Web apps. The difference in memory usage was >90 MB and was present across multiple platforms, architectures, and multiple different uses of the webview (Tauri and Safari). These results are potentially surprising due to the popular notion that Electron is memory-hungry. Tauri using Edge WebView 2 had similar memory usage with Electron, which makes sense given both are based on Chromium.

While the base memory of i.e. Tauri and Sockets seem lower on macOS and Linux, they seem to increase very rapidly given actual real-world usage compared to i.e. Electron.

The tool that Tauri is using to do their memory benchmark, mprof, actually supports multiple backends for running. The default is just psutil, but they also have psutil_pss and psutil_uss.

We tested the default apps for Electron and Tauri, measuring USS and PSS, on Ubuntu and got these results

	USS	PSS
Electron	118 MB	207 MB
Tauri	125 MB	185 MB

[jviotti]

If this is correct, it might mean that both Tauri’s memory benchmark and Sockets low-memory usage mention in the front page is inaccurate once you account for shared memory, and that even Tauri’s default app consumes more RAM than Electron’s default app (which sounds counter-intuitive).

A lot of people are evaluating native web views under these benefits (including me) and this might be a surprising outcome to many. Not sure how native web views compare to Chromium on other axis, though. They might still have other significant benefits.

cc @heapwolf as results seem similar in Sockets too.

Am I missing something?

[nothingismagick]

@wravery and co had some great insights about this (with specific regard to WebView2) on twitter a while back. Maybe we could collate those here in a lead-up to a proper bit of documentation and some deeper dives.

[JonasKruckenberg]

Maybe we can use this as groundwork for another evaluation of our benchmarks, @jviotti can you share more about your methodology so we are able to reproduce your data? Maybe share the exact apps (did you just point the apps to the websites or did you include local assets?), the kind of usage (did you just open the app and let it run for a few seconds? Or did you perform some actions?) and the specific tools you used to capture those numbers?

Edit: re the tools I guess you answered that already, but knowing the exact steps would be helpful nonetheless. Especially on windows where precise RAM usage is very difficult define with all the shared processes and stuff

[Beanow]

Thank you for sharing your findings, and having this discussion.

I would very much welcome both documentation and technical / research contributions when it comes to benchmarks.

So first the disclaimer: the benchmarks on https://tauri.app/v1/references/benchmarks/ need to be taken with a good handful of salt. It's by no means a scientific report and honestly only should be considered a smoke test / regression test. Where the main usefulness in running these benchmarks is to compare one Tauri version against itself, given similar testing conditions.

The other is, memory profiling is a more complex topic than numbers in a table usually conveys. Or data on a graph, the issue happens when boiling it down to a single number. Because as you've shown, different measuring setups, tools and interpretations can give different results.

For example, caching strategies of the OS, your dependencies, and apps themselves tend to behave very different depending on the current memory pressure and capacity. For example running the exact same benchmarks on a 2GB Raspberry Pi is likely to give you wildly different results than a 64GB DDR5 workstation currently using 7GB including caches. Similarly, testing when your root partition is Ext4 vs ZFS I also expect will give you wildly different system-wide memory snapshots :laughing:

There's also a question of, under what conditions will a system webview optimize resources when multiple bespoke apps are using it. Like in how I tend to use my workstation, 3-4 messaging apps, IDE open, browser open, etc. you get the idea. And do you care about it, or is your use case a single app low-resource kiosk?

All of that is not to make excuses. Is it thinkable that for a given scenario, especially when it's including the webview's resources, a Tauri app uses more RAM than Electron? I wouldn't be surprised. But is it shared? Are they caches? Will it be dropped under memory pressure? We can look at more I think.

[jviotti]

@nothingismagick @wravery I would love to read more about those insights. I'm specifically interested in which scenarios native web views beat Chromium (in i.e. Electron) and vice-versa.

[jviotti]

@JonasKruckenberg

@jviotti can you share more about your methodology so we are able to reproduce your data? Maybe share the exact apps (did you just point the apps to the websites or did you include local assets?), the kind of usage (did you just open the app and let it run for a few seconds? Or did you perform some actions?) and the specific tools you used to capture those numbers?

• We closed unnecessary apps running on the system. Before running tests observe memory usage on the system and confirm it's mostly stable
• On Ubuntu, we measured using "System Monitor", free and smem
• On macOS, we used the "Activity Monitor" using the "Memory" column (not "Real Memory")
• On Windows, we used the "Resource Monitor" (linked from "Task Manager")
• All tests window sizes have been kept at a standardardized 800x600
• Do a few dry runs before measuring
• On each framework, load the corresponding websites using window.location.replace('https://....') on DevTools
• After starting up the app being tested, it was left to sit for 30-90 seconds to "rest" before taking measurements.
• We did at least two measurements for each scenario under test, to account for noise and ensure repeatability and took the mean of these measurements
• We tested https://www.postman.com/, https://vscode.dev/ and a blank HTML page

[jviotti]

It'd be good to try to test the exact same local-first app on each framework. We went for web apps because there is no app that is available in exactly the same way in all of them.

[jviotti]

There's also a question of, under what conditions will a system webview optimize resources when multiple bespoke apps are using it. Like in how I tend to use my workstation, 3-4 messaging apps, IDE open, browser open, etc. you get the idea. And do you care about it, or is your use case a single app low-resource kiosk?

That is a good one. We definitely care about these cases too. As frameworks that make use of native web views become more popular, this becomes more of a point to keep in mind. We should try to run the experiments with multiple app instances at the same time.

Is it thinkable that for a given scenario, especially when it's including the webview's resources, a Tauri app uses more RAM than Electron? I wouldn't be surprised. But is it shared? Are they caches? Will it be dropped under memory pressure? We can look at more I think.

Yeah, I think is the key think we (at Postman) are investigating. Chromium seems to be relying much more on shared memory (therefore "winning" on certain cases when shared memory is taken into account), but we are also interested in how different web views compare in terms of runtime performance, battery consumption, etc.

Any insights on these topics would be very beneficial for us and the general community.

[JonasKruckenberg]

Okay so given the specific workload you tested with the results make sense.

Since you're not using any native functionality (which uses nodejs with electron and rust with Tauri) you're basically benchmarking Webkit against Chromium.

So for this benchmark it means since Tauri uses webkit on macOS you would expect it's memory consumption to be equal to Safari, same for chrome on windows. That there is a memory consumption difference between webkit and chromium doesn't surprise much.

I would be interested to see us investigate the following things:

1. Benchmarks using native code. This is where I would predict the biggest difference, since Tauri uses Rust for this. But it would be interesting to see exact numbers.
2. Testing the amount of sharing between apps. Basically see how the memory consumption grows as you open more instances of the same app. Does it grow linearly or not? What's the growth factor?
3. Benchmark against more frameworks. While this issue got started with a comparison to electron it would be interesting to know how we compare to other frameworks like Wails, Flutter, etc.

[jviotti]

you're basically benchmarking Webkit against Chromium.

Exactly, this is the point. We are trying to understand the underlying differences between WebKit and Chromium as the foundation for building apps on top. Complex applications make heavy use of web technologies, React, etc and then branch out to native functionality as required, so the web side of things tends to dominate.

Testing the amount of sharing between apps. Basically see how the memory consumption grows as you open more instances of the same app. Does it grow linearly or not? What's the growth factor?

We are running some more in-depth tests on this, but initial findings show that:

• Any application making use of native web views of course reaps the benefits
• Sharing happens on Electron when multiple applications use the exact same Electron version (which comes back to the initial shared memory observation)

[jviotti]

Apart from memory usage, we are interested in benchmarking native web views vs Chromium/Electron in terms of batter usage, CPU consumption, etc. Any findings you have would be hugely appreciated :)

[VictorBulba]

Hey! I am struggling to find the correct issue, but I just wanted to let you know about the potential feature request. I've been trying out Tauri and really appreciate its integration with Rust, but safari really sucks, and that kills all the crossplatformness when you still need to consider how webkit behaves. It would be great if Tauri could provide a feature that uses chromium on all platforms (bundling it into the app for mac/lin). Essentially, I love Tauri for its easy to setup framework that works with Rust, but I'm still considering using Electron for better compatibility due to these concerns 😥

[FabianLars]

@VictorBulba Yeah, we're aware of that feature request, you can search for qtwebengine for example to find some issues. Right now we still don't have any plans (or time) for different webviews on linux/macos. Btw, i'm going to minimize our comments because they don't really belong here :)

[AlbertInRC]

Look forward to the update of Jviotti's last question: "Apart from memory usage, we are interested in benchmarking native web views vs Chromium/Electron in terms of batter usage, CPU consumption, etc. Any findings you have would be hugely appreciated :)" And I agree with Jviotti, complex applications make heavy use of web technologies so the performance/memory is dominated by the web on WebKit or Chromium. If the native functionality is very lightweight, I don't think it makes a big difference between Tauri and Electron.

May I know if we compare the performance/RAM usage between Tauri + Rust and Electron + native modules(C/C++) ?

[jviotti]

I'm no longer involved with Electron and this was more of a discussion with no specific action items, so closing this issue!