High memory usage when invoking commands [bug]

[twharmon]

Describe the bug

High memory usage when sending lots of commands for rust from js.

Reproduction

Invoke a command in an interval:

setInterval(() => {
  invoke<number[][]>('get_data', args).then(drawOnHtmlCanvasElement)
}, 50)

Return large amount of data from rust to js:

#[tauri::command]
fn get_data(path: &str, start: u64, length: usize, filters: Vec<FilterConfig>) -> Vec<Vec<f32>> {
    vec![vec![0.0; 2560]; 21]
    // data::get(path, start, length, filters)
}

After several thousand invocations, watch memory usage creep up:

Eventually, when memory pressure is sufficient, the usage plateaus. It appears that garbage collection runs regularly, reducing the memory usage of both the processes that hog memory.

Expected behavior

I would expect the app to run without hogging memory resources.

Platform and versions

> tauri-app@0.1.0 tauri
> tauri "info"

Environment
  › OS: Mac OS 12.2.1 X64
  › Node.js: 16.13.1
  › npm: 8.1.2
  › pnpm: Not installed!
  › yarn: Not installed!
  › rustup: 1.24.3
  › rustc: 1.57.0
  › cargo: 1.57.0
  › Rust toolchain: stable-aarch64-apple-darwin 

Packages
  › @tauri-apps/cli [NPM]: 1.0.0-rc.9
  › @tauri-apps/api [NPM]: 1.0.0-rc.4
  › tauri [RUST]: 1.0.0-rc.8,
  › tauri-build [RUST]: 1.0.0-rc.7,
  › tao [RUST]: 0.8.3,
  › wry [RUST]: 0.15.1,

App
  › build-type: bundle
  › CSP: unset
  › distDir: ../build
  › devPath: http://localhost:3000/
  › framework: React

App directory structure
  ├─ node_modules
  ├─ public
  ├─ src-tauri
  ├─ build
  ├─ .git
  └─ src

Stack trace

N/A

Additional context

The get_data command only needs 5ms on my machine. That means the interval is not running too fast, since it has > 40ms of rest time between invocations.

[lucasfernog]

I feel this is the same problem as #3921, @wusyong can confirm.

[FabianLars]

Oh didn't see you closed it. When we talked about it on discord i was able to reproduce this on windows too, although it felt less severe. I'm gonna test it again later today with this specific repro.

Edit: Thought about this issue again because of #4038 btw

[lucasfernog]

From what @wusyong said all platform webviews have some kind of memory leak issue (webview2 has it even in the custom protocol stuff).

[FabianLars]

Does it leak "in" the rust process or one of the webview ones, because on windows it was the rust process that had increasing memory usage (well, the webview did too but idc about that).

Buuut it wasn't that severe iirc, like i couldn't get it to 12gigs like in the screenshot. Will re-test later or tomorrow anyway just to be sure.

Oh and now that i think about it, opening a file picker dialog increases the rust's process memory usage by 100mb and for whatever reason this won't get cleaned up completely until you exit the app. (Subsequent dialogs don't increase them that much so i guess this is just how it works, memory is meant to be used after all, so i guess it's windows itself not cleaning up if it doesn't see it as necessary?)

[lucasfernog]

The Linux leak was in the webkit process itself. I didn't test the macOS one yet.

[lucasfernog]

In my test for the HTTP memory leak the Rust process didn't increase its memory usage, but the webview2 one did (a lot), in less than a minute it was using an extra 300MB.

[betamos]

Do y'all have one or more upstream issues to track? Memory leaks are nasty, and can cover up more leaks being added. Rust is not immune to it either, it's fairly easy to accidentally create ref cycles. I've had some experience tracking those down so let me know if you're interested in techniques.

Also, workarounds can help. For instance in https://github.com/MicrosoftEdge/WebView2Feedback/issues/1633 it appears to only accumulate when the webview is attached, so if closing + creating new window is better than hide + show, it's worth knowing as a Tauri user.

[qu1ck]

@lucasfernog are you sure this is not like #852 ? When looking at dangling references to leaked http response objects it looked like they were from some closures originating in cmd invoke wrappers but I haven't looked at not minimized code.

[lucasfernog]

Do y'all have one or more upstream issues to track? Memory leaks are nasty, and can cover up more leaks being added. Rust is not immune to it either, it's fairly easy to accidentally create ref cycles. I've had some experience tracking those down so let me know if you're interested in techniques.

Also, workarounds can help. For instance in MicrosoftEdge/WebView2Feedback#1633 it appears to only accumulate when the webview is attached, so if closing + creating new window is better than hide + show, it's worth knowing as a Tauri user.

The one i'm currently tracking is https://github.com/MicrosoftEdge/WebView2Feedback/issues/2171 not sure about webkit issues yet.

[lucasfernog]

@lucasfernog are you sure this is not like #852 ? When looking at dangling references to leaked http response objects it looked like they were from some closures originating in cmd invoke wrappers but I haven't looked at not minimized code.

I could reproduce the issue with only the eval function:

use tauri::Manager;

fn main() {
  tauri::Builder::default()
    .setup(|app| {
      let w = app.get_window("main").unwrap();
      std::thread::spawn(move || loop {
        std::thread::sleep_ms(500);
        let v = vec![0xff; 1024 * 1024 * 2];
        w.eval(&format!("{:?}", v)).unwrap();
      });
      Ok(())
    })
    .run(tauri::generate_context!())
    .expect("error while running tauri application");
}

seems like the webview doesn't handle it very well.

[lzdyes]

It also memory leak on Windows.

[qu1ck]

@lucasfernog I tried your suggestion in https://github.com/tauri-apps/tauri/issues/3921#issuecomment-1115089647 to use http server as bypass invoke system. It does NOT help, memory for http response objects is still leaking. If I understand correctly this completely bypasses the rust->webview messaging and should not have the same root cause as the webview bug linked above.

Am I missing something or are there 2 separate issues here?

Minimal repro attached: httpleakrepro.zip

[ahkohd]

Also, a memory leak happens when you emit an event to a window in a loop.

During my investigation, I found out that if you emit an event to the window but on the window, there is no listener registered there will be no memory leak. But if a listener is registered, a memory leak occurs.

[qu1ck]

Ok, I'm fairly certain now that I have proof of memory leak in tauri code, not webview.

I modified the repro I linked in comment above. Now it has 2 http fetch variants: they both go through localhost server but one is via tauri invoke system and another is directly proxying requests. First one leaks memory while the second one doesn't. httpleakrepro.zip

[lucasfernog]

The leak happening in the tauri code doesn't mean it is not a webview leak, the invoke system use webview APIs (we already traced the issue to the eval function, which the invoke system uses).

[qu1ck]

Does tauri-invoke-http-like system also use eval somewhere? Because if not then we are dealing with 2 memory leaks.

[ahkohd]

Found something: https://bugs.webkit.org/show_bug.cgi?id=215729 https://github.com/Danesz/Sidewalk

[ahkohd]

Since we know we have a memory leak when calling eval, I will suggest that we should update the invoke system to use WebSockets to create some kind of RPC system.

Thinking out loud.

[lucasfernog]

tauri-invoke-http doesn't use eval - it uses a localhost server.

[ahkohd]

tauri-invoke-http doesn't use eval - it uses a localhost server.

Can we upgrade this to use WebSocket instead of HTTP? Maybe I fork the repo and see what I can do.

[lucasfernog]

Hey just copy it and create tauri-invoke-ws :joy: should be easy

[qu1ck]

tauri-invoke-http doesn't use eval - it uses a localhost server.

And it leaks memory. So, like I said, 2 independent memory leaks. See my last attachment in https://github.com/tauri-apps/tauri/issues/4026#issuecomment-1130633129

[lucasfernog]

If you're using Windows it's the https://github.com/MicrosoftEdge/WebView2Feedback/issues/2171 issue. webkit might have a similar problem too, i didn't check it yet.

[ahkohd]

@qu1ck use https://github.com/ahkohd/tauri-awesome-rpc

Works like charm.

Also avoid using tauri APIs that uses eval under the hood such as window.emit(). Use AwesomeEmit and AwesomeEvent instead. See the repo for usage docs.

[qu1ck]

@ahkohd your invoke implementation does not help.

Lucas is claiming that it's yet another issue not related to eval leak. It's not obvious to me how c++/rust land streams have effect on javascript objects that were created entirely in javascript but here we are.

[ahkohd]

I'm on OSX. Using the WS RPC, I have seen a significant memory usage reduction. It seems there is another cause of your memory leak.

[ahkohd]

@qu1ck Yeah it seems some memory leaks still exist, but I can't really pinpoint what is causing them.

[lucasfernog]

4274 doesn't fix this issue, but it helps a little bit at least with a problem on our side.

[iilyak]

I I think I am experiencing the same bug as well on windows. In my case the rust part sends events at a very high rate 20K events/second. Then these events are rendered using chartjs. The memory consumption grows at about 5Mb per second and the application cannot stay up for more then few minutes.

[f91kdash]

4274 doesn't fix this issue, but it helps a little bit at least with a problem on our side.

I think, there is another problem in wry (wkwebview mod). Rust calls OC using objc crate, an NSString (which is the invoke response data that will be posted to webview via eval method) allocs in rust and never be released in the correct way so that memory usage keeps increasing in the Tauri process.

[MaximilianLiebsch]

Good day @lucasfernog, the tauri project is very promising, gives a great developer experience and great performance. Nevertheless in our project, we are handling files/buffer >1GB, which are written in 5 seconds in Electron, in Tauri we run out of memory. Unfortunately that this a blocker of the kind, we will switch the platform back to Electron.

Since we see that the discussions in several threads are not really promising, varying from blaming webview2 to poking in the dark. So we are concerned that the bug/bugs/architectural mistakes will persist over an unknown time horizan. If there is a timeline for fixing, we would consider continuing using tauri.

Best Max.

[lzdyes]

https://github.com/MicrosoftEdge/WebView2Feedback/issues/2171#issuecomment-1150483952 I tested this version: 104.0.1293.47. From the website: https://www.catalog.update.microsoft.com/Search.aspx?q=webview2 Memory is still growing in Windows.

[MaximilianLiebsch]

Hello @lzdyes, I doubt that we are looking here at a webview bug. Since it does not matter what you do may it be writing, reading or uploading large files (100+mb), the memory always explodes. Everytime invoke is involved, boom.

[FabianLars]

The issue 2171 was only about the asset: protocol (or custom protocols in general), which is unrelated to the invoke/eval system :/

[kvnxiao]

Hello! I understand that the Tauri team is saying this is a leak related to the webview2 process (the issue different from the asset: protocol), but does there exist a github issue / ticket open in webview2feedback repo regarding this, focused on the invoke/eval system?

I see that you guys keep mentioning that webview2#2171 is unrelated, but then where exactly is the relevant issue reported then?

[MaximilianLiebsch]

Indeed, i mean everything is possible but if there is a bug in webview2 where is the ticket and where is de precise explanation why this bug is causing the memory in a tauri app to explode every time you try to send something big?

I know stupid annoying user but if anybody would have mercy with me and gives an explanation, i can create a test application to show it also

[FabianLars]

@MaximilianLiebsch

i can create a test application to show it also

That would be nice because i'm currently looking into it to double check if it's really not us leaking memory, and well, i can't get it to leak memory at all anymore. The only leak-like behavior shows up when you have the devtools open until you close them again. Well, that's not even "leak-like", but i don't find a better word to describe it.

Edit: Talking about Windows only right now.

Edit2: I do see an increase in memory while the invoke/eval/events stuff running, but it doesn't endlessly increase, just the normal GC increase/delay.

[qu1ck]

@FabianLars I added a repro in https://github.com/tauri-apps/tauri/issues/4026#issuecomment-1130633129 Just checked it with current tauri and webview, tauri invoke system still leaks.

[kvnxiao]

@MaximilianLiebsch What is the amount of data in bytes you are trying to send via invoke, per second? Maybe you are dealing with back pressure from sending data too quickly that the GC cannot keep up with clearing things from memory? And Windows is known to keep some memory allocated and reserved for applications, in case it needs to be used again by the application.

I cannot seem to reproduce this issue that is being mentioned here when trying to send ~500 KB of string data from Rust, every second. What I've tested: Have a button with setInterval @ periodic 1000ms that updates the button's text by using the string returned by invoke fn from Rust, where the string contents is a base64 encoded ~400kb PNG image + current unix epoch timestamp.

On Windows after compiling the app for release, memory for the whole application starts out at ~70 MB, and peaks at 160 MB where the GC brings it down to 120. I've been running this for 30 minutes straight.

[kvnxiao]

Can you guys also include the windows webview version you are using? @qu1ck @MaximilianLiebsch

[qu1ck]

Webview 104.0.1293.47. My test repeatedly downloads a 1MB file using tauri's APIs but with invoke system swapped to talk to rust backend using localhost server (tauri-invoke-http). It leaks memory even though it's not calling eval. Memory is growing unbounded until the window OOMs if I leave it long enough.

[lucasfernog]

The command system uses eval internally FYI.

[qu1ck]

Your post earlier:

tauri-invoke-http doesn't use eval - it uses a localhost server.

[lucasfernog]

Ahh sorry I didn't see you're using the HTTP invoke.

[wusyong]

Is this Windows only now? We fixed a few issues on macOS IIRC. We'll probably need a wry minimum example to test this too.

[MaximilianLiebsch]

Sorry for the drlays guys. I will find time this week to make a test app for Windows.

[jayvdb]

@MaximilianLiebsch , did you manage to get a minimum example test app created? If not, should others take over? This issue has come up as the biggest potential hurdle/concern for choosing tauri for my team as Windows will be our primary/sole customer userbase.

[RepublicOfKorokke]

I have same issue on macOS. Found related memory leak issue on WebKit, but I don't know the best way to workaround this problem.

https://bugs.webkit.org/show_bug.cgi?id=31253

[qu1ck]

I worked around this bug by routing all data heavy requests through a localhost server instead of going through tauri command api.

[zhouyangtingwen]

The issue still remains unresolved. Is there any progress on this?