search

wez / wezterm

A GPU-accelerated cross-platform terminal emulator and multiplexer written by @wez and implemented in Rust
https://wezfurlong.org/wezterm/
Other
18.15k stars 810 forks source link

Excessive memory usage #2626

Open dotdash opened 2 years ago

dotdash commented 2 years ago

What Operating System(s) are you seeing this problem on?

Linux X11

Which Wayland compositor or X11 Window manager(s) are you using?

awesome wm

WezTerm version

wezterm 20221012-092613-6415ba0e

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

After a few hours, the memory usage of wezterm grows to excessive amounts and rarely seems to go down. My current wezterm instance is running for ~ 18hours and has grown its RSS to 1.4G with just a single open window (there have been up to ~10 at some point in time). Max terminal sizes for the windows were 426x111. 

doener@atjola:~ $ ps axuww | grep wezterm
doener     49667  0.5  4.5 5562436 1484760 ?     Ssl  Oct13   6:19 /usr/bin/wezterm-gui

To Reproduce

Work with wezterm for some hours, mostly using vim.

Configuration

Full config is as follows. I don't think anything is really relevant, but unless required, I'd rather not have to work with the default settings for an extended amount of time to reproduce this problem :sweat_smile: 

local wezterm = require 'wezterm'

return {
  term = 'wezterm',
  color_scheme = 'Gruvbox dark, medium (base16)',

  font = wezterm.font_with_fallback {
    'CaskaydiaCove Nerd Font Mono',
    'DejaVu Sans',
  },
  font_size = 12,

  adjust_window_size_when_changing_font_size = false,
  hide_tab_bar_if_only_one_tab = true,
  window_padding = {
    top = 0,
    bottom = 0,
    left = 0,
    right = 0,
  },
}

Expected Behavior

Memory usage should not grow as much, or at least drop down again when windows get closed.

Logs

Debug Overlay wezterm version: 20221012-092613-6415ba0e OpenGL version: NVIDIA GeForce GTX 1080 Ti/PCIe/SSE2 4.6.0 NVIDIA 510.85.02 Enter lua statements or expressions and hit Enter. Press ESC or CTRL-D to exit 23:20:37.273 WARN mux::localpane > unknown DeviceControlMode::Enter EnterDeviceControlMode(params: [], intermediates: [], byte: 'z' 0x7a, ignored_extra_intermediates=false) 23:20:37.273 WARN mux::localpane > unhandled DeviceControlMode::Data 7a z 23:20:37.273 WARN wezterm_term::terminalstate::performer > unknown unspecified CSI: "0%m" 23:45:21.544 WARN mux::localpane > unknown DeviceControlMode::Enter EnterDeviceControlMode(params: [], intermediates: [], byte: 'z' 0x7a, ignored_extra_intermediates=false) 23:45:21.544 WARN mux::localpane > unhandled DeviceControlMode::Data 7a z 23:45:21.544 WARN wezterm_term::terminalstate::performer > unknown unspecified CSI: "0%m" 23:58:06.376 WARN mux::localpane > unknown DeviceControlMode::Enter EnterDeviceControlMode(params: [], intermediates: [], byte: 'z' 0x7a, ignored_extra_intermediates=false) 23:58:06.377 WARN mux::localpane > unhandled DeviceControlMode::Data 7a z 23:58:06.377 WARN wezterm_term::terminalstate::performer > unknown unspecified CSI: "0%m" 10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3492 > 1908 10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3501 > 1908 10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3501 > 1908 10:35:41.226 INFO wezterm_gui::termwindow::render > breaking on overflow 3510 > 1908 10:35:41.249 INFO wezterm_gui::termwindow::render > breaking on overflow 3483 > 1908 10:35:41.249 INFO wezterm_gui::termwindow::render > breaking on overflow 3492 > 1908 10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1755 > 1737 10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1764 > 1737 10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1773 > 1737 10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1782 > 1737 10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1791 > 1737 10:38:22.164 INFO wezterm_gui::termwindow::render > breaking on overflow 1791 > 1737

Anything else?

No response

wez commented 2 years ago

What does nvidia-smi show as the graphic memory usage for wezterm? Do you use any image protocols in your workflow?

mfiano commented 2 years ago

I was curious to test this on my end so I wrote a quick hack of a shell one-liner to print out the RSS usage every 5 seconds for 5 minutes, across a few different scenarios.

If I:

  1. make sure all wezterm processes are not running
  2. execute wezterm with the default config (no config)
  3. execute htop in the launched pane
  4. sort by the Command column by clicking it
  5. find and click the wezterm-gui row
  6. watch the RES cell for wezterm for about 5 minutes.

For me, it rises from about 100M to 200M usually, and the increase seems non-linear. It seems to allocate about 3x as more as it deallocated during the beginning, but around 200M is where it starts to increase much more slowly.

Below is a quick hacky script to print out the timings of the RES/RSS memory every 5 seconds for 5 minutes, with the results I get under various workloads of both the default configuratopm. as well as my custom configuration:

Default/empty config's memory increase rate (Idle workload, just this script running, and htop running as a base line in 1 of 2 vertical splits) 
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 22:15:13 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   46C    P0    29W / 150W |    948MiB /  6144MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          160MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    641475      G   /usr/bin/wezterm-gui                2MiB |
+-----------------------------------------------------------------------------+
mfiano@workstation ~$ for i in `seq 60`; do { sleep 5 && echo "Time: $((i * 5))s, RES: $(ps --format rss -$(pidof wezterm-gui) | tail -1)"; } done
Time: 5s, RES: 96280
Time: 10s, RES: 105748
Time: 15s, RES: 107580
Time: 20s, RES: 110212
Time: 25s, RES: 112044
Time: 30s, RES: 114412
Time: 35s, RES: 116476
Time: 40s, RES: 123580
Time: 45s, RES: 125428
Time: 50s, RES: 127004
Time: 55s, RES: 128584
Time: 60s, RES: 130164
Time: 65s, RES: 126896
Time: 70s, RES: 134836
Time: 75s, RES: 136940
Time: 80s, RES: 138780
Time: 85s, RES: 139568
Time: 90s, RES: 140356
Time: 95s, RES: 147416
Time: 100s, RES: 143060
Time: 105s, RES: 145952
Time: 110s, RES: 146748
Time: 115s, RES: 148852
Time: 120s, RES: 151484
Time: 125s, RES: 154380
Time: 130s, RES: 156488
Time: 135s, RES: 157804
Time: 140s, RES: 159896
Time: 145s, RES: 161804
Time: 150s, RES: 164424
Time: 155s, RES: 166844
Time: 160s, RES: 169736
Time: 165s, RES: 169992
Time: 170s, RES: 169992
Time: 175s, RES: 170760
Time: 180s, RES: 171020
Time: 185s, RES: 171020
Time: 190s, RES: 171020
Time: 195s, RES: 171020
Time: 200s, RES: 171020
Time: 205s, RES: 171276
Time: 210s, RES: 171276
Time: 215s, RES: 171276
Time: 220s, RES: 171276
Time: 225s, RES: 171276
Time: 230s, RES: 171276
Time: 235s, RES: 171276
Time: 240s, RES: 171276
Time: 245s, RES: 171276
Time: 250s, RES: 171276
Time: 255s, RES: 171276
Time: 260s, RES: 174984
Time: 265s, RES: 174984
Time: 270s, RES: 174984
Time: 275s, RES: 174984
Time: 280s, RES: 174984
Time: 285s, RES: 174984
Time: 290s, RES: 174984
Time: 295s, RES: 174984
Time: 300s, RES: 174984
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 22:26:00 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   47C    P2    27W / 150W |    998MiB /  6144MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          208MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    641955      G   /usr/bin/wezterm-gui                4MiB |
+-----------------------------------------------------------------------------+
My full personal config's memory increase rate (Idle workload, just this script running, and htop running as a base line in 1 of 2 vertical splits) 
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 22:13:39 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   44C    P0    29W / 150W |    948MiB /  6144MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          160MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    640378      G   /usr/bin/wezterm-gui                2MiB |
+-----------------------------------------------------------------------------+
mfiano@workstation ~$ for i in `seq 60`; do { sleep 5 && echo "Time: $((i * 5))s, RES: $(ps --format rss -$(pidof wezterm-gui) | tail -1)"; } done
Time: 5s, RES: 117288
Time: 10s, RES: 119920
Time: 15s, RES: 120708
Time: 20s, RES: 124392
Time: 25s, RES: 125964
Time: 30s, RES: 130180
Time: 35s, RES: 131760
Time: 40s, RES: 135264
Time: 45s, RES: 135788
Time: 50s, RES: 138404
Time: 55s, RES: 141040
Time: 60s, RES: 141560
Time: 65s, RES: 146512
Time: 70s, RES: 147824
Time: 75s, RES: 151248
Time: 80s, RES: 153620
Time: 85s, RES: 156244
Time: 90s, RES: 157748
Time: 95s, RES: 159064
Time: 100s, RES: 160640
Time: 105s, RES: 162484
Time: 110s, RES: 163532
Time: 115s, RES: 165084
Time: 120s, RES: 168244
Time: 125s, RES: 175244
Time: 130s, RES: 176824
Time: 135s, RES: 178668
Time: 140s, RES: 181304
Time: 145s, RES: 182608
Time: 150s, RES: 182608
Time: 155s, RES: 183656
Time: 160s, RES: 183656
Time: 165s, RES: 183656
Time: 170s, RES: 183656
Time: 175s, RES: 183656
Time: 180s, RES: 183656
Time: 185s, RES: 183656
Time: 190s, RES: 183656
Time: 195s, RES: 183908
Time: 200s, RES: 183908
Time: 205s, RES: 183908
Time: 210s, RES: 183908
Time: 215s, RES: 183908
Time: 220s, RES: 183908
Time: 225s, RES: 183908
Time: 230s, RES: 183908
Time: 235s, RES: 183908
Time: 240s, RES: 183908
Time: 245s, RES: 183908
Time: 250s, RES: 183908
Time: 255s, RES: 183908
Time: 260s, RES: 183908
Time: 265s, RES: 183908
Time: 270s, RES: 183908
Time: 275s, RES: 183908
Time: 280s, RES: 183908
Time: 285s, RES: 183908
Time: 290s, RES: 183908
Time: 295s, RES: 183908
Time: 300s, RES: 183908
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 22:12:49 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   40C    P5    14W / 150W |    949MiB /  6144MiB |     29%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          160MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    637008      G   /usr/bin/wezterm-gui                3MiB |
+-----------------------------------------------------------------------------+
Default/empty config's memory increase rate (High frequency output: this script running, and mpv with libcaca running in 1 of 2 vertical splits) 
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:01:38 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   49C    P0    31W / 150W |   1039MiB /  6144MiB |     13%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          250MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    717419      G   /usr/bin/wezterm-gui                3MiB |
+-----------------------------------------------------------------------------+
mfiano@workstation ~$ for i in `seq 60`; do { sleep 5 && echo "Time: $((i * 5))s, RES: $(ps --format rss -$(pidof wezterm-gui) | tail -1)"; } done
Time: 5s, RES: 255904
Time: 10s, RES: 256320
Time: 15s, RES: 259496
Time: 20s, RES: 260016
Time: 25s, RES: 260016
Time: 30s, RES: 260016
Time: 35s, RES: 260016
Time: 40s, RES: 260016
Time: 45s, RES: 260016
Time: 50s, RES: 260016
Time: 55s, RES: 260016
Time: 60s, RES: 261596
Time: 65s, RES: 262388
Time: 70s, RES: 262388
Time: 75s, RES: 262388
Time: 80s, RES: 262388
Time: 85s, RES: 262388
Time: 90s, RES: 262388
Time: 95s, RES: 262388
Time: 100s, RES: 266020
Time: 105s, RES: 266544
Time: 110s, RES: 266808
Time: 115s, RES: 266808
Time: 120s, RES: 266808
Time: 125s, RES: 266808
Time: 130s, RES: 266808
Time: 135s, RES: 266808
Time: 140s, RES: 266808
Time: 145s, RES: 266808
Time: 150s, RES: 266808
Time: 155s, RES: 266808
Time: 160s, RES: 267860
Time: 165s, RES: 267860
Time: 170s, RES: 267860
Time: 175s, RES: 267860
Time: 180s, RES: 267860
Time: 185s, RES: 267860
Time: 190s, RES: 267860
Time: 195s, RES: 268112
Time: 200s, RES: 268112
Time: 205s, RES: 268112
Time: 210s, RES: 268112
Time: 215s, RES: 268112
Time: 220s, RES: 268112
Time: 225s, RES: 268748
Time: 230s, RES: 268748
Time: 235s, RES: 268748
Time: 240s, RES: 268748
Time: 245s, RES: 268748
Time: 250s, RES: 268748
Time: 255s, RES: 268748
Time: 260s, RES: 268748
Time: 265s, RES: 268748
Time: 270s, RES: 268748
Time: 275s, RES: 268748
Time: 280s, RES: 268748
Time: 285s, RES: 268748
Time: 290s, RES: 268748
Time: 295s, RES: 268748
Time: 300s, RES: 268748
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:06:47 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   51C    P0    33W / 150W |   1025MiB /  6144MiB |     13%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          234MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    717419      G   /usr/bin/wezterm-gui                5MiB |
+-----------------------------------------------------------------------------+
My full personal config's memory increase rate (High frequency output: this script running, and mpv with libcaca running in 1 of 2 vertical splits) 
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:08:39 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   50C    P0    34W / 150W |    994MiB /  6144MiB |     14%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          204MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    721776      G   /usr/bin/wezterm-gui                3MiB |
+-----------------------------------------------------------------------------+
mfiano@workstation ~$ for i in `seq 60`; do { sleep 5 && echo "Time: $((i * 5))s, RES: $(ps --format rss -$(pidof wezterm-gui) | tail -1)"; } done
Time: 5s, RES: 300188
Time: 10s, RES: 307644
Time: 15s, RES: 307644
Time: 20s, RES: 308084
Time: 25s, RES: 308312
Time: 30s, RES: 308312
Time: 35s, RES: 318764
Time: 40s, RES: 316520
Time: 45s, RES: 316520
Time: 50s, RES: 314912
Time: 55s, RES: 317360
Time: 60s, RES: 317360
Time: 65s, RES: 317360
Time: 70s, RES: 323636
Time: 75s, RES: 322692
Time: 80s, RES: 322692
Time: 85s, RES: 322692
Time: 90s, RES: 326408
Time: 95s, RES: 326408
Time: 100s, RES: 326408
Time: 105s, RES: 326408
Time: 110s, RES: 326672
Time: 115s, RES: 326672
Time: 120s, RES: 326672
Time: 125s, RES: 326672
Time: 130s, RES: 326672
Time: 135s, RES: 326672
Time: 140s, RES: 326672
Time: 145s, RES: 320936
Time: 150s, RES: 320936
Time: 155s, RES: 323308
Time: 160s, RES: 323308
Time: 165s, RES: 321320
Time: 170s, RES: 321320
Time: 175s, RES: 323428
Time: 180s, RES: 323428
Time: 185s, RES: 323428
Time: 190s, RES: 323428
Time: 195s, RES: 323428
Time: 200s, RES: 321452
Time: 205s, RES: 321452
Time: 210s, RES: 321452
Time: 215s, RES: 321452
Time: 220s, RES: 321452
Time: 225s, RES: 323560
Time: 230s, RES: 321240
Time: 235s, RES: 323668
Time: 240s, RES: 323668
Time: 245s, RES: 323668
Time: 250s, RES: 323668
Time: 255s, RES: 324028
Time: 260s, RES: 324028
Time: 265s, RES: 322516
Time: 270s, RES: 322516
Time: 275s, RES: 322516
Time: 280s, RES: 322516
Time: 285s, RES: 322516
Time: 290s, RES: 322516
Time: 295s, RES: 322516
Time: 300s, RES: 322564
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:14:09 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   50C    P0    32W / 150W |    993MiB /  6144MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          204MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    721776      G   /usr/bin/wezterm-gui                3MiB |
+-----------------------------------------------------------------------------+
Default/empty config's memory increase rate (Regular workflow with a few tabs and panels containing vim and some external processes I use while coding) 
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:38:08 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   47C    P0    26W / 150W |    913MiB /  6144MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          125MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    741051      G   /usr/bin/wezterm-gui                2MiB |
+-----------------------------------------------------------------------------+
mfiano@workstation ~$ for i in `seq 60`; do { sleep 5 && echo "Time: $((i * 5))s, RES: $(ps --format rss -$(pidof wezterm-gui) | tail -1)"; } done
Time: 5s, RES: 125300
Time: 10s, RES: 139404
Time: 15s, RES: 165268
Time: 20s, RES: 165532
Time: 25s, RES: 167092
Time: 30s, RES: 166728
Time: 35s, RES: 166728
Time: 40s, RES: 166728
Time: 45s, RES: 183824
Time: 50s, RES: 190060
Time: 55s, RES: 190060
Time: 60s, RES: 190060
Time: 65s, RES: 196288
Time: 70s, RES: 192232
Time: 75s, RES: 192232
Time: 80s, RES: 188216
Time: 85s, RES: 188216
Time: 90s, RES: 188216
Time: 95s, RES: 188216
Time: 100s, RES: 188216
Time: 105s, RES: 188216
Time: 110s, RES: 188216
Time: 115s, RES: 188740
Time: 120s, RES: 190060
Time: 125s, RES: 190060
Time: 130s, RES: 190060
Time: 135s, RES: 190060
Time: 140s, RES: 190060
Time: 145s, RES: 190060
Time: 150s, RES: 190324
Time: 155s, RES: 190324
Time: 160s, RES: 190324
Time: 165s, RES: 190324
Time: 170s, RES: 190324
Time: 175s, RES: 190520
Time: 180s, RES: 190520
Time: 185s, RES: 190520
Time: 190s, RES: 190520
Time: 195s, RES: 187868
Time: 200s, RES: 188040
Time: 205s, RES: 232700
Time: 210s, RES: 239160
Time: 215s, RES: 250916
Time: 220s, RES: 271300
Time: 225s, RES: 271416
Time: 230s, RES: 271420
Time: 235s, RES: 271420
Time: 240s, RES: 271420
Time: 245s, RES: 271420
Time: 250s, RES: 266448
Time: 255s, RES: 266540
Time: 260s, RES: 266540
Time: 265s, RES: 266540
Time: 270s, RES: 266540
Time: 275s, RES: 266540
Time: 280s, RES: 266540
Time: 285s, RES: 266540
Time: 290s, RES: 266540
Time: 295s, RES: 266540
Time: 300s, RES: 266540
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:43:22 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   48C    P0    32W / 150W |    915MiB /  6144MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     691MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          125MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    741051      G   /usr/bin/wezterm-gui                4MiB |
+-----------------------------------------------------------------------------+
My full personal config's memory increase rate (Regular workflow with a few tabs and panels containing vim and some external processes I use while coding) 
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:18:19 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   49C    P0    30W / 150W |   1142MiB /  6144MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     863MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          168MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    726429      G   /usr/bin/wezterm-gui                6MiB |
|    0   N/A  N/A    727191      G   alacritty                           8MiB |
+-----------------------------------------------------------------------------+
mfiano@workstation ~$ for i in `seq 60`; do { sleep 5 && echo "Time: $((i * 5))s, RES: $(ps --format rss -$(pidof wezterm-gui) | tail -1)"; } done
Time: 5s, RES: 296212
Time: 10s, RES: 296212
Time: 15s, RES: 309900
Time: 20s, RES: 309900
Time: 25s, RES: 310144
Time: 30s, RES: 315332
Time: 35s, RES: 335624
Time: 40s, RES: 335624
Time: 45s, RES: 335864
Time: 50s, RES: 345024
Time: 55s, RES: 345024
Time: 60s, RES: 345540
Time: 65s, RES: 346068
Time: 70s, RES: 346068
Time: 75s, RES: 346068
Time: 80s, RES: 346068
Time: 85s, RES: 346068
Time: 90s, RES: 346068
Time: 95s, RES: 346068
Time: 100s, RES: 346068
Time: 105s, RES: 346280
Time: 110s, RES: 346280
Time: 115s, RES: 346280
Time: 120s, RES: 346280
Time: 125s, RES: 346280
Time: 130s, RES: 346280
Time: 135s, RES: 346280
Time: 140s, RES: 346280
Time: 145s, RES: 346280
Time: 150s, RES: 346280
Time: 155s, RES: 346280
Time: 160s, RES: 346280
Time: 165s, RES: 346280
Time: 170s, RES: 346280
Time: 175s, RES: 346280
Time: 180s, RES: 346280
Time: 185s, RES: 346280
Time: 190s, RES: 346280
Time: 195s, RES: 346280
Time: 200s, RES: 346280
Time: 205s, RES: 346280
Time: 210s, RES: 346280
Time: 215s, RES: 346280
Time: 220s, RES: 346280
Time: 225s, RES: 346280
Time: 230s, RES: 346280
Time: 235s, RES: 344444
Time: 240s, RES: 344444
Time: 245s, RES: 344444
Time: 250s, RES: 344444
Time: 255s, RES: 344444
Time: 260s, RES: 344444
Time: 265s, RES: 344444
Time: 270s, RES: 344444
Time: 275s, RES: 344444
Time: 280s, RES: 344444
Time: 285s, RES: 344444
Time: 290s, RES: 344444
Time: 295s, RES: 344444
Time: 300s, RES: 344444
mfiano@workstation ~$ nvidia-smi
Fri Oct 14 23:23:28 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.56.06    Driver Version: 520.56.06    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
|  0%   48C    P2    25W / 150W |   1097MiB /  6144MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A    474461      G   /usr/lib/Xorg                     863MiB |
|    0   N/A  N/A    474489      G   alacritty                           8MiB |
|    0   N/A  N/A    474549      G   alacritty                           8MiB |
|    0   N/A  N/A    474693      G   /usr/lib/firefox/firefox          125MiB |
|    0   N/A  N/A    538776      G   alacritty                           8MiB |
|    0   N/A  N/A    726429      G   /usr/bin/wezterm-gui                5MiB |
|    0   N/A  N/A    727191      G   alacritty                           8MiB |
+-----------------------------------------------------------------------------+

Observations

Interesting 🤔

dotdash commented 2 years ago

What does nvidia-smi show as the graphic memory usage for wezterm?

I'm currently working on a different PC, which has a Ryzen 5700U with integrated graphics, same issue. Currently one terminal open, started 5 hours ago, memory usage is at ~700MB. Is there a similar tool for Radeon graphics that I should run on that box?

Do you use any image protocols in your workflow?

No

wfjt commented 2 years ago

Same here, Ubuntu 22.04 LTS Gnome+Wayland with integrated Intel GPU on a Thinkpad. Memory consumption was about the same as my two day old firefox with 40 tabs. I was running 6 tabs and 12 panes maximized, a single os//mux window. The terminal overall got sluggish, key presses lagging, repeating keys coming in bursts, search mode was particularly bad. In the end Gnome crashed so could've been a Gnome issue in part. I also get tons of wezterm errors from gsd-media-keys

I'm on a recent nightly: don't do anything with images, just symbol/emoji font in prompt/nvim.

Versions and logs if of any help. #### Logs I have thousands of these coming from gsd-edia-keys: over a few days: ``` Oct 16 05:50:41 x1g9 gsd-media-keys[379493]: 05:50:41.089 ERROR window::os::wayland::pointer > Unable to set cursor to ew-resize: cursor not found Oct 16 05:50:41 x1g9 gsd-media-keys[379493]: 05:50:41.090 ERROR window::os::wayland::pointer > Unable to set cursor to ew-resize: cursor not found Oct 16 05:50:41 x1g9 gsd-media-keys[379493]: 05:50:41.108 ERROR window::os::wayland::pointer > Unable to set cursor to ew-resize: cursor not found Oct 16 05:50:41 x1g9 gsd-media-keys[379493]: 05:50:41.122 ERROR window::os::wayland::pointer > Unable to set cursor to ew-resize: cursor not found Oct 16 05:50:41 x1g9 gsd-media-keys[379493]: 05:50:41.136 ERROR window::os::wayland::pointer > Unable to set cursor to ew-resize: cursor not found Oct 18 17:28:48 x1g9 gsd-media-keys[379493]: 17:28:48.876 INFO wezterm_gui::termwindow::render > breaking on overflow 550 > 390 Oct 18 17:28:48 x1g9 gsd-media-keys[379493]: 17:28:48.876 INFO wezterm_gui::termwindow::render > breaking on overflow 550 > 390 Oct 18 17:28:48 x1g9 gsd-media-keys[379493]: 17:28:48.876 INFO wezterm_gui::termwindow::render > breaking on overflow 560 > 390 Oct 18 17:28:48 x1g9 gsd-media-keys[379493]: 17:28:48.876 INFO wezterm_gui::termwindow::render > breaking on overflow 560 > 390 Oct 18 17:28:48 x1g9 gsd-media-keys[379493]: 17:28:48.876 INFO wezterm_gui::termwindow::render > breaking on overflow 580 > 390 Oct 18 17:28:48 x1g9 gsd-media-keys[379493]: 17:28:48.876 INFO wezterm_gui::termwindow::render > breaking on overflow 580 > 390 Oct 18 17:28:48 x1g9 gsd-media-keys[379493]: 17:28:48.877 INFO wezterm_gui::termwindow::render > breaking on overflow 590 > 390 Oct 18 19:08:12 x1g9 gsd-media-keys[29999]: 19:08:12.196 WARN wezterm_term::terminalstate::performer > unhandled ControlCode PU2 Oct 18 19:08:12 x1g9 gsd-media-keys[29999]: 19:08:12.196 WARN wezterm_term::terminalstate::performer > unhandled ControlCode DataLinkEscape Oct 18 19:08:12 x1g9 gsd-media-keys[29999]: 19:08:12.196 WARN wezterm_term::terminalstate::performer > unhandled ControlCode StartOfHeading ``` Past 14 hours: ``` ❯ journalctl --user -x -n 100000 -all -t gsd-media-keys --no-pager | grep -iE '(:[[:alnum:]]+::)+|wezterm' | grep -v 'lua: '| wc -l 458 ``` A couple days: ``` ❯ head -n1 /var/log/syslog|awk '{print $1" "$2" "$3}'; grep -iE '(:[[:alnum:]]+::)+|wezterm' /var/log/syslog | grep -v 'lua: '| wc -l; tail -n1 /var/log/syslog | awk '{print $1" "$2" "$3}' Oct 16 00:00:00 4354 Oct 18 20:53:56 ``` #### Some system details ``` Debug Overlay wezterm version: 20221015-164502-7b904f05 OpenGL version: Mesa Intel(R) Xe Graphics (TGL GT2) 4.6 (Compatibility Profile) Mesa 22.0.5 ❯ uname -r 5.15.0-50-generic ❯ gnome-shell --version GNOME Shell 42.4 ❯ dpkg -l | grep libwayland | sed ' s/ \+/ /g' ii libwayland-bin 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - binary utilities ii libwayland-client0:amd64 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - client library ii libwayland-cursor++0:amd64 0.2.8-2 amd64 wayland compositor infrastructure - cursor library C++ bindings ii libwayland-cursor0:amd64 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - cursor library ii libwayland-dev:amd64 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - development files ii libwayland-egl1:amd64 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - EGL library ii libwayland-server0:amd64 1.20.0-1ubuntu0.1 amd64 wayland compositor infrastructure - server library ❯ dpkg -l | grep xserver-xorg-video-intel | sed ' s/ \+/ /g' ixserver-xorg-video-intel 2:2.99.917+git20210115-1 amd64 X.Org X server -- Intel i8xx, i9xx display driver ❯ glxinfo -B name of display: :0 display: :0 screen: 0 direct rendering: Yes Extended renderer info (GLX_MESA_query_renderer): Vendor: Intel (0x8086) Device: Mesa Intel(R) Xe Graphics (TGL GT2) (0x9a49) Version: 22.0.5 Accelerated: yes Video memory: 10009MB Unified memory: yes Preferred profile: core (0x1) Max core profile version: 4.6 Max compat profile version: 4.6 Max GLES1 profile version: 1.1 Max GLES[23] profile version: 3.2 OpenGL vendor string: Intel OpenGL renderer string: Mesa Intel(R) Xe Graphics (TGL GT2) OpenGL core profile version string: 4.6 (Core Profile) Mesa 22.0.5 OpenGL core profile shading language version string: 4.60 OpenGL core profile context flags: (none) OpenGL core profile profile mask: core profile OpenGL version string: 4.6 (Compatibility Profile) Mesa 22.0.5 OpenGL shading language version string: 4.60 OpenGL context flags: (none) OpenGL profile mask: compatibility profile OpenGL ES profile version string: OpenGL ES 3.2 Mesa 22.0.5 OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.20 ❯ lspci -k | grep -EA3 'VGA|3D|Display' 00:02.0 VGA compatible controller: Intel Corporation TigerLake-LP GT2 [Iris Xe Graphics] (rev 01) Subsystem: Lenovo TigerLake-LP GT2 [Iris Xe Graphics] Kernel driver in use: i915 Kernel modules: i915 ```

Update: After upgrading wezterm to 20221017-204532-ec4d5eb1 RSS is staying below 600 MiB after two days but that's still quite a lot for a term. I'm still getting the same tons of errors in logs.

dotdash commented 2 years ago

Back on the nvidia box, I had 2 wezterm windows opened, which consume ~900MB of memory, and the whole system was lagging a lot. Xorg CPU usage was upwards of 60% whenever I tried to use firefox (which felt like a slideshow at best).

nvidia-smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.85.02    Driver Version: 510.85.02    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:0A:00.0  On |                  N/A |
| 17%   55C    P0    70W / 275W |    525MiB / 11264MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1752      G   /usr/lib/xorg/Xorg                512MiB |
|    0   N/A  N/A      2872      G   /usr/bin/wezterm-gui               10MiB |
+-----------------------------------------------------------------------------+
wez commented 2 years ago

I've made some adjustments to help keep the memory usage bounded better. Please give the nightly build a try in about an hour.

If you find that the usage is still on the high side, then I would suggest that you experiment with adjusting some cache sizes in your config; the items below are shown with their default values:

   -- This should be set to at least the sum of the number of lines in the panes in a tab.
   -- eg: if you have an 80x24 terminal split left/right then you should set this to at least 2x24 = 48
   -- Setting it smaller than that will harm performance
   line_quad_cache_size = 1024,

   -- Should also be set >= number of lines as above.
   -- Values are relatively small, may not need adjustment.
   line_state_cache_size = 1024,

   -- Should also be set >= number of lines as above.
   -- Values are relatively small, may not need adjustment.
   line_to_ele_shape_cache_size = 1024,

   -- should be >= the number of different attributed runs on the screen.
   -- hard to suggest a min size: try reducing until you notice performance getting bad.
   shape_cache_size = 1024,
dotdash commented 2 years ago

Memory usage seems to be growing slower, but it's still growing. Currently at 700MB with a single window opened.

And things got really sluggish. Cursor motion in vim when I hold down j isn't smooth at all. while it is perfectly smooth when I start a fresh instance of wezterm.

With strace attached I noticed that every keypress produces a >13k calls to clock_gettime:

117644 clock_gettime(CLOCK_MONOTONIC, {tv_sec=124814, tv_nsec=684613325}) = 0
117644 clock_gettime(CLOCK_MONOTONIC, {tv_sec=124814, tv_nsec=684661166}) = 0
117644 clock_gettime(CLOCK_MONOTONIC, {tv_sec=124814, tv_nsec=684697344}) = 0
117644 clock_gettime(CLOCK_MONOTONIC, {tv_sec=124814, tv_nsec=684732124}) = 0
...
117644 clock_gettime(CLOCK_MONOTONIC, {tv_sec=124815, tv_nsec=532758132}) = 0
117644 clock_gettime(CLOCK_MONOTONIC, {tv_sec=124815, tv_nsec=532813447}) = 0
117644 clock_gettime(CLOCK_MONOTONIC, {tv_sec=124815, tv_nsec=532868272}) = 0

There's nothing but clock_gettime in that part of the strace log.

With a fresh wezterm instance, the number of such calls per keypress is closer to 700.

dotdash commented 2 years ago

If tried to lower the settings you mentioned, and maybe that made a bit of a difference in terms of memory usage, but it also made the whole system get usably slow even quicker than before.

If you want me to try something specific, I'll be happy to help, but for daily work, I'll have to switch back to alacritty for now.

wfjt commented 2 years ago

I'm still seeing high memory usage (near 600 MiB) and stuttering in shell and especially nvim with the new settings applied. I tested nvim in Kitty and it's as fast as expected, only Wezterm lags and stutters. Attaching a recording of editing a text file in vim: both scrolling a plain text file and editing lags. Key repeats come in chunks and sometimes with 100ms+ delay.

As said, Kitty doesn't exhibit any of these issues so it's not nvim lagging. Same file as in recording is blazingly fast to scroll and edit without any stuttering.

If I open a new window it's slightly better until it deteriorates again.

The lag is so bad I had to switch to Kitty despite preferring Wezterm. Hope this can be fixed soon.

wezterm version: 20221023-205047-43f2265e
OpenGL version: Mesa Intel(R) Xe Graphics (TGL GT2) 4.6 (Compatibility Profile) Mesa 22.0.5
       PID  MEM (MiB)  %MEM  %CPU   EXE
    826840        790   5.0   3.8   /usr/lib/firefox/firefox
     75126        586   3.7   3.8   /usr/bin/wezterm-gui
      5990        501   3.1   4.2   /usr/lib/firefox/firefox
      5583        346   2.2   1.8   /usr/bin/gnome-shell

Also wonder why it's consuming 3.8% CPU, same as the top Firefox instance. Kitty running idle at the same time ranges 0.0%-0.6% and consumes 67 MiB. wezterm-recording-Krqt5O.cast.txt.zip

wez commented 2 years ago

I've made a couple of improvements to cache efficiency that may help here; I'd appreciate it if you could try a852e47dba2bf576b226b95ee1a329887a354e2a which should show up as a nightly build soonish.

I want to note that I cannot reproduce the growing/leaking characteristics that you have described; I wonder if there is something missing from your reproduction scenario?

I also wonder if there might be some graphics driver specific behavior that I just don't see: I'm using a radeon card.

dotdash commented 2 years ago

I've made a couple of improvements to cache efficiency that may help here; I'd appreciate it if you could try https://github.com/wez/wezterm/commit/a852e47dba2bf576b226b95ee1a329887a354e2a which should show up as a nightly build soonish.

I'll give it a spin, thanks!

I want to note that I cannot reproduce the growing/leaking characteristics that you have described; I wonder if there is something missing from your reproduction scenario?

The only thing I could think of would be something in my vim config. If you're interested, you can find it at https://github.com/dotdash/dotvim

I also wonder if there might be some graphics driver specific behavior that I just don't see: I'm using a radeon card.

I've seen the issue on both an nVidia 1080 TI as well as integrated graphics on a Ryzen 5600U.

listx commented 2 years ago

FWIW I use the Nix package manager and the issue is present (on both Mac and Linux) in the newest wezterm version packaged here: https://github.com/NixOS/nixpkgs/blob/497db553014c63d95f61a902bcde07761732b9ff/pkgs/applications/terminal-emulators/wezterm/default.nix#L33. This version is a little outdated (from September) but it's another data point that the excessive memory usage has been observed from other users as well.

For me it's not a big issue because I use tmux to manage session state, so I just restart wezterm once a day or two (or three) whenever the memory creeps up past the 1 GB mark (and tmux has everything for me where I left off). Interestingly on Linux the wezterm instance appears to make X server reach 100% CPU for ~5-10 seconds when (1) I leave wezterm open for a while and it has > 500MB memory usage and (2) I close wezterm or hover my mouse over it to gain focus, but I think that's a separate problem related to my X configuration with Xmonad.

dotdash commented 2 years ago

Interestingly on Linux the wezterm instance appears to make X server reach 100% CPU [...], but I think that's a separate problem related to my X configuration with Xmonad.

I've seen constant CPU usage over 60% with 1-2 second 100% spikes when wezterm has been running for a while, so probably not specific to your setup.

dotdash commented 2 years ago

With wezterm started about 3 hours ago and currently 6 windows opened (one at 426x111, the others at 212x111), I see 1.4G RSS, while it was at 1.6GB a few minutes ago. Xorg CPU usages hovers around 20%.

wezterm 20221114-144425-a852e47d

dotdash commented 2 years ago

Some more data points: The same wezterm instance from before is still running, the memory usage stayed around 1.4-1.5G, but for a few minutes now, whenever I move my mouse, Xorg CPU usage skyrockets, and moving the mouse quickly makes the whole screen freeze. I've closed all gui apps except wezterm and the freezing effect persisted. Stopping wezterm immediately made the problem go away.

nvidia-smi output:

Wed Nov 16 14:55:39 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 510.85.02    Driver Version: 510.85.02    CUDA Version: 11.6     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:0A:00.0  On |                  N/A |
| 13%   55C    P0    68W / 275W |    708MiB / 11264MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A     10608      G   /usr/lib/xorg/Xorg                468MiB |
|    0   N/A  N/A     11492      G   /usr/bin/wezterm-gui                7MiB |
|    0   N/A  N/A    267604      G   firefox                           229MiB |
+-----------------------------------------------------------------------------+
listx commented 2 years ago

For me the symptom of high CPU usage/lagginess when moving the mouse in/out of a long-running wezterm instance is only present on Linux with X server (where I use an NVIDIA card as well). This lagginess is not present on my M1 Mac (but the high memory usage is still there).

listx commented 2 years ago

There is an open issue for high CPU usage at https://github.com/wez/wezterm/issues/2665 which appears to describe very similar symptoms.

correabuscar commented 2 years ago

There is an open issue for high CPU usage at #2665 which appears to describe very similar symptoms.

To summarize, the solution for that seems to be:

Please use cargo build --release like the docs suggest; turning on all features at once is not recommended!

src: https://github.com/wez/wezterm/issues/2665#issuecomment-1315604042

ie. don't do this cargo build --release --all-features

listx commented 2 years ago

There is an open issue for high CPU usage at #2665 which appears to describe very similar symptoms.

To summarize, the solution for that seems to be:

Please use cargo build --release like the docs suggest; turning on all features at once is not recommended!

src: #2665 (comment)

ie. don't do this cargo build --release --all-features

Indeed this is the first thing I checked but AFAICS the Nix build already does the recommended setting: https://github.com/NixOS/nixpkgs/blob/497db553014c63d95f61a902bcde07761732b9ff/pkgs/applications/terminal-emulators/wezterm/default.nix#L80

dotdash commented 2 years ago

I'm using wez's Debian11 Nightly package, which I assume is built with the recommended settings.

correabuscar commented 2 years ago

This might not apply here in the case of wezterm(aside from the fact that they're both written in Rust and maybe using Vec`s), but here's what I'm doing to alacritty to avoid memory usage doubling due to Vec doubling its capacity.

listx commented 2 years ago

The new release has landed in Nixpkgs here and I've been using it for over 48 hours. Normally memory would exceed 1GB by this point but it is 259MB. This is at least a 75% reduction in memory usage. In addition, all of the CPU spike behavior and lagginess has disappeared!

Thank you @wez for the new release! I assume this matter can be closed if @dotdash can also confirm the new improvements on their end.

listx commented 2 years ago

In addition, all of the CPU spike behavior and lagginess has disappeared!

I have spoken too soon (by a few hours...?!). The lag has returned and I had to restart WezTerm to make it go away. For now I'll just lurk at #2665.

vars1ty commented 1 year ago

Happens here as well (even when building from source), noticed it when I was checking btop and kept seeing WezTerm climbing up non-stop.

1 - WezTerm 2 - Kitty

https://user-images.githubusercontent.com/54314240/222287935-6286fbfa-b3cb-4036-a7a9-d2490d69434c.mp4

Edit: Config:


local wezterm = require 'wezterm'

return {
    colors = {
        tab_bar = {
            inactive_tab_edge = "none",
            active_tab = {
                bg_color = '#1a1a1a',
                fg_color = 'silver'
            },

            inactive_tab = {
                bg_color = '#0a0a0a',
                fg_color = '#808080'
            },

            inactive_tab_hover = {
                bg_color = '#202020',
                fg_color = '#909090'
            },

            new_tab = {
                bg_color = '#0a0a0a',
                fg_color = '#808080'
            },

            new_tab_hover = {
                bg_color = '#202020',
                fg_color = '#909090'
            },
        },

        background = 'transparent',
        cursor_bg = 'silver',
        cursor_border = 'white',
        selection_bg = 'silver',
    },

    window_frame = {
        active_titlebar_bg = "#0a0a0a",
        inactive_titlebar_bg = "#0a0a0a",
        font_size = 10.0
    },

    font = wezterm.font 'JetBrains Mono',
    font_size = 10.0
}
ecmatthee commented 1 year ago

I am having the same issue as @vars1ty from the Nixpkgs. Memory usage keeps climbing over time (even with only one window open).

I am using wezterm with the default config on Hyprland (and thus wayland). I am on NixOS and all packages are on the unstable branch of nixpkg.

The following are my observation using btop: The initial wezterm window (which launches the wezterm-gui process) takes about 150MB of memory. Memory usage then rapidly climbs (around 3MB/s) to 210MB before it rate of increase slows to around 1MB/s at 220MB and further decreases to 0.1MB/s at 230MB. It should be noted that the memory usage keeps climbing even after this point but much slower.

There seems to be a 10MB increase in memory usage by wezterm-gui when the second window (this only happens to the second window, all successive windows only seem to add 1MB at most) and it gives a short burst to the above mentioned rate (pushes it to about 1.5MB/s for 2 seconds), this seems to only happen for the first 4-7 windows.

No memory is freed after windows are closed (with at least 1 window still open). Over the period of 2 hours the memory usage climbed to 330MB (with only a single open window). There seems to be an occasional burst of 1MB/s over 5-8 seconds but I can't figure out what causes it. Over the period of 30 hours (window was left open over night) the memory usage was 1.1GB. After closing said window and relaunching the process the above pattern repeats.

Unlike @listx, I have not experienced any lag or cpu spikes during the above testing or normal usage - so that seems to be an x11 specific bug.

EDIT: It should be noted the system was running for the entire duration of the 30 hour test - I don't have any idle or hibernation setup on the system. The display was turned off (as in the manual switch on the monitor) for about 7 hours if that makes any difference.

ecmatthee commented 1 year ago

After more testing I realized that running btop in wezterm might contaminate the results. I ran btop in foot and left wezterm to idle. This did not cause an increasing in memory (thus stayed static at 144MB). Running a simple command like ls or cd did not cause an increase in memory usage.

Commands that produce lots of visual output (such as tree) cause memory usage to jump about 5MB and simply running journalctl and scrolling down caused an increase of 40MB (it should be noted rerunning the commands, and thus getting the same output, does not increase memory usage). TUI progams that remain mostly static (such as neovim when idle or calcurse) caused an increase of about 2MB.

Running something like Donut.c increases the memory usage by about 10MB before it plateaus, whereas something like this falling matric script causes a constant increase in memory between 0.5MB/s-1.5MB/s depending on the amount of visual clutter. The memory usage eventually plateaus - I assume it is because all permutations of the falling text has been seen.

Running btop in wezterm (like @vars1ty) causes a constant increase in memory usage that never stops (see my previous comment). That being said the rate of increase is directly related to update time (by default 2000ms) of btop.

The above information leads me to believe (please note I have not read wezterm's source code so this is an educated guess) that wezterm either:

  1. Caches visual output to memory. This would explain why rerunning commands cause no increase in memory and dynamic animations stop increasing memory after they loop. This would also explain why btop causes constant increases in memory as which slows down with infrequent boosts - it is directly related on the general randomness of other processes resource and network usage, as this would produce new "noise" as output.
  2. Fails to properly free the visual buffer - terminal such as foot never have an increase in memory usage after printing data, but if said visual data where to be kept in memory we would see an increase in memory usage.

Either way the issue seems to be state/cache data which is kept in memory. Whether this is intentional (cache) or a bug the memory should be freed eventually.

DISCLAIMER: My machine does not have a swap partition/file, but does has zswap enabled (I believe is allotted 16GB) so I have no idea if said memory increases would have been written to swap if available - which does alleviate the problem somewhat, but is still undesirable.

wez commented 1 year ago

See also: https://github.com/wez/wezterm/issues/3815

ecmatthee commented 1 year ago

Thanks for the response, while I agree this isn't a priority issue for most workflows (and wezterm otherwise has great performance) this does make wezterm unsuitable on very low memory machines (as in 4GB or less).

I understand the feeling of burnout and would like to thank you for your work thus far. It might be worth marking this issue as a known-issue or closing this one and opening an new issue with an "enhancement" tag (something like "Improve Memory Allocation") just to centralize the discussion and increase visibility - maybe someone else has the time to delve into the problem.

Again, thank you. I have only been using Wezterm for a week and I am very happy with the terminal's performance on my desktop thus far. I guess I will just have to stick with Foot on my old laptop (4GB ram) for the time being :)

CyberShadow commented 7 months ago

Some users set unrealistically large scrollback sizes without considering that it costs them RAM for each pane

I don't know how much this actually contributes to wezterm's memory usage specifically, but from browsing the code, I noticed that the Cell type (which seems to be a core type that directly affects memory usage, scaled with the scrollback size) is not very memory efficient.

For example, here is SmallColor, a type used in two Cell members:

https://github.com/wez/wezterm/blob/cce0706b1f2a9e2d1f02c57f2d1cd367c91df1ae/termwiz/src/cell.rs#L18-L21

Here, PaletteIndex is an u8, so it takes one byte. But when added to an enum with another option, it is now two bytes. Since there are two such fields (one for the foreground color and one for the background), it means that only (257*257)/(256^4) = 0.0015% bit representations are meaningful, with the other 99.9984% wasted.

(One way to improve this could be to move the representation of the "has (non-)default foreground/background color" situation into two new bits in the attributes bitfield, which would allow storing the colors themselves as just u8s, unused if the corresponding bits are (un)set.)

rxvt-unicode is very careful about the per-cell overhead, and in fact has avoided adding more attributes because it "ran out of bits" in its cell attribute bitfield. The author is considering moving to extents to represent attributes (since they're often the same for contiguous cells), which would allow significantly lower memory use, but would also significantly complicate all involved algorithms.

ghost commented 2 months ago

This really should be fixed, one of the things holding me back from using wezterm, I run awesomewm and with wezterm it takes up almost the same amount of ram usage as running kde/gnome simply because of this issue. To compare, stterm takes about 16mb RAM idle and when stuff is open it's about 20-30MB. I can see it being around the 50-100mb range because of all the features but still about 400mb of ram for a terminal emulator is way too much