search

dart-lang / sdk

The Dart SDK, including the VM, JS and Wasm compilers, analysis, core libraries, and more.
https://dart.dev
BSD 3-Clause "New" or "Revised" License
10.06k stars 1.56k forks source link

Memory leak in Dart VM? #44009

Open wytrych opened 3 years ago

wytrych commented 3 years ago

I'm running a HTTP server which is using websockets. (Dart version used is 2.9.3) The load is about 500-1500 clients at any time with sessions lifetime ~30 minutes on average. I noticed a high increase in memory over the days and did some inspecting and I've encountered something strange.

Observatory reports memory usage to be ~255 MB. But when I pmap the process I can see that it's using 1675440K, and the amount reported by Observatory is I think equal to the dart processes but there is a huge amount of memory taken by [ anon ]. See output from pmap:

1:   /usr/bin/dart --enable-vm-service:8181/0.0.0.0 bin/server.dart
000055fb527de000  25676K r---- dart
000055fb540f1000   7672K r-x-- dart
000055fb5486f000    568K r---- dart
000055fb548fd000    224K rw--- dart
000055fb54935000   1760K rw---   [ anon ]
000055fb569a8000 1230056K rw---   [ anon ]
00007fc8a3b80000    512K rw-s- memfd:dart-newspace (deleted)
00007fc8a3c00000    512K rw-s- memfd:dart-newspace (deleted)
00007fc8a4180000    512K rw-s- memfd:dart-newspace (deleted)
00007fc8a4800000    512K rw-s- memfd:dart-newspace (deleted)
...(around 760 more lines of memfd:dart-newspace (deleted))

The memory consumption is increasing daily with the average load being constant.

devoncarew commented 3 years ago

Triaging this to the io library under the assumption that the memory growth is related to the io libraries. We'd need to actually profile this to know though.

@wytrych - when you use the Observatory, do you have a sense for which objects are leaking? From your report we might be leading native resources, but they could have some corresponding dart objects.

wytrych commented 3 years ago

@devoncarew No I don't see anything suspicious when looking at the memory stack, there doesn't seem to be any increase in the memory that is seen in the Observatory. I found this issue and take extra care to unsubscribe to any listeners after a connection is closed but it doesn't look like it helped much. Do you have any tips on how can I provide you extra info to help with debugging?

wytrych commented 3 years ago

The bulk of the memory as seen by Observatory is _Int64List and _ZlibDeflateFilter but these total to about 160MB so looks ok. I'm attaching some screenshots if it's of any help:

Screenshot 2020-11-02 at 21 33 44 Screenshot 2020-11-02 at 21 35 25 Screenshot 2020-11-02 at 21 37 58
wytrych commented 3 years ago

@devoncarew any updates on this?

mraleph commented 3 years ago

It looks like your application is leaking websocket connections for some reason. You need to look at the snapshot to try and figure out retaining paths leading to _WebSocketPerMessageDeflate.

wytrych commented 3 years ago

Hmm how did you come to that conclusion @mraleph ? The 1328 instances of _WebSocketPerMessageDeflate look correct to me, as I explained previously there are usually between 500 and 1500 live connections, and the number of _WebSocketPerMessageDeflate instances correlates to that well. If I were leaking the websocket connections would I not see it somehow in the snapshot in a more meaningful way than that mysterious block of 1200MB memory that Observatory doesn't understand?

mraleph commented 3 years ago

@wytrych ah, I assumed that snapshot is taken not under load. in that case indeed it looks fine. In this case I think there is some sort of native memory leak, might also be that there is no leak but malloc implementation has for some reason accumulated a lot of fragmentation. I think this will be pretty hard to diagnose without reproduction. We might need to build you some custom dart binaries to dump malloc implementation state.

Here is an idea. Try running with TCMALLOC_SKIP_SBRK environment variable set to 1 (export TCMALLOC_SKIP_SBRK=1) and see if memory consumption pattern changes.

wytrych commented 3 years ago

Thanks for the tip @mraleph I'll try that in the next few days. I'm running this in a Kubernetes cluster on the google/dart-base image so I'd be surprised if that had a broken malloc implementation?

mraleph commented 3 years ago

malloc implementation is linked into dart binary statically - we use tcmalloc. All memory allocators have some sort of issues and failure modes, I would not exclude that as a possibility (in fact we have seen strange behaviour before - causing us to roll our replacements in few places, e.g. 20dd7c88e9890ead04a8bc7b44483bc1d6ec4556 ). You might very well be hitting one of those cases.

mraleph commented 3 years ago

You can also try setting TCMALLOC_RELEASE_RATE to 10 to force tcmalloc aggressively return memory to the OS.

wytrych commented 3 years ago

So I've been running the server with the settings you suggested @mraleph but it doesn't look like it changed much unfortunately. Here's my memory/cpu graphs: Screenshot 2020-11-20 at 15 18 02

mraleph commented 3 years ago

Did pmap output change in any way?

wytrych commented 3 years ago

Here it is, it looks a little bit different maybe, but similar still:

1:   /usr/bin/dart --enable-vm-service:8181/0.0.0.0 bin/server.dart
000055aef0e7a000  25676K r---- dart
000055aef278d000   7672K r-x-- dart
000055aef2f0b000    568K r---- dart
000055aef2f99000    224K rw--- dart
000055aef2fd1000   1760K rw---   [ anon ]
000055aef3d1d000   9220K rw---   [ anon ]
00007ff0bdc80000  18432K rw---   [ anon ]
00007ff0bf000000   7168K rw---   [ anon ]
00007ff0bf980000  21504K rw---   [ anon ]
00007ff0c1900000  38912K rw---   [ anon ]
00007ff0c4000000   1024K rw---   [ anon ]
00007ff0c5100000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0c5e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c5e80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c5f80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c6100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c6180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c6280000   1024K rw---   [ anon ]
00007ff0c6500000  17408K rw---   [ anon ]
00007ff0c7700000  17408K rw---   [ anon ]
00007ff0c8980000   8192K rw---   [ anon ]
00007ff0c9700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c9780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c9800000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0c9900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c9980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0c9a00000  21504K rw---   [ anon ]
00007ff0caf80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0cb000000  20480K rw---   [ anon ]
00007ff0cc480000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0cc500000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0cc580000   1024K rw---   [ anon ]
00007ff0cc700000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0cc800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0cc880000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0cc900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0cc980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0cca00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0cca80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0ccb00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0ccb80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0ccc80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0ccd00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0ccd80000  22528K rw---   [ anon ]
00007ff0ce400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0ce500000  22528K rw---   [ anon ]
00007ff0cfb80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0cfc00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0cfc80000   8192K rw---   [ anon ]
00007ff0d0500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0d0580000  48128K rw---   [ anon ]
00007ff0d3500000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0d3580000  16384K rw---   [ anon ]
00007ff0d467d000      4K -----   [ anon ]
00007ff0d467e000   1024K rw---   [ anon ]
00007ff0d477e000      4K -----   [ anon ]
00007ff0d477f000   1024K rw---   [ anon ]
00007ff0d487f000      4K -----   [ anon ]
00007ff0d4880000   1024K rw---   [ anon ]
00007ff0d4a00000  21504K rw---   [ anon ]
00007ff0d5f80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0d6000000  25600K rw---   [ anon ]
00007ff0d7980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0d7a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0d7b00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0d7b80000  19456K rw---   [ anon ]
00007ff0d8f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0d8f80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0d9080000  14336K rw---   [ anon ]
00007ff0d9f00000  14336K rw---   [ anon ]
00007ff0dad80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0dae80000  23552K rw---   [ anon ]
00007ff0dc5ff000      4K -----   [ anon ]
00007ff0dc600000  26624K rw---   [ anon ]
00007ff0de080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0de100000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0de280000  18432K rw---   [ anon ]
00007ff0df500000   9216K rw---   [ anon ]
00007ff0dfe80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0dff00000  14336K rw---   [ anon ]
00007ff0e0d80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e0e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e0f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e0f80000   1024K rw---   [ anon ]
00007ff0e1100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e1200000  19456K rw---   [ anon ]
00007ff0e2580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e2600000  17408K rw---   [ anon ]
00007ff0e377f000      4K -----   [ anon ]
00007ff0e3780000  24576K rw---   [ anon ]
00007ff0e5000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e5080000  23552K rw---   [ anon ]
00007ff0e6900000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0e6980000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0e6a00000   2048K rw---   [ anon ]
00007ff0e6c7f000      4K -----   [ anon ]
00007ff0e6c80000   1024K rw---   [ anon ]
00007ff0e6d80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0e6e7f000      4K -----   [ anon ]
00007ff0e6e80000   3072K rw---   [ anon ]
00007ff0e7200000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0e7280000  22528K rw---   [ anon ]
00007ff0e8900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e8980000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0e8a00000   7168K rw---   [ anon ]
00007ff0e9180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e9200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0e9300000   1024K rw---   [ anon ]
00007ff0e9480000  15360K rw---   [ anon ]
00007ff0ea400000   1024K rw---   [ anon ]
00007ff0ea580000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0ea600000  23552K rw---   [ anon ]
00007ff0ebd80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0ebe80000  20480K rw---   [ anon ]
00007ff0ed37e000      4K -----   [ anon ]
00007ff0ed37f000   1024K rw---   [ anon ]
00007ff0ed47f000      4K -----   [ anon ]
00007ff0ed480000   8192K rw---   [ anon ]
00007ff0edd00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0edd80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0ede00000   4096K rw---   [ anon ]
00007ff0ee280000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0ee300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0ee380000  11264K rw---   [ anon ]
00007ff0eef00000  11264K rw---   [ anon ]
00007ff0efa80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0efb00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0efb80000  16384K rw---   [ anon ]
00007ff0f0c00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f0d00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f0e00000  33792K rw---   [ anon ]
00007ff0f2f7f000      4K -----   [ anon ]
00007ff0f2f80000   1024K rw---   [ anon ]
00007ff0f3100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0f3180000  11264K rw---   [ anon ]
00007ff0f3d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0f3d80000   2048K rw---   [ anon ]
00007ff0f4000000   1024K rw---   [ anon ]
00007ff0f4100000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f41ff000      4K -----   [ anon ]
00007ff0f4200000   9216K rw---   [ anon ]
00007ff0f4b7f000      4K -----   [ anon ]
00007ff0f4b80000   7168K rw---   [ anon ]
00007ff0f52ff000      4K -----   [ anon ]
00007ff0f5300000   1024K rw---   [ anon ]
00007ff0f5480000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f557f000      4K -----   [ anon ]
00007ff0f5580000   1024K rw---   [ anon ]
00007ff0f5700000  26624K rw---   [ anon ]
00007ff0f7180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0f7200000   4096K rw---   [ anon ]
00007ff0f7680000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f7780000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f7800000  17408K rw---   [ anon ]
00007ff0f897f000      4K -----   [ anon ]
00007ff0f8980000  10240K rw---   [ anon ]
00007ff0f9400000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f9480000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0f9500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0f9600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0f9680000  11264K rw---   [ anon ]
00007ff0fa200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0fa280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0fa380000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0fa400000  19456K rw---   [ anon ]
00007ff0fb780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0fb800000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0fb880000  25600K rw---   [ anon ]
00007ff0fd200000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0fd300000   9216K rw---   [ anon ]
00007ff0fdc80000   2048K rw---   [ anon ]
00007ff0fde80000    512K r-xs- memfd:dart-codespace (deleted)
00007ff0fdf00000    512K r--s- memfd:dart-codespace (deleted)
00007ff0fe000000   3072K rw---   [ anon ]
00007ff0fe380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff0fe400000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0fe480000    512K rw-s- memfd:dart-newspace (deleted)
00007ff0fe500000  17408K rw---   [ anon ]
00007ff0ff680000   7168K rw---   [ anon ]
00007ff0ffe00000   2048K rw---   [ anon ]
00007ff100080000   2048K rw---   [ anon ]
00007ff100300000   8192K rw---   [ anon ]
00007ff100b80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff100c00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff100c80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff100d00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff100e00000  16384K rw---   [ anon ]
00007ff101e80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff101f80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff102080000    512K rw-s- memfd:dart-newspace (deleted)
00007ff102100000    512K r-xs- memfd:dart-codespace (deleted)
00007ff102200000    512K rw-s- memfd:dart-newspace (deleted)
00007ff102280000    512K rw-s- memfd:dart-newspace (deleted)
00007ff102300000    512K rw-s- memfd:dart-newspace (deleted)
00007ff102380000  64512K rw---   [ anon ]
00007ff106300000    512K rw-s- memfd:dart-newspace (deleted)
00007ff106380000   1024K rw---   [ anon ]
00007ff106500000    512K rw-s- memfd:dart-newspace (deleted)
00007ff106600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff106680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff106700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff106780000  77824K rw---   [ anon ]
00007ff10b380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10b780000  13312K rw---   [ anon ]
00007ff10c500000  44032K rw---   [ anon ]
00007ff10f000000    512K r-xs- memfd:dart-codespace (deleted)
00007ff10f080000    512K r--s- memfd:dart-codespace (deleted)
00007ff10f100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff10f600000    512K rw-s- memfd:dart-newspace (deleted)
00007ff10f680000    512K rw-s- memfd:dart-newspace (deleted)
00007ff10f700000    512K rw-s- memfd:dart-newspace (deleted)
00007ff10f800000    512K rw-s- memfd:dart-newspace (deleted)
00007ff10f880000   2048K rw---   [ anon ]
00007ff10fb00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff10fb80000   3072K rw---   [ anon ]
00007ff10ff00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff10ff80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff110000000   1024K rw---   [ anon ]
00007ff110180000   3072K rw---   [ anon ]
00007ff110500000    512K r--s- memfd:dart-codespace (deleted)
00007ff110580000   1024K rw---   [ anon ]
00007ff110700000   3072K rw---   [ anon ]
00007ff110a80000   1024K rw---   [ anon ]
00007ff110c00000   2048K rw---   [ anon ]
00007ff110e80000   1024K rw---   [ anon ]
00007ff111000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff111080000   2048K rw---   [ anon ]
00007ff111300000   1024K rw---   [ anon ]
00007ff111480000   3072K rw---   [ anon ]
00007ff111800000   1024K rw---   [ anon ]
00007ff111980000    512K rw-s- memfd:dart-newspace (deleted)
00007ff111a00000   2048K rw---   [ anon ]
00007ff111c80000   2048K rw---   [ anon ]
00007ff111f00000   1024K rw---   [ anon ]
00007ff112080000    512K rw-s- memfd:dart-newspace (deleted)
00007ff112100000   1024K rw---   [ anon ]
00007ff112280000   1024K rw---   [ anon ]
00007ff112400000   2048K rw---   [ anon ]
00007ff112680000   1024K rw---   [ anon ]
00007ff112800000   2048K rw---   [ anon ]
00007ff112a80000   1024K rw---   [ anon ]
00007ff112c00000   3072K rw---   [ anon ]
00007ff112f80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff113000000   2048K rw---   [ anon ]
00007ff113280000   1024K rw---   [ anon ]
00007ff113400000   2048K rw---   [ anon ]
00007ff113680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff113700000   2048K rw---   [ anon ]
00007ff113980000   7168K rw---   [ anon ]
00007ff114100000   7168K rw---   [ anon ]
00007ff114880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114b80000    512K r-xs- memfd:dart-codespace (deleted)
00007ff114c00000    512K r--s- memfd:dart-codespace (deleted)
00007ff114c80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114d80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff114e78000      4K -----   [ anon ]
00007ff114e79000   1024K rw---   [ anon ]
00007ff114f79000      4K -----   [ anon ]
00007ff114f7a000   1024K rw---   [ anon ]
00007ff11507a000      4K -----   [ anon ]
00007ff11507b000   1024K rw---   [ anon ]
00007ff11517b000      4K -----   [ anon ]
00007ff11517c000   1024K rw---   [ anon ]
00007ff11527c000      4K -----   [ anon ]
00007ff11527d000   1024K rw---   [ anon ]
00007ff11537d000      4K -----   [ anon ]
00007ff11537e000   1024K rw---   [ anon ]
00007ff11547e000      4K -----   [ anon ]
00007ff11547f000   1024K rw---   [ anon ]
00007ff11557f000      4K -----   [ anon ]
00007ff115580000   1024K rw---   [ anon ]
00007ff115680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11577b000      4K -----   [ anon ]
00007ff11577c000   1024K rw---   [ anon ]
00007ff11587c000      4K -----   [ anon ]
00007ff11587d000   1024K rw---   [ anon ]
00007ff11597d000      4K -----   [ anon ]
00007ff11597e000   1024K rw---   [ anon ]
00007ff115a7e000      4K -----   [ anon ]
00007ff115a7f000   1024K rw---   [ anon ]
00007ff115b7f000      4K -----   [ anon ]
00007ff115b80000   1024K rw---   [ anon ]
00007ff115c80000    512K r-xs- memfd:dart-codespace (deleted)
00007ff115d00000    512K r--s- memfd:dart-codespace (deleted)
00007ff115dff000      4K -----   [ anon ]
00007ff115e00000   1024K rw---   [ anon ]
00007ff115f00000     68K rw-s- memfd:dart-oldspace (deleted)
00007ff116000000     68K rw-s- memfd:dart-oldspace (deleted)
00007ff116080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116100000    512K r-xs- memfd:dart-codespace (deleted)
00007ff116180000    512K r--s- memfd:dart-codespace (deleted)
00007ff116200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116400000    260K rw-s- memfd:dart-oldspace (deleted)
00007ff116480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116580000    512K r-xs- memfd:dart-codespace (deleted)
00007ff116600000    512K r--s- memfd:dart-codespace (deleted)
00007ff116680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff1169ff000      4K -----   [ anon ]
00007ff116a00000   1024K rw---   [ anon ]
00007ff116b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116b80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116c00000    640K rw-s- memfd:dart-oldspace (deleted)
00007ff116d00000    124K rw-s- memfd:dart-oldspace (deleted)
00007ff116d80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116e80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff116fc6000  20712K rw---   [ anon ]
00007ff118400000  16388K rw-s- memfd:dart-oldspace (deleted)
00007ff119480000   4100K rw-s- memfd:dart-oldspace (deleted)
00007ff119900000   1028K rw-s- memfd:dart-oldspace (deleted)
00007ff119a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff119b00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff119b80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff119c00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff119c80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff119d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff119d80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff119e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff119f00000    260K rw-s- memfd:dart-oldspace (deleted)
00007ff119f80000    260K rw-s- memfd:dart-oldspace (deleted)
00007ff11a000000    260K rw-s- memfd:dart-oldspace (deleted)
00007ff11a080000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11a100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11a980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11aa00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11aa80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ab00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ab80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ac00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ac80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ad00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ad80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ae00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11af00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11af80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11b080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b800000    512K r-xs- memfd:dart-codespace (deleted)
00007ff11b900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11b980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ba00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ba80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11bb00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11bb80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11bc80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11bd00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11bd80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11be00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11bf00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11bf80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c300000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11c380000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11c400000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11c500000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11c580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c680000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11c700000   1024K rw---   [ anon ]
00007ff11c880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11c980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ca80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11cb80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11cc00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11cd00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11cd80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11ce00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11ce80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11cf80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11d000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11d080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11d100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11d180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11d280000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11d300000   1024K rw---   [ anon ]
00007ff11d480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11d500000   1024K rw---   [ anon ]
00007ff11d680000   1024K rw---   [ anon ]
00007ff11d800000   1024K rw---   [ anon ]
00007ff11d980000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11da00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11da80000    260K rw-s- memfd:dart-oldspace (deleted)
00007ff11db00000    156K rw-s- memfd:dart-oldspace (deleted)
00007ff11db80000    512K r--s- memfd:dart-codespace (deleted)
00007ff11dc80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11dd00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11dd80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11de00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11de80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11df00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11df80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e300000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11e400000   1024K rw---   [ anon ]
00007ff11e580000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11e680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11e900000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11e980000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11ea00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11ea80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11eb00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11eb80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11ec80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11ed80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ee00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ef00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11ef80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11f080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f180000    512K rw-s- memfd:dart-newspace (deleted)
00007ff11f280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11f980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11fa00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11fa80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11fb00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11fb80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11fc00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11fc80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11fd80000   1024K rw---   [ anon ]
00007ff11fe80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ff00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff11ff80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120280000    512K rw-s- memfd:dart-newspace (deleted)
00007ff120380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120400000   1024K rw---   [ anon ]
00007ff120580000   1024K rw---   [ anon ]
00007ff120680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120900000     68K rw-s- memfd:dart-oldspace (deleted)
00007ff120980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120b80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120c00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120d00000   1024K rw---   [ anon ]
00007ff120e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120e80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff120f80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121780000    512K rw-s- memfd:dart-newspace (deleted)
00007ff121800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121b80000   1024K rw---   [ anon ]
00007ff121d00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff121d80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff121e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121e80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff121f80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122180000    512K rw-s- memfd:dart-newspace (deleted)
00007ff122200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122700000    512K rw-s- memfd:dart-newspace (deleted)
00007ff122800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff122a00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff122b00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff122c00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff122c80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff122d00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff122d80000   1024K rw---   [ anon ]
00007ff122f00000   1024K rw---   [ anon ]
00007ff123080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123280000    512K rw-s- memfd:dart-newspace (deleted)
00007ff123300000    320K rw-s- memfd:dart-oldspace (deleted)
00007ff123400000    512K rw-s- memfd:dart-newspace (deleted)
00007ff123480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123580000     88K rw-s- memfd:dart-oldspace (deleted)
00007ff123600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123b00000     76K rw-s- memfd:dart-oldspace (deleted)
00007ff123b80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff123c80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123d80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123e80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff123f80000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124000000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124080000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124100000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124180000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124200000   1024K rw---   [ anon ]
00007ff124300000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124380000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124500000    512K r-xs- memfd:dart-codespace (deleted)
00007ff124580000    512K r--s- memfd:dart-codespace (deleted)
00007ff124600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124700000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124780000    512K rw-s- memfd:dart-newspace (deleted)
00007ff124800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124980000    164K rw-s- memfd:dart-oldspace (deleted)
00007ff124a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124b80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124c00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124c80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124d80000    516K rw-s- memfd:dart-oldspace (deleted)
00007ff124e80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff124f80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125b80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125c00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125c80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125d80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125e00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125e80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125f00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff125f80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126000000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126080000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126100000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126180000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126200000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126280000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126300000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126380000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126400000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126480000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126500000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126680000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126780000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126900000    512K rw-s- memfd:dart-newspace (deleted)
00007ff126980000    512K rw-s- memfd:dart-newspace (deleted)
00007ff126a00000    512K rw-s- memfd:dart-newspace (deleted)
00007ff126a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126b80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff126c76000   8408K r-x-- kernel-service.dart.snapshot
00007ff1274ac000  23376K r---- kernel-service.dart.snapshot
00007ff128b80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff128c00000   1024K rw---   [ anon ]
00007ff128d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff128e00000    132K rw-s- memfd:dart-oldspace (deleted)
00007ff128e7c000      4K -----   [ anon ]
00007ff128e7d000   1024K rw---   [ anon ]
00007ff128f7d000      4K -----   [ anon ]
00007ff128f7e000   1024K rw---   [ anon ]
00007ff12907e000      4K -----   [ anon ]
00007ff12907f000   1024K rw---   [ anon ]
00007ff12917f000      4K -----   [ anon ]
00007ff129180000   1024K rw---   [ anon ]
00007ff129280000    512K r-xs- memfd:dart-codespace (deleted)
00007ff129300000    512K r--s- memfd:dart-codespace (deleted)
00007ff129380000    512K rw-s- memfd:dart-newspace (deleted)
00007ff129400000    512K r-xs- memfd:dart-codespace (deleted)
00007ff129480000    512K r--s- memfd:dart-codespace (deleted)
00007ff129580000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129600000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129680000    164K rw-s- memfd:dart-oldspace (deleted)
00007ff129700000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129780000    164K rw-s- memfd:dart-oldspace (deleted)
00007ff1297c0000     64K rw-s- memfd:dart-zone (deleted)
00007ff1297d0000     64K rw-s- memfd:dart-zone (deleted)
00007ff129800000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129880000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129900000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129980000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129a00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129a80000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129b00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129c00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129cc0000     64K rw-s- memfd:dart-zone (deleted)
00007ff129cd0000     64K rw-s- memfd:dart-zone (deleted)
00007ff129ce0000     64K rw-s- memfd:dart-zone (deleted)
00007ff129cf0000     64K rw-s- memfd:dart-zone (deleted)
00007ff129d00000    512K rw-s- memfd:dart-oldspace (deleted)
00007ff129d85000     64K rw-s- memfd:dart-zone (deleted)
00007ff129d95000     64K rw-s- memfd:dart-zone (deleted)
00007ff129da5000     64K rw-s- memfd:dart-zone (deleted)
00007ff129db5000     64K rw-s- memfd:dart-zone (deleted)
00007ff129dd5000     64K rw-s- memfd:dart-zone (deleted)
00007ff129de5000     16K r---- libresolv-2.28.so
00007ff129de9000     52K r-x-- libresolv-2.28.so
00007ff129df6000     16K r---- libresolv-2.28.so
00007ff129dfa000      4K ----- libresolv-2.28.so
00007ff129dfb000      4K r---- libresolv-2.28.so
00007ff129dfc000      4K rw--- libresolv-2.28.so
00007ff129dfd000      8K rw---   [ anon ]
00007ff129dff000      4K -----   [ anon ]
00007ff129e00000   1024K rw---   [ anon ]
00007ff129f00000    512K r-xs- memfd:dart-codespace (deleted)
00007ff129f80000    512K r--s- memfd:dart-codespace (deleted)
00007ff12a00b000     12K r---- libnss_files-2.28.so
00007ff12a00e000     28K r-x-- libnss_files-2.28.so
00007ff12a015000      8K r---- libnss_files-2.28.so
00007ff12a017000      4K ----- libnss_files-2.28.so
00007ff12a018000      4K r---- libnss_files-2.28.so
00007ff12a019000      4K rw--- libnss_files-2.28.so
00007ff12a01a000     24K rw---   [ anon ]
00007ff12a020000     64K rw-s- memfd:dart-zone (deleted)
00007ff12a030000     64K rw-s- memfd:dart-zone (deleted)
00007ff12a040000     64K rw-s- memfd:dart-zone (deleted)
00007ff12a050000     64K rw-s- memfd:dart-zone (deleted)
00007ff12a060000     64K rw-s- memfd:dart-zone (deleted)
00007ff12a080000    512K r--s- memfd:dart-oldspace (deleted)
00007ff12a100000      4K r---- libnss_dns-2.28.so
00007ff12a101000     16K r-x-- libnss_dns-2.28.so
00007ff12a105000      4K r---- libnss_dns-2.28.so
00007ff12a106000      4K r---- libnss_dns-2.28.so
00007ff12a107000      4K rw--- libnss_dns-2.28.so
00007ff12a108000   3092K rw-s- memfd:dart-timeline (deleted)
00007ff12a40d000      4K -----   [ anon ]
00007ff12a40e000   1032K rw---   [ anon ]
00007ff12a510000    136K r---- libc-2.28.so
00007ff12a532000   1312K r-x-- libc-2.28.so
00007ff12a67a000    304K r---- libc-2.28.so
00007ff12a6c6000      4K ----- libc-2.28.so
00007ff12a6c7000     16K r---- libc-2.28.so
00007ff12a6cb000      8K rw--- libc-2.28.so
00007ff12a6cd000     16K rw---   [ anon ]
00007ff12a6d1000     52K r---- libm-2.28.so
00007ff12a6de000    636K r-x-- libm-2.28.so
00007ff12a77d000    852K r---- libm-2.28.so
00007ff12a852000      4K r---- libm-2.28.so
00007ff12a853000      4K rw--- libm-2.28.so
00007ff12a854000     24K r---- libpthread-2.28.so
00007ff12a85a000     60K r-x-- libpthread-2.28.so
00007ff12a869000     24K r---- libpthread-2.28.so
00007ff12a86f000      4K r---- libpthread-2.28.so
00007ff12a870000      4K rw--- libpthread-2.28.so
00007ff12a871000     16K rw---   [ anon ]
00007ff12a875000      4K r---- libdl-2.28.so
00007ff12a876000      4K r-x-- libdl-2.28.so
00007ff12a877000      4K r---- libdl-2.28.so
00007ff12a878000      4K r---- libdl-2.28.so
00007ff12a879000      4K rw--- libdl-2.28.so
00007ff12a87a000      8K rw---   [ anon ]
00007ff12a87f000      4K r---- ld-2.28.so
00007ff12a880000    120K r-x-- ld-2.28.so
00007ff12a89e000     32K r---- ld-2.28.so
00007ff12a8a6000      4K r---- ld-2.28.so
00007ff12a8a7000      4K rw--- ld-2.28.so
00007ff12a8a8000      4K rw---   [ anon ]
00007ffe4ff4a000    136K rw---   [ stack ]
00007ffe4ffb9000     12K r----   [ anon ]
00007ffe4ffbc000      8K r-x--   [ anon ]
ffffffffff600000      4K r-x--   [ anon ]
 total          1663916K
wytrych commented 3 years ago

So the difference I can see is that the memory is more fragmented now, there doesn't seem to be one huge chunk of memory anymore, but the total memory occupied is similar.

wytrych commented 3 years ago

@mraleph is there anything I can do to help you debug this?

mraleph commented 3 years ago

I don't really know if there is a good way to debug this - without a reproduction available on our side - looks like some sort of native memory leak. You need to profile malloc allocations on your server somehow, e.g. by using something like https://github.com/CarloWood/libmemleak

Do you use any third party packages which would include native code or is your application 100% Dart?

wytrych commented 3 years ago

We use very few dependencies in that server, nothing seems to have binary code AFAIK, is there a way to check that for sure? Do you think running the server compiled with dart2native could be an improvement?

mraleph commented 3 years ago

We use very few dependencies in that server, nothing seems to have binary code AFAIK, is there a way to check that for sure?

I don't think there is an easy way to check if you have native dependencies. The easiest I can come up with is to delete your pubcache (usually in $HOME/.pub-cache) then run pub get in your server and then search find $HOME/.pub-cache -iname '*.so'. That should reveal any packages with native code inside. But I'd assume there are none given you answer.

Do you think running the server compiled with dart2native could be an improvement?

You could try, but I would not bet on this. The leak is in native code so compiling Dart itself should not make a difference (unless it is JIT compiler that is leaking - and JIT compiler uses arena allocator so all memory should be freed once it exits).

Have you tried provoking the leak in local development environment? e.g. by starting the server and then simulating client connections using some simple local client which just connects to your websocket does something (whatever your application is doing and disconnects?) and doing it again and again.

mraleph commented 3 years ago

Actually, I remembered that we have a similar memory leak report sitting stale. Try applying this workaround: https://github.com/dart-lang/sdk/issues/27414#issuecomment-406459872 (make sure to cancel subscriptions to WebSocket.listen when you get onDone). I think this might be the leak.

I will take a look at the repro we have at https://github.com/dart-lang/sdk/issues/27414 to see if I understand where it leaks.

wytrych commented 3 years ago

Yep I found that issue @mraleph and I applied these changes too but it doesn't seem to help :(

I will try some local reproduction .

seceba commented 1 year ago

Any news? I have an active chat app and it is constantly running out of memory.

Every chat takes up space in the memory and is not deleted, so the memory is becoming insufficient.

mraleph commented 1 year ago

Any news? I have an active chat app and it is constantly running out of memory.

Is this on client side or server side? Do you use WebSocket.listen?

/cc @brianquinlan

seceba commented 1 year ago

Hello, this is happening on the server side. I did a few experiments. 500 people are connected to the socket and when they talk to each other only with short chat sentences, there is no memory overflow.

It seems that they are somehow deleted from memory.

But when users start sending each other large 1-2mb images, then the memory starts to fill up and stays full even if no one is connected to the socket.

I also tried using socket.close() in the onDone method, no change.

Ps: Im using this package, https://github.com/rknell/alfred/blob/2f79287e8b1f1bf85d05515034b642b436f5ff4e/lib/src/type_handlers/websocket_type_handler.dart