search

swiftlang / swift

The Swift Programming Language
https://swift.org
Apache License 2.0
67.57k stars 10.36k forks source link

Increased peak memory usage in 5.8 #63938

Open freef4ll opened 1 year ago

freef4ll commented 1 year ago

Our CI stack is running in a Docker container which is restricted on memory usage, it was noticed that with Swift 5.8 toolchain sometimes runs out of memory in the container with 7GB memory limit.

Investigating a clean build comparison between the two toolchains, it is visible that peak memory usage with 5.7 is under 3.7GB, while with 5.8 this reaches 7GB. The memory consumption occurs when a large number of swift-frontend processes are created and are alive simultaneously (100-200+). It appears that the individual swift-frontend memory usage has risen in 5.8.

Steps to reproduce Monitor peak memory usage in a container for the same project and a clean build.

Start a container with 5.7 and 5.8:

docker run -v $HOME:/build --name swift5.7 -ti swift:5.7-focal /bin/bash
docker run -v $HOME:/build --name swift5.7 -ti swiftlang/swift:nightly-5.8-jammy /bin/bash

Set up process monitoring:

while true; do echo "$(date): $(docker stats --no-stream --format "{{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}"  swift5.7)" ; sleep 1; done >> docker_stats_5.7

while true; do echo "$(date): $(docker stats --no-stream --format "{{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}"  swift5.8)" ; sleep 1; done >> docker_stats_5.8

In each of them also can setup processes monitoring every second:

docker exec -ti swift5.7 top -o RES -b 1 > top_batch_5.7
docker exec -ti swift5.8 top -o RES -b 1 > top_batch_5.8

With this setup, it is visible that the peak is at 16:06:30-16:06:43

Mon Feb 27 16:06:27 UTC 2023: swift5.8  104.08% 46.89MiB / 125.7GiB
Mon Feb 27 16:06:30 UTC 2023: swift5.8  99.10%  80.21MiB / 125.7GiB
Mon Feb 27 16:06:33 UTC 2023: swift5.8  2342.30%        4.486GiB / 125.7GiB
Mon Feb 27 16:06:36 UTC 2023: swift5.8  2201.98%        7.044GiB / 125.7GiB <---- 7GB
Mon Feb 27 16:06:40 UTC 2023: swift5.8  2250.05%        4.862GiB / 125.7GiB
Mon Feb 27 16:06:43 UTC 2023: swift5.8  2238.20%        1.426GiB / 125.7GiB
Mon Feb 27 16:06:46 UTC 2023: swift5.8  408.49% 383.6MiB / 125.7GiB
Mon Feb 27 16:06:49 UTC 2023: swift5.8  1587.51%        371.6MiB / 125.7GiB
Mon Feb 27 16:06:52 UTC 2023: swift5.8  2112.01%        1.066GiB / 125.7GiB
Mon Feb 27 16:06:55 UTC 2023: swift5.8  545.89% 437.8MiB / 125.7GiB

And top analysis shows swift-frontend processes get created, each using up to 200MB of memory:

top - 16:06:36 up 304 days, 47 min,  0 users,  load average: 20.00, 15.43, 14.16
Tasks: 212 total,  91 running, 117 sleeping,   0 stopped,   4 zombie
%Cpu0  : 87.1 us, 11.6 sy,  0.0 ni,  1.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu2  : 87.3 us, 10.0 sy,  0.0 ni,  2.3 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st 
%Cpu4  : 86.3 us, 11.7 sy,  0.0 ni,  2.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu6  : 86.0 us, 12.7 sy,  0.0 ni,  1.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu8  : 95.7 us,  3.7 sy,  0.0 ni,  0.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu10 : 89.1 us,  9.3 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu12 : 89.0 us,  9.6 sy,  0.0 ni,  1.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu14 : 89.3 us,  9.0 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu16 : 89.0 us, 10.7 sy,  0.0 ni,  0.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu18 : 90.7 us,  9.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu20 : 97.0 us,  3.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu22 : 86.4 us, 12.6 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu24 : 91.7 us,  5.6 sy,  0.0 ni,  2.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu26 : 84.7 us, 12.3 sy,  0.0 ni,  3.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu28 : 91.0 us,  7.3 sy,  0.0 ni,  1.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
%Cpu30 : 86.8 us, 11.9 sy,  0.0 ni,  1.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem : 128729.6 total,   7202.2 free,  67575.5 used,  53951.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  59949.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  16293 actions   20   0  426196 207604 123080 S  34.9   0.2   0:01.05 swift-f+
  15906 actions   20   0  360912 192736  65596 R  45.2   0.1   0:01.36 clang
  16202 actions   20   0  334256 177596 111288 S  28.6   0.1   0:00.86 swift-f+
  16238 actions   20   0  333000 174792 109744 S  25.2   0.1   0:00.76 swift-f+
  16233 actions   20   0  331944 172720 108784 S  19.3   0.1   0:00.58 swift-f+
  16157 actions   20   0  328088 169916 109940 R  16.3   0.1   0:00.49 swift-f+
  16074 actions   20   0  326804 169272 108116 R  18.3   0.1   0:00.55 swift-f+
  16371 actions   20   0  334184 168600 102380 S  25.2   0.1   0:00.76 swift-f+
  16322 actions   20   0  324920 166996 109856 R  16.9   0.1   0:00.51 swift-f+
  16609 actions   20   0  332300 166628 102092 S  21.9   0.1   0:00.66 swift-f+
  16735 actions   20   0  331772 165880 101940 S  20.3   0.1   0:00.61 swift-f+
  16617 actions   20   0  331832 165076 101080 S  19.9   0.1   0:00.60 swift-f+
  16139 actions   20   0  323572 165044 109292 R  18.6   0.1   0:00.56 swift-f+
  16706 actions   20   0  330484 164044 101412 S  17.9   0.1   0:00.54 swift-f+
  16508 actions   20   0  331296 163232  99896 S  18.9   0.1   0:00.57 swift-f+
  16237 actions   20   0  330920 162992  99956 S  19.6   0.1   0:00.59 swift-f+
  16088 actions   20   0  320104 162152 109744 R  16.6   0.1   0:00.50 swift-f+
  16720 actions   20   0  330824 162116  99184 S  18.6   0.1   0:00.56 swift-f+
  16712 actions   20   0  330336 161924  99464 S  17.9   0.1   0:00.54 swift-f+
  16644 actions   20   0  327952 159756  99748 R  16.3   0.1   0:00.49 swift-f+
  16751 actions   20   0  326036 159500 101236 R  17.3   0.1   0:00.52 swift-f+
  16412 actions   20   0  324688 156736 100200 R  15.3   0.1   0:00.46 swift-f+
  16478 actions   20   0  321848 156076 101884 R  15.0   0.1   0:00.45 swift-f+
  16275 actions   20   0  311688 154748 110336 R  12.6   0.1   0:00.38 swift-f+
  16312 actions   20   0  314040 154172 107752 R  13.0   0.1   0:00.39 swift-f+
  16532 actions   20   0  319420 153196 101396 R  13.6   0.1   0:00.41 swift-f+
  16464 actions   20   0  318004 152868 102392 R  14.6   0.1   0:00.44 swift-f+
  16177 actions   20   0  318412 152336 101376 R  14.3   0.1   0:00.43 swift-f+
  16331 actions   20   0  309488 152128 109748 R  14.6   0.1   0:00.44 swift-f+
  16764 actions   20   0  318112 151852 101032 R  12.3   0.1   0:00.37 swift-f+
  16071 actions   20   0  365900 151336 103992 R  20.9   0.1   0:00.63 swift-f+
  16255 actions   20   0  305780 149492 110968 R  18.9   0.1   0:00.57 swift-f+
  16622 actions   20   0  314336 147548 100584 R  13.3   0.1   0:00.40 swift-f+
  16657 actions   20   0  314964 147344  99904 R  12.6   0.1   0:00.38 swift-f+
  16505 actions   20   0  312092 146860 102168 R  15.3   0.1   0:00.46 swift-f+
  16696 actions   20   0  312752 146676 101172 R  12.0   0.1   0:00.36 swift-f+

With 5.7, the peak memory for me doesn't go over at more than 3.7GB and here only managed 2GB peak:

27 16:20:50 UTC 2023: swift5.7  7.10%   27.11MiB / 125.7GiB
Mon Feb 27 16:20:53 UTC 2023: swift5.7  100.97% 33.77MiB / 125.7GiB
Mon Feb 27 16:20:56 UTC 2023: swift5.7  411.05% 341.5MiB / 125.7GiB
Mon Feb 27 16:20:59 UTC 2023: swift5.7  210.12% 67.35MiB / 125.7GiB
Mon Feb 27 16:21:01 UTC 2023: swift5.7  1719.73%        2.144GiB / 125.7GiB
Mon Feb 27 16:21:04 UTC 2023: swift5.7  2234.33%        1.874GiB / 125.7GiB
Mon Feb 27 16:21:07 UTC 2023: swift5.7  2182.42%        599.8MiB / 125.7GiB
Mon Feb 27 16:21:10 UTC 2023: swift5.7  1053.11%        205.7MiB / 125.7GiB
Mon Feb 27 16:21:13 UTC 2023: swift5.7  1502.74%        625.4MiB / 125.7GiB
Mon Feb 27 16:21:17 UTC 2023: swift5.7  317.82% 385.3MiB / 125.7GiB
Mon Feb 27 16:21:20 UTC 2023: swift5.7  862.58% 392.8MiB / 125.7GiB
Mon Feb

and from top each one is also using less memory:

top - 16:21:03 up 304 days,  1:01,  0 users,  load average: 10.26, 11.51, 13.44
Tasks: 224 total,  95 running, 127 sleeping,   0 stopped,   2 zombie
%Cpu0  : 32.8 us,  5.0 sy,  0.0 ni, 50.6 id,  0.0 wa,  0.0 hi, 11.5 si,  0.0 st
%Cpu1  : 40.6 us,  6.1 sy,  0.0 ni, 48.9 id,  0.0 wa,  0.0 hi,  4.5 si,  0.0 st
%Cpu2  : 39.7 us,  7.3 sy,  0.0 ni, 51.7 id,  0.0 wa,  0.0 hi,  1.3 si,  0.0 st
%Cpu3  : 68.8 us,  2.9 sy,  0.0 ni, 24.7 id,  0.6 wa,  0.0 hi,  2.9 si,  0.0 st
%Cpu4  : 42.6 us,  7.0 sy,  0.0 ni, 49.7 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu5  : 40.3 us,  7.3 sy,  0.0 ni, 50.8 id,  0.0 wa,  0.0 hi,  1.7 si,  0.0 st
%Cpu6  : 72.4 us,  5.6 sy,  0.0 ni, 21.4 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu7  : 69.0 us,  7.3 sy,  0.0 ni, 23.3 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu8  : 37.4 us,  6.8 sy,  0.0 ni, 55.4 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu9  : 29.1 us, 11.7 sy,  0.0 ni, 59.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu10 : 39.0 us,  7.3 sy,  0.0 ni, 53.3 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu11 : 31.8 us,  9.0 sy,  0.0 ni, 58.9 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu12 : 34.9 us,  6.4 sy,  0.0 ni, 58.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu13 : 55.1 us,  6.6 sy,  0.0 ni, 37.5 id,  0.0 wa,  0.0 hi,  0.7 si,  0.0 st
%Cpu14 : 33.3 us,  5.3 sy,  0.0 ni, 60.0 id,  0.0 wa,  0.0 hi,  1.3 si,  0.0 st
%Cpu15 : 46.7 us,  4.6 sy,  0.0 ni, 48.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu16 : 68.8 us,  3.3 sy,  0.0 ni, 28.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu17 : 40.4 us,  5.6 sy,  0.0 ni, 53.6 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu18 : 41.1 us,  2.7 sy,  0.0 ni, 55.9 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu19 : 37.2 us,  4.7 sy,  0.0 ni, 58.1 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu20 : 36.2 us,  7.0 sy,  0.0 ni, 56.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu21 : 98.7 us,  1.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu22 : 32.7 us,  8.7 sy,  0.0 ni, 58.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu23 : 36.6 us,  6.4 sy,  0.0 ni, 57.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu24 : 42.1 us,  7.0 sy,  0.0 ni, 51.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu25 : 34.8 us, 11.4 sy,  0.0 ni, 53.8 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu26 : 36.7 us,  4.0 sy,  0.0 ni, 59.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu27 : 44.2 us,  7.3 sy,  0.0 ni, 48.2 id,  0.0 wa,  0.0 hi,  0.3 si,  0.0 st
%Cpu28 : 33.3 us,  8.1 sy,  0.0 ni, 58.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu29 : 66.1 us,  2.7 sy,  0.0 ni, 31.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu30 : 96.3 us,  1.3 sy,  0.0 ni,  2.3 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
%Cpu31 : 95.3 us,  4.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem : 128729.6 total,  10229.0 free,  64513.4 used,  53987.2 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.  63008.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   6476 actions   20   0  307156 137848  62312 R  17.7   0.1   0:00.53 clang
   6483 actions   20   0  258584 123476  62628 R  13.3   0.1   0:00.40 clang
   6720 actions   20   0  277968 122804  67420 R  15.3   0.1   0:00.46 swift-f+
   6982 actions   20   0  268616 114164  67692 R  10.7   0.1   0:00.32 swift-f+
   6595 actions   20   0  231544 113572  66644 R  19.7   0.1   0:00.59 clang
   6719 actions   20   0  268240 113540  67700 R  10.0   0.1   0:00.30 swift-f+
   6516 actions   20   0  225808 110524  69424 R  14.3   0.1   0:00.43 clang
   6512 actions   20   0  295512 108700  70524 R  14.0   0.1   0:00.42 clang
   6801 actions   20   0  262512 107280  66816 R   9.0   0.1   0:00.27 swift-f+
   6511 actions   20   0  283012 106884  66100 S  11.0   0.1   0:00.33 clang
   6517 actions   20   0  221016 105380  68712 R  14.0   0.1   0:00.42 clang
   6594 actions   20   0  216468  98712  66040 R  10.0   0.1   0:00.30 clang
   6597 actions   20   0  211940  97000  68268 R   9.3   0.1   0:00.28 clang
   6520 actions   20   0  210708  95156  67368 R   8.7   0.1   0:00.26 clang
   6639 actions   20   0  208864  93312  67284 R   8.0   0.1   0:00.24 clang
   6991 actions   20   0  246660  92816  67712 R   5.3   0.1   0:00.16 swift-f+
   6662 actions   20   0  208364  92376  67116 R   7.0   0.1   0:00.21 clang
   7098 actions   20   0  240176  92076  73688 R   4.0   0.1   0:00.12 swift-f+
   6563 actions   20   0  207648  91796  67044 R   6.3   0.1   0:00.19 clang
   6730 actions   20   0  247116  91712  66092 R   5.0   0.1   0:00.15 swift-f+
   6953 actions   20   0  245548  91152  67352 R   4.7   0.1   0:00.14 swift-f+
   6850 actions   20   0  244336  90340  67648 R   5.0   0.1   0:00.15 swift-f+
   6815 actions   20   0  245804  89072  64812 R   4.7   0.1   0:00.14 swift-f+
   6651 actions   20   0  305152  88844  58840 S   9.0   0.1   0:00.27 swift-f+
   6892 actions   20   0  242852  88576  67408 R   4.3   0.1   0:00.13 swift-f+
   7029 actions   20   0  241744  88128  67784 R   4.7   0.1   0:00.14 swift-f+
   6928 actions   20   0  242636  87176  66188 R   4.3   0.1   0:00.13 swift-f+
   7071 actions   20   0  241172  87124  67580 R   3.7   0.1   0:00.11 swift-f+
   7050 actions   20   0  237460  86732  70916 R   5.3   0.1   0:00.16 swift-f+
   7188 actions   20   0  240188  86464  67724 R   3.3   0.1   0:00.10 swift-f+
   7148 actions   20   0  240712  86392  67348 R   4.3   0.1   0:00.13 swift-f+
   6934 actions   20   0  239684  85628  67360 R   3.7   0.1   0:00.11 swift-f+
   7027 actions   20   0  238908  85492  68068 R   4.7   0.1   0:00.14 swift-f+
   7295 actions   20   0  234840  85012  71768 R   2.7   0.1   0:00.08 swift-f+

Expected behavior

Restricted memory usage growth during the peak.

Should the rate at which swift-frontend processes are spawned be controlled?

Environment

 ################################################################
 #                                                              #
 # Swift Nightly Docker Image                                   #
 # Tag: swift-5.8-DEVELOPMENT-SNAPSHOT-2023-02-23-a                     #
 #                                                              #
 ################################################################
actions@e02dc215fcbd:~$ swift --version
Swift version 5.8-dev (LLVM 44d4f9d4b49845f, Swift b9562e1a860ec0b)
Target: x86_64-unknown-linux-gnu

And 5.7:

$ swift --version
Swift version 5.7.3 (swift-5.7.3-RELEASE)
Target: x86_64-unknown-linux-gnu
0xTim commented 11 months ago

@freef4ll this should be solved in 5.9 I believe