[SR-10613] High memory usage when compiling dependencies with SPM on Linux

[swift-ci]

Previous ID	SR-10613
Radar	rdar://problem/50542597
Original Reporter	Tainnor (JIRA User)
Type	Bug

Attachment: Download

Environment

Ubuntu 16.04 Swift 5.0.1

Additional Detail from JIRA

| | | |------------------|-----------------| |Votes | 2 | |Component/s | Compiler | |Labels | Bug, 5.0Regression | |Assignee | None | |Priority | Medium | md5: db9578961ffd2983b17e0faa05077960

Issue Description:

I'm observing a very high memory usage when compiling SPM dependencies on Linux with Swift 5 that I wasn't able to observe with Swift 4. This is causing problems on CI.

One example where this happens is the Kitura repository.

Steps to reproduce:

- Run "docker run -it --entrypoint /bin/bash swift:5.0.1-xenial"
- Run "docker stats" in another window
- Inside the container run "apt install openssl libssl-dev libcurl4-openssl-dev"
- Then run "git clone https://github.com/IBM-Swift/Kitura.git && cd Kitura"
- Then run "swift build" and observe the output of "docker stats"

I'm not sure if this somehow depends on the host machine, but on my Mac, docker stats will show a memory increase to up to about 1.3GB before going down.

Repeating the previous steps with the swift:4.2.4 base image instead doesn't show the same memory behaviour (here memory stays consistently far below 1GB).

[swift-ci]

Comment by markus sähn (JIRA)

I can reproduce this and I see memory usages of \~5.5GB. It looks like that the memory is increasing exponential by the number of cores which are used. My example uses max 300MB when I use 1 core, but when I use 8 cores it is increasing to 5.6GB.

To reproduce it:

- Run "docker run -it --entrypoint /bin/bash swift:5.0.1-xenial"
- Run "docker stats" in another window
- Inside the container run "mkdir Bug && cd Bug && swift package init --name Bug --type executable"
- Then run: 
echo '// swift-tools-version:4.2
// The swift-tools-version declares the minimum version of Swift required to build this package.import PackageDescriptionlet package = Package(
    name: "Bug",
    dependencies: [
        .package(url: "https://github.com/lukaskubanek/OrderedDictionary.git", .exact("2.1.0")),
        .package(url: "https://github.com/PerfectlySoft/Perfect-NewRelic-linux.git", .upToNextMinor(from: "3.0.1")),
        .package(url: "https://github.com/Quick/Nimble.git", .upToNextMajor(from: "8.0.1")),
        .package(url: "https://github.com/evgenyneu/SigmaSwiftStatistics.git", .upToNextMajor(from: "9.0.2")),
        .package(url: "https://github.com/IBM-Swift/LoggerAPI.git", .upToNextMajor(from: "1.8.0")),
        .package(url: "https://github.com/typelift/SwiftCheck.git", .exact("0.12.0"))
    ],
    targets: [
        .target(
            name: "Bug",
            dependencies: ["OrderedDictionary", "PerfectNewRelic", "Nimble", "SigmaSwiftStatistics", "LoggerAPI", "SwiftCheck"]
        )
    ]
)
' > Package.swift
- Then run "swift build" and observe the output of "docker stats"

[belkadan]

I wonder if there's a bad edge case here buffering output, if one of the dependencies has tons of warnings or something.

[swift-ci]

Comment by markus sähn (JIRA)

Any update on the ticket? It feels like nobody is having big swift projects so we are the only team with such issues?

[weissi]

CC @belkadan/@aciidb0mb3r I have recently heard quite a few reports of people running out of memory on CI and being killed. We really should look at this. This might also be SwiftPM spawning a very large number of swiftc processes?

Thanks papierschiff (JIRA User) for the repro instructions.

[aciidgh]

SwiftPM uses `ProcessInfo.processInfo.activeProcessorCount` to set the `num-threads` on swiftc invocations and then llbuild also has its own parallelism. I think there is a `--jobs` option to limit that.

[weissi]

See my attached screenshot and screen recording. I see the container taking up to about 1.6 GB of memory even with this comparatively small project. There are about 9 swift processes each having more than 200 MB resident which then adds up...

[weissi]

@aciidb0mb3r

$ docker run -it --rm --privileged swift:5.0.2
root@5fa18204d503:/# swift
import Foundation
Welcome to Swift version 5.0.2 (swift-5.0.2-RELEASE).
Type :help for assistance.
  1> import Foundation
  2> ProcessInfo.processInfo.activeProcessorCount
$R0: Int = 4

So it thinks it's 4 CPUs but I get 9 compilers running and -j isn't a thing in 5.0

$ swift --version
Swift version 5.0.2 (swift-5.0.2-RELEASE)
Target: x86_64-unknown-linux-gnu
johannes:~/devel/swift (jw-bench-find-string)
$ jw-docker-swift-5.0 swift build -j 1
error: unknown option -j; use --help to list available options