Open swift-ci opened 5 years ago
Comment by markus sähn (JIRA)
I can reproduce this and I see memory usages of \~5.5GB. It looks like that the memory is increasing exponential by the number of cores which are used. My example uses max 300MB when I use 1 core, but when I use 8 cores it is increasing to 5.6GB.
To reproduce it:
- Run "docker run -it --entrypoint /bin/bash swift:5.0.1-xenial"
- Run "docker stats" in another window
- Inside the container run "mkdir Bug && cd Bug && swift package init --name Bug --type executable"
- Then run:
echo '// swift-tools-version:4.2
// The swift-tools-version declares the minimum version of Swift required to build this package.import PackageDescriptionlet package = Package(
name: "Bug",
dependencies: [
.package(url: "https://github.com/lukaskubanek/OrderedDictionary.git", .exact("2.1.0")),
.package(url: "https://github.com/PerfectlySoft/Perfect-NewRelic-linux.git", .upToNextMinor(from: "3.0.1")),
.package(url: "https://github.com/Quick/Nimble.git", .upToNextMajor(from: "8.0.1")),
.package(url: "https://github.com/evgenyneu/SigmaSwiftStatistics.git", .upToNextMajor(from: "9.0.2")),
.package(url: "https://github.com/IBM-Swift/LoggerAPI.git", .upToNextMajor(from: "1.8.0")),
.package(url: "https://github.com/typelift/SwiftCheck.git", .exact("0.12.0"))
],
targets: [
.target(
name: "Bug",
dependencies: ["OrderedDictionary", "PerfectNewRelic", "Nimble", "SigmaSwiftStatistics", "LoggerAPI", "SwiftCheck"]
)
]
)
' > Package.swift
- Then run "swift build" and observe the output of "docker stats"
I wonder if there's a bad edge case here buffering output, if one of the dependencies has tons of warnings or something.
Comment by markus sähn (JIRA)
Any update on the ticket? It feels like nobody is having big swift projects so we are the only team with such issues?
CC @belkadan/@aciidb0mb3r I have recently heard quite a few reports of people running out of memory on CI and being killed. We really should look at this. This might also be SwiftPM spawning a very large number of swiftc
processes?
Thanks papierschiff (JIRA User) for the repro instructions.
SwiftPM uses `ProcessInfo.processInfo.activeProcessorCount` to set the `num-threads` on swiftc invocations and then llbuild also has its own parallelism. I think there is a `--jobs` option to limit that.
See my attached screenshot and screen recording. I see the container taking up to about 1.6 GB of memory even with this comparatively small project. There are about 9 swift
processes each having more than 200 MB resident which then adds up...
@aciidb0mb3r
$ docker run -it --rm --privileged swift:5.0.2
root@5fa18204d503:/# swift
import Foundation
Welcome to Swift version 5.0.2 (swift-5.0.2-RELEASE).
Type :help for assistance.
1> import Foundation
2> ProcessInfo.processInfo.activeProcessorCount
$R0: Int = 4
So it thinks it's 4 CPUs but I get 9 compilers running and -j
isn't a thing in 5.0
$ swift --version
Swift version 5.0.2 (swift-5.0.2-RELEASE)
Target: x86_64-unknown-linux-gnu
johannes:~/devel/swift (jw-bench-find-string)
$ jw-docker-swift-5.0 swift build -j 1
error: unknown option -j; use --help to list available options
Attachment: Download
Environment
Ubuntu 16.04 Swift 5.0.1Additional Detail from JIRA
| | | |------------------|-----------------| |Votes | 2 | |Component/s | Compiler | |Labels | Bug, 5.0Regression | |Assignee | None | |Priority | Medium | md5: db9578961ffd2983b17e0faa05077960Issue Description:
I'm observing a very high memory usage when compiling SPM dependencies on Linux with Swift 5 that I wasn't able to observe with Swift 4. This is causing problems on CI.
One example where this happens is the Kitura repository.
Steps to reproduce:
I'm not sure if this somehow depends on the host machine, but on my Mac, docker stats will show a memory increase to up to about 1.3GB before going down.
Repeating the previous steps with the swift:4.2.4 base image instead doesn't show the same memory behaviour (here memory stays consistently far below 1GB).