swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.75k stars 1.35k forks source link

Large fingerprint store causes build failures #6602

Open freef4ll opened 1 year ago

freef4ll commented 1 year ago

Description

For CI servers that are building continuously and updating to the latest releases, the ~/Library/org.swift.swiftpm/security/fingerprints has reached some 49k fingerprints and I'm speculating that somewhere file descriptor limit has been reached, causing strange build failures such as:

error: Invalid manifest (compiled with: ["/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swiftc", "-vfsoverlay", "/var/folders/53/lkfcmw916cg2k01t65tt60380000gw/T/TemporaryDirectory.doVQzq/vfs.yaml", "-L", "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-lPackageDescription", "-Xlinker", "-rpath", "-Xlinker", "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-target", "arm64-apple-macosx12.0", "-sdk", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk", "-F", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks", "-I", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-L", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib", "-swift-version", "5", "-I", "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/pm/ManifestAPI", "-sdk", "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk", "-package-description-version", "5.6.0", "/Package.swift", "-Xfrontend", "-disable-implicit-concurrency-module-import", "-Xfrontend", "-disable-implicit-string-processing-module-import", "-o", "/var/folders/53/lkfcmw916cg2k01t65tt60380000gw/T/TemporaryDirectory.76W3Hp/swift-nio-ssl-manifest"])
/Package.swift:21:14: error: class 'ProcessInfo' does not exist in module 'Foundation'
import class Foundation.ProcessInfo
             ^          ~~~~~~~~~~~
/Package.swift:38:8: error: cannot find 'ProcessInfo' in scope
    if ProcessInfo.processInfo.environment["SWIFTCI_USE_LOCAL_DEPS"] == nil {
       ^~~~~~~~~~~ in https://github.com/apple/swift-nio-ssl.git

As soon as fingerprints directory is moved, build succeeds.

The default file limits are in place and same under sysctl:

$ sudo launchctl limit   
    cpu         unlimited      unlimited      
    filesize    unlimited      unlimited      
    data        unlimited      unlimited      
    stack       8372224        67092480       
    core        unlimited      unlimited      
    rss         unlimited      unlimited      
    memlock     unlimited      unlimited      
    maxproc     5333           8000           
    maxfiles    256            unlimited      

I'll update the case when I'm able to reproduce this behaviour again, have observed this on 2 different CI machines.

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.75.2 Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100)
Target: arm64-apple-macosx13.0
Darwin ice.local 22.5.0 Darwin Kernel Version 22.5.0: Mon Apr 24 20:52:24 PDT 2023; root:xnu-8796.121.2~5/RELEASE_ARM64_T6000 arm64
tomerd commented 1 year ago

cc @yim-lee @neonichu

neonichu commented 1 year ago

Sounds like we should have some kind of limit for fingerprints and delete older ones if we reach it?