search

swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.74k stars 1.34k forks source link

Package loading is slow for large packages #6769

Open adam-fowler opened 1 year ago

adam-fowler commented 1 year ago

Description

Package loading seems unnecessarily slow for larger packages. This can be seen when running swift package describe or during a swift package resolve. The package I am seeing this most often in is https://github.com/soto-project/soto. Yes it is a large package with over 300 targets, but it can take over 15 seconds to run swift package describe on it. This seems like an incredibly long time to load the Package.swift and list all the files in each target.

A resolve can spend a similar amount of time computing the version for Soto.

Expected behavior

I would expect package loading to be faster than this

Actual behavior

No response

Steps to reproduce

  1. git clone https://github.com/soto-project/soto
  2. cd soto
  3. swift package describe

Swift Package Manager version/commit hash

No response

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.75.2 Apple Swift version 5.8.1 (swiftlang-5.8.0.124.5 clang-1403.0.22.11.100) Target: arm64-apple-macosx13.0 Darwin Adams-MBP-M1-Max.local 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:20 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T6000 arm64

MaxDesiatov commented 1 year ago

Tracked internally as rdar://113207176

neonichu commented 1 year ago

swift package describe is about loading the entire package graph, so it’s not limited to what is in the package itself but the entire graph.

neonichu commented 1 year ago

The package has 12 dependencies, that's really not that much, so ~15s seems like a lot.

MaxDesiatov commented 4 months ago

This is likely to be resolved by https://github.com/swiftlang/swift-package-manager/pull/7724, https://github.com/swiftlang/swift-package-manager/pull/7672, and https://github.com/swiftlang/swift-package-manager/pull/7452. Due to current overabundant use of semaphores waiting for callbacks to complete we've seen a variety of issues caused by this, including thread explosion, and sometimes even deadlocks in package loading and resolution.

The PRs are currently blocked due to SwiftPM plugins IPC not supporting backpressure, which will be exacerbated with a transition to async with messages possibly arriving out-of-order and other possible reentrancy issues. We need to lay some groundwork there first, which will allow us to preserve correctness in the Swift Concurrency transition.