This package lets you collect and easily visualize performance data about data structure implementations and collection algorithms. It was created to help develop the Swift Collections package, but it's useful for so much more!
This project primarily concentrates on benchmarking Swift code, but it can also be used to run benchmarks (and, especially, to analyze benchmark results) in other languages, too.
Here is a short benchmark, measuring the performance of Array.sorted()
and Set.contains(_:)
:
import CollectionsBenchmark
var benchmark = Benchmark(title: "Demo Benchmark")
benchmark.addSimple(
title: "Array<Int> sorted",
input: [Int].self
) { input in
blackHole(input.sorted())
}
benchmark.add(
title: "Set<Int> contains",
input: ([Int], [Int]).self
) { input, lookups in
let set = Set(input)
return { timer in
for value in lookups {
precondition(set.contains(value))
}
}
}
benchmark.main()
Here is how you run it:
$ swift run -c release benchmark run results --cycles 5
Running 2 tasks on 76 sizes from 1 to 1M:
Array<Int> sorted
Set<Int> contains
Output file: /Users/klorentey/Projects/swift-collections-benchmark-demo/Demo/results
Appending to existing data (if any) for these tasks/sizes.
Collecting data:
1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.31s
1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.35s
1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.29s
1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.3s
1.2.4...8...16...32...64...128...256...512...1k...2k...4k...8k...16k...32k...64k...128k...256k...512k...1M -- 5.34s
Finished in 26.6s
$ swift run -c release benchmark render results chart.png
$ open chart.png
And this is what you get:
Today I learned that sorting 20 integers in an Array
takes about as much time as looking at all items in a 20-member Set
. Fascinating! 🤓
For a tour of the features provided by this library, please be sure check out our Getting Started Guide!
Swift Collections Benchmark is intended primarily as a developer tool, rather than something that would be used in production apps.
It exposes a source-level API and a command line interface, neither of which are technically stable yet. After an initial period of experimentation, we expect to stabilize both interfaces. Until then, new releases may sometimes come with changes that might break existing benchmark definitions, or that may change the command line interface in ways that could break existing scripts. We'll try our best to keep this to a minimum, though (or mitigate with a multi-release deprecation period), even during this chaotic initial period.
To use this package in a SwiftPM project, add the following line to the dependencies in your Package.swift
file:
.package(url: "https://github.com/apple/swift-collections-benchmark", from: "0.0.1"),
In the typical case, you'll want to set up a standalone executable target that is dedicated to benchmarking:
// swift-tools-version:5.3
import PackageDescription
let package = Package(
name: "MyPackage",
products: [
.executable(name: "my-benchmark", targets: ["MyBenchmark"]),
],
dependencies: [
.package(url: "https://github.com/apple/swift-collections-benchmark", from: "0.0.1"),
// ... other dependencies ...
],
targets: [
// ... other targets ...
.target(
name: "MyBenchmark",
dependencies: [
.product(name: "CollectionsBenchmark", package: "swift-collections-benchmark"),
]),
]
)
We can use the Swift Collections Forum to ask and answer questions on how to use or work on this package. It's also a great place to discuss its evolution.
If you find something that looks like a bug, please open a Bug Report! Fill out as many details as you can.
(Note: The package doesn't currently come with many tests, reflecting its origins as a supporting project -- we strive to improve that!)
By submitting a pull request, you represent that you have the right to license your contribution to Apple and the community, and agree by submitting the patch that your contributions are licensed under the Swift License, a copy of which is provided in this repository.
Like all Swift.org projects, we would like the Swift Collections Benchmark project to foster a diverse and friendly community. We expect contributors to adhere to the Swift.org Code of Conduct. A copy of this document is available in this repository.
The current code owner of this package is Karoy Lorentey (@lorentey). You can contact him on the Swift forums, or by writing an email to klorentey at apple dot com. (Please keep it related to this project.)
In case of moderation issues, you can also directly contact a member of the Swift Core Team.