[SR-9551] Failure to run benchmarks with build-script

[56ed3025-6cad-4bf8-b972-9c308205a0af]

Previous ID	SR-9551
Radar	None
Original Reporter	@palimondo
Type	Bug

Additional Detail from JIRA

| | | |------------------|-----------------| |Votes | 0 | |Component/s | Compiler | |Labels | Bug, BuildScript | |Assignee | None | |Priority | Medium | md5: 172d28eeb0a096aefb2802a72fbc6c5a

Issue Description:

Running benchmarks with build-script like this:
swift-source $ ./swift/utils/build-script -R -B --no-assertions
fails because the values for --independent-samples parameter are somehow not passed on here.

Here's the tail of the full console output:

[1/1] cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/...ipt /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.p
FAILED: benchmark/CMakeFiles/check-swift-benchmark-macosx-x86_64
cd /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o O --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples  && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver run -o Onone --output-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --independent-samples  && /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/bin/Benchmark_Driver compare --log-dir /Users/mondo/Developer/swift-source/build/Ninja-Release/swift-macosx-x86_64/benchmark/logs --swift-repo /Users/mondo/Developer/swift-source/swift --compare-script /Users/mondo/Developer/swift-source/swift/benchmark/scripts/compare_perf_tests.py
usage: Benchmark_Driver run [-h] [-f PATTERN] [-t TESTS] [-o OPT]
                            [-i INDEPENDENT_SAMPLES] [--output-dir OUTPUT_DIR]
                            [--swift-repo SWIFT_REPO]
                            [BENCHMARK [BENCHMARK ...]]
Benchmark_Driver run: error: argument -i/--independent-samples: expected one argument

ninja: build stopped: subcommand failed.

See also https://forums.swift.org/t/changes-to-how-build-script-runs-benchmarks/18730

[belkadan]

@atrick, who do you think should be looking at this?

[atrick]

@palimondo I've never run the benchmarks using the build-script. Didn't know it was possible. Was `--independent-samples` added to the driver, or did someone change the build-script causing it to break?

If you know the responsible party can you ask them?

I attempted to debug the build-script last night for a different reason and can say for sure that either it's a trivial fix (which you could handle) or it will take me more time than I have right now. (I'm also strongly opposed to opaque layers of abstraction over software configuration).

Meanwhile, I'll file a radar to track it.

[atrick]

\rdar://problem/46874722\ [SR-9551] Failure to run benchmarks with build-script

I also indicated in the radar that I'd be happy removing this option from the build-script.

[56ed3025-6cad-4bf8-b972-9c308205a0af]

I'm aware that build-script isn't used internally by many Apple devs now. But it's the officially documented method, right there in the first paragraph.

The Benchmark_Driver parameter `--independent-samples` is not new, it was just renamed from the old and misleading --iterations few months ago.

On casual inspection of build-script, everything seems to be correct. There's also no apparent culprit to ask for a fix. I asked in the forums before filing this, but got dead silence... This mysterious issue was there once before: SR-4357

[atrick]

@palimondo Things like this usually get fixed by devs who have sufficient motivation to do it in their spare time. It sounds like you at least have the motivation.

[56ed3025-6cad-4bf8-b972-9c308205a0af]

:-) I do have are a very particular set of skills...