[SR-11576] [Workaround in place] Flaky Driver tests

[typesanitizer]

Previous ID	SR-11576
Radar	rdar://problem/53507844
Original Reporter	@typesanitizer
Type	Bug
Status	Reopened
Resolution

Additional Detail from JIRA

| | | |------------------|-----------------| |Votes | 0 | |Component/s | Compiler | |Labels | Bug, Driver, Test | |Assignee | @beccadax | |Priority | Medium | md5: 030f49a4cea15f1cd5854a4e823825ae

Issue Description:

I've occasionally run into issues with the tests Driver/linker.swift, Driver/linker-clang_rt.swift and Driver/embed-bitcode.swift being flaky.

The following test results are with e052da7d8886fa0439677852e8f7830b20c2e1da (tag: swift-DEVELOPMENT-SNAPSHOT-2019-09-25-a) but I've seen the same issues at other points as well.

Here's a Python runner to check this:

#!/usr/bin/env python3

import subprocess
import sys

total_runs = 200
num_fails = 0

for _ in range(total_runs):
  devnull = subprocess.DEVNULL
  done = subprocess.run(['../llvm/utils/lit/lit.py',
    '../build/Ninja-RelWithDebInfoAssert/swift-macosx-x86_64/test-macosx-x86_64/Driver',
    '--filter=(linker|embed-bitcode)'], stdout=devnull, stderr=devnull)
  if done.returncode != 0:
    num_fails += 1
    sys.stdout.write('-')
  else:
    sys.stdout.write('+')
  sys.stdout.flush()

print("\nEncountered {} failures out of {} total runs".format(
    num_fails, total_runs))

Running the script once gave me 4 failures out of 200 runs.

N.B. The script also runs 4 other tests, not sure how those are interacting with the flaky tests, but it seems hard/impossible to get the failure if only the 2 flaky tests are run.

[typesanitizer]

Fixed by PR 27780.

[beccadax]

What landed in #27780 is a workaround for the problem, which is actually a bug in macOS. I’ve filed a bug report with the appropriate team at Apple. The workaround makes some tests slower, so I’m leaving this bug open so we remember to remove the workaround once we can do so.