test process bootstrap/async setup hook

[heckj]

Description

In doing integration tests for some server-side swift (and related client) libraries, some of the libraries I'm using require a single bootstrapping sequence per process (swift-log, swift-distributed-tracing). Today there's no place (XCTest, or swift-testing) that I can arrange that.

Expected behavior

I'd really like to see a hook in swift-testing where I could stand up related infrastructure for my tests, before they're executed. Ideally as an async throws kind of thing where an exception would terminate the test suite with a relevant error before any tests were invoked (assuming there was some unresolvable failure in that setup hook)

Actual behavior

At the moment, I'm hacking around it and invoking a per-test setup sequence (in XCTest) that embeds the pieces in a shared, global actor and makes only the first of the calls to it's bootstrapping sequence do anything, the rest are quietly no-op.

Steps to reproduce

No response

swift-testing version/commit hash

No response

Swift & OS version (output of swift --version && uname -a)

No response

[grynspan]

Can you elaborate a bit about what you mean by "hook" here? Presumably not the typical usage of the term where a callback can be set for some functionality, since there'd be no opportunity for you to set the callback.

If you're looking to run code in the process before your main function executes, that's perhaps something that the Swift language itself would need to provide functionality for. If you're using the binary compiled by SwiftPM, there's not really any room for extra code to execute earlier than main().

If you're building your own binary and synthesizing your own main() function, you can do what SwiftPM does, but be aware that __swiftPMEntryPoint() is not guaranteed to have a stable signature over time.

Is this perhaps a duplicate of #36?

[heckj]

This probably could be a duplicate of #36 - a very similar concept, but making assumptions about a test runner and the number of suites to be run.

In my use case I want a setup mechanism that I can use that's only called once per process. Somewhere in the sequence when the test runner was standing up was my initial thought, but that concept is predicated on the assumption all tests were run/invoked in the same process.

It would be different from #36 only in the case where there was more than a single test suite in the flow.

I'm uncertain what an API for this would look like - maybe an async delegate-style call from the test runner? I haven't dug to see what the replacements are for TestRunner in this repo yet - the process that gets created and run when you invoke 'swift test' was what I was thinking would make the most sense.

[grynspan]

Is it necessary in this scenario for the logic to run before main(), or just that it run once and run early?

[heckj]

Just run once, and early (before the test runner invokes any test in the same process)

[grynspan]

I think in the general case that can be provided with a static or global let:

// at top level of file
private let setupWork: Void = {
  // do work here
}()

func ensureSetupIsCompleted() {
  setupWork
}

Or if it needs to be async or throws:

private let setupTask: Task<Void, SetupFailedError> = Task {
  // do work here
}

func ensureSetupIsCompleted() async throws(SetupFailedError) {
  try await setupTask.value
}()

And then tests or suites that need this work to be completed before they run can invoke that helper function:

@Test func f() {
  ensureSetupIsCompleted()
}

@Test func g() {
  ensureSetupIsCompleted()
}

@Suite struct S {
  init() {
    ensureSetupIsCompleted()
  }

  ...
}

A benefit here of not making it a test-target-wide operation is that only tests that actually depend on the relevant state/setup need to opt in, while tests that aren't affected don't have to pay for the overhead when run in isolation (e.g. with swift test --filter testThatDoesNotNeedSetup.)

I'm tempted to suggest that this is better-suited as a language-wide feature rather than a test-only feature given how close the above is to the platonic ideal (minus the boilerplate.) Or put another way: Swift doesn't have the concept of static constructors, global initializers, etc. but perhaps it should have such a concept.

[heckj]

That is pretty much exactly what I'm doing today, slightly different format though - spinning up what I need in a global (shared) actor to be able to access the bits I spun up. If an initialization hook from the test-runner isn't an option, that's entirely what I'll do - it's what I'm already doing with existing XCTests, I just hoped to remove/extract the lines verifying that everything is set up from each test into to a single location, if that was an option.

I hadn't mapped the concept back to the breadth of the overall swift language, but I can see your point. I don't feel like I understand enough of the ins and outs to elucidate that idea in a meaningful way to propose it for the swift language overall though.

[grynspan]

@kubamracek Does this sort of thing (a global run-once function akin to a static initializer) dovetail at all with your @section work?

[mman]

Just dropping here possibly related discussion: https://forums.swift.org/t/using-swift-log-with-swift-testing-how-to-bootstrap-logging-system/75235/4

There is currently no easy way to call LoggingSystem.bootstrap when a swift project contains multiple testTargets. Calling it once from each testTarget clashes because all testTargets are run in parallel in the same process.