ianlancetaylor
commented
1 year ago CC @bcmills @matloob
Open howardjohn opened 1 year ago
ianlancetaylor
commented
1 year ago CC @bcmills @matloob
rsc
commented
1 year ago To clarify, you are looking for a way to run each package test one at a time? What about the tests inside a given package that are using t.Parallel? Do you want to disable that parallelism too?
howardjohn
commented
1 year ago To clarify, you are looking for a way to run each package test one at a time?
Yes, only one package should execute at a time during go test -somenewflag ./.... Similar to go test -p 1 ./..., but without the side effect of compilation also being serialized.
What about the tests inside a given package that are using t.Parallel? Do you want to disable that parallelism too?
I think keeping inner-package parallelism is fine and could be controlled with the existing -test.parallel
bcmills
commented
1 year ago Put synchronization into the test binary itself. This is challenging and ineffective. Things like mutex cannot be used as it is cross-binary, so we need an external locking system.
Compare #33974, which would add a lockedfile.Mutex that could be used by such tests.
JeremyLoy
commented
2 months ago At work, we have a mix of NodeJS and Golang services. Those services communicated with a database, and as such have tests that exercise queries against a Dockerized database. As that is a shared resource, tests that hit the database must be serialized, while tests that operate purely in memory are allowed to be parallelized.
The NodeJS services have a pretty easy way of solving this using a custom Jest runner, which I'll link below. Our way of doing this in Golang is by putting all of the tests that use an external resource (like a database) in a singular package so that they run sequentially. This causes the test files to be located not alongside the source files though, which is a shame.
In the NodeJS way, the stream of tests that must be sequential is defined by filename. That isn't very robust and is error prone.
I think there should be a way to do this in Go using a testing.T method, something like t.Mutex("aKeyToIdentifyResource"). That way each test is self contained, and the package/file/folder organization isn't impacted by which tests need exclusive access to a particular resource. Multiple distinct resources can exist as well.
import { Config } from "@jest/types";
import TestRunner, { Test, TestEvents, TestRunnerContext, TestWatcher, UnsubscribeFn } from "jest-runner";
export default class CustomTestRunner {
public supportsEventEmitters = true;
private parallelRunner: ParallelTestRunner;
private sequentialRunner: SequentialTestRunner;
constructor(globalConfig: Config.GlobalConfig, context: TestRunnerContext) {
this.parallelRunner = new ParallelTestRunner(globalConfig, context);
this.sequentialRunner = new SequentialTestRunner({ ...globalConfig, maxWorkers: 1 }, context);
}
on<Name extends keyof TestEvents>(
eventName: Name,
listener: (eventData: TestEvents[Name]) => void | Promise<void>,
): UnsubscribeFn {
const parallelUnsubscribe = this.parallelRunner.on(eventName, listener);
const sequentialUnsubscribe = this.sequentialRunner.on(eventName, listener);
return () => {
parallelUnsubscribe();
sequentialUnsubscribe();
};
}
async runTests(tests: Test[], watcher: TestWatcher): Promise<void> {
await Promise.all([
this.sequentialRunner.runTests(tests, watcher),
this.parallelRunner.runTests(tests, watcher),
]);
}
}
class ParallelTestRunner extends TestRunner {
async runTests(tests: Test[], watcher: TestWatcher): Promise<void> {
const isParallelizeable = (test: Test) => !test.path.includes(".sequential.");
const parallelizeableTests = tests.filter(isParallelizeable);
await super.runTests(parallelizeableTests, watcher, { serial: false });
}
}
class SequentialTestRunner extends TestRunner {
async runTests(tests: Test[], watcher: TestWatcher): Promise<void> {
const isSequentialTest = (test: Test) => test.path.includes(".sequential.");
const sequential = tests.filter(isSequentialTest);
// { serial: false } triggers the parallel test execution leveraging workers,
// but along with { maxWorkers: 1, workerIdleMemoryLimit: ... } configuration
// this sole worker would be killed and restarted if it exceeds the memory threshold.
await super.runTests(sequential, watcher, { serial: false });
}
}
matloob
commented
2 months ago cc @samthanawalla
I think we should think harder about if there's a way to better support having the individual tests synchronize access to the external resource. I wonder if there's a better way to act on tests being blocked.
It is a common pattern to want to run Go tests in sequence rather than in parallel. Within a package, this is controlled by
t.Parallel()and-test.parallel. However, across packages there is a different control:-p N(typically-p 1for this use case).This is commonly used for integration tests which are depending on exclusive access to some resource outside of the test.
However, this comes at a huge cost:
-p 1does not control just test parallelism, but all compiler actions. This means all compiling and linking is done sequentially. In our project, this has caused a 10x increase in test execution times - with-p 1, it takes 8m25s; without, 50s.I would like a way to run test packages sequentially, while still benefiting from the parallelism in building.
Ecosystem usage
I did a rough analysis of usage of
go test -p 1across Github, and found a number of large projects doing the same. I use stars just as a rough measure of the project scope.If you search "go run tests sequentually", basically all sources will tell you to use
-p 1. This includes AI chatbots, popular stack overflow answers, blogs, and even some Go core maintainer recommendations.Alternatives
go test ./a && go test ./b). This is not great; we lose parallelism in building.go test -c -o tests/ ./...; go test -p 1 ./...). This is demonstrated as effective here. However, it is highly limited unless https://github.com/golang/go/issues/61199 is resolved as well.