autoconfig does not respect jest maxworkers setting

[wknd]

Issue description or question

Wallaby.js does not take jest maxworkers into account which can cause it to use an unacceptable amount of ram and CPU load. This is especially bad on machines with a lot of CPU cores.

It is made worse by jest having some sort of memory leak which may or may not finally be resolved when using node 21.

proof of concept

I made a repo with the minimum to reproduce the problem, that will run 240000 simple identical tests to illustrate the problem. https://github.com/wknd/jest-wallaby-poc

Run it with node 18 to reproduce the problem.

https://github.com/wknd/jest-wallaby-poc

setup

npm ci

baseline test:

npm run test

problematic test: in vscode do a Wallaby.js: Start

Keep an eye on your CPU load during the tests.

expected result

about 20% of your cpu cores are used while running the test with wallaby. ram usage goes up.

actual result

100% of your cpu cores are used while running the test with wallaby. ram usage goes up even more than when running it jest directly. (5GB+ on my home machine with 16 threads)

when is this a problem

Since every test runner can in some cases take up multiple gigabytes of ram, this results in wallaby using up all available ram on a machine (and getting killed) on computers with many cores.

In my case I have 24 cores available on my work machine and it was encountered in an angular monorepo where running tests on the main app through wallaby would eventually bring ram usage to over the 64GB I have available.
Obviously that is a more extreme situation, but one that's way easier to fix if I can use wallaby with maxworkers.

Wallaby diagnostics report

{
  editorVersion: '1.88.0',
  pluginVersion: '1.0.374',
  editorType: 'VSCode',
  osVersion: 'linux 6.6.25-1-MANJARO',
  nodeVersion: 'v18.19.1',
  coreVersion: '1.0.1553',
  checksum: 'ZWJjNzUwZDZkMjhkMjgzMDI1YWY4ZDVjMzE0NzFlMWIsMTcxMzU3MTIwMDAwMCww',
  config: {
    diagnostics: {
      jest: {
        config: {
          configs: [
            {
              automock: false,
              cache: true,
              cacheDirectory: '/tmp/jest_rs',
              clearMocks: true,
              collectCoverageFrom: [],
              coverageDirectory: '<homeDir>/Code/jest-wallaby-poc/coverage',
              coveragePathIgnorePatterns: [ '/node_modules/' ],
              cwd: '<homeDir>/Code/jest-wallaby-poc',
              dependencyExtractor: undefined,
              detectLeaks: false,
              detectOpenHandles: false,
              displayName: undefined,
              errorOnDeprecated: false,
              extensionsToTreatAsEsm: [],
              fakeTimers: { enableGlobally: false },
              filter: undefined,
              forceCoverageMatch: [],
              globalSetup: undefined,
              globalTeardown: undefined,
              globals: {},
              haste: { computeSha1: false, enableSymlinks: false, forceNodeFilesystemAPI: true, throwOnModuleCollision: false },
              id: '339da7074bed51dab5c251bb7b41ac7e',
              injectGlobals: true,
              moduleDirectories: [ 'node_modules' ],
              moduleFileExtensions: [
                'js',   'mjs',
                'cjs',  'jsx',
                'ts',   'tsx',
                'json', 'node'
              ],
              moduleNameMapper: [],
              modulePathIgnorePatterns: [],
              modulePaths: undefined,
              openHandlesTimeout: 1000,
              prettierPath: 'prettier',
              resetMocks: false,
              resetModules: false,
              resolver: undefined,
              restoreMocks: false,
              rootDir: '<homeDir>/Code/jest-wallaby-poc',
              roots: [ '<homeDir>/Code/jest-wallaby-poc' ],
              runner: '<homeDir>/Code/jest-wallaby-poc/node_modules/jest-runner/build/index.js',
              runtime: undefined,
              sandboxInjectedGlobals: [],
              setupFiles: [],
              setupFilesAfterEnv: [],
              skipFilter: false,
              skipNodeResolution: undefined,
              slowTestThreshold: 5,
              snapshotFormat: { escapeString: false, printBasicPrototype: false },
              snapshotResolver: undefined,
              snapshotSerializers: [],
              testEnvironment: '<homeDir>/Code/jest-wallaby-poc/node_modules/jest-environment-node/build/index.js',
              testEnvironmentOptions: {},
              testLocationInResults: false,
              testMatch: [ '**/__tests__/**/*.[jt]s?(x)', '**/?(*.)+(spec|test).[tj]s?(x)' ],
              testPathIgnorePatterns: [ '/node_modules/' ],
              testRegex: [],
              testRunner: '<homeDir>/Code/jest-wallaby-poc/node_modules/jest-circus/runner.js',
              transform: [ [ '^.+\\.tsx?$', '<homeDir>/Code/jest-wallaby-poc/node_modules/ts-jest/dist/index.js', {} ] ],
              transformIgnorePatterns: [ '/node_modules/', '\\.pnp\\.[^\\/]+$' ],
              unmockedModulePathPatterns: undefined,
              watchPathIgnorePatterns: []
            }
          ],
          globalConfig: {
            bail: 0,
            changedFilesWithAncestor: false,
            changedSince: undefined,
            ci: false,
            collectCoverage: true,
            collectCoverageFrom: [],
            coverageDirectory: '<homeDir>/Code/jest-wallaby-poc/coverage',
            coverageProvider: 'v8',
            coverageReporters: [ 'json', 'text', 'lcov', 'clover' ],
            coverageThreshold: undefined,
            detectLeaks: false,
            detectOpenHandles: false,
            errorOnDeprecated: false,
            expand: false,
            filter: undefined,
            findRelatedTests: false,
            forceExit: false,
            globalSetup: undefined,
            globalTeardown: undefined,
            json: false,
            lastCommit: false,
            listTests: false,
            logHeapUsage: false,
            maxConcurrency: 1,
            maxWorkers: 3,
            noSCM: undefined,
            noStackTrace: false,
            nonFlagArgs: undefined,
            notify: false,
            notifyMode: 'failure-change',
            onlyChanged: false,
            onlyFailures: false,
            openHandlesTimeout: 1000,
            outputFile: undefined,
            passWithNoTests: false,
            projects: [],
            randomize: undefined,
            replname: undefined,
            reporters: undefined,
            rootDir: '<homeDir>/Code/jest-wallaby-poc',
            runInBand: undefined,
            runTestsByPath: false,
            seed: 87325212,
            shard: undefined,
            showSeed: undefined,
            silent: undefined,
            skipFilter: false,
            snapshotFormat: { escapeString: false, printBasicPrototype: false },
            testFailureExitCode: 1,
            testNamePattern: undefined,
            testPathPattern: '',
            testResultsProcessor: undefined,
            testSequencer: '<homeDir>/Code/jest-wallaby-poc/node_modules/@jest/test-sequencer/build/index.js',
            testTimeout: undefined,
            updateSnapshot: 'new',
            useStderr: false,
            verbose: undefined,
            watch: false,
            watchAll: false,
            watchPlugins: undefined,
            watchman: true,
            workerIdleMemoryLimit: undefined,
            workerThreads: false
          },
          hasDeprecationWarnings: false,
          wallaby: {
            roots: [],
            watchPathIgnorePatterns: [ '/node_modules/', '\\./dist/|\\./build/|\\./coverage/|\\./git/|/\\..+/', '/tmp/jest_rs', '\\./coverage' ],
            testPathIgnorePatterns: [ '/node_modules/', '\\./dist/|\\./build/|\\./coverage/|\\./git/|/\\..+/', '/tmp/jest_rs', '\\./coverage' ],
            testMatch: [ '**/__tests__/**/*.[jt]s?(x)', '**/?(*.)+(spec|test).[tj]s?(x)' ],
            testRegex: []
          }
        }
      }
    },
    testFramework: { version: 'jest@24.8.0', configurator: 'jest@24.8.0', reporter: 'jest@24.8.0', starter: 'jest@24.8.0', autoDetected: true },
    filesWithCoverageCalculated: [],
    filesWithNoCoverageCalculated: [],
    globalSetup: false,
    dot: true,
    files: [
      { pattern: '/node_modules/', regexp: /\/node_modules\//, ignore: true, trigger: true, load: true, file: true, test: true },
      {
        pattern: '\\./dist/|\\./build/|\\./coverage/|\\./git/|/\\..+/',
        regexp: /\.\/dist\/|\.\/build\/|\.\/coverage\/|\.\/git\/|\/\..+\//,
        ignore: true,
        trigger: true,
        load: true,
        file: true,
        test: true
      },
      { pattern: '/tmp/jest_rs', regexp: /\/tmp\/jest_rs/, ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '\\./coverage', regexp: /\.\/coverage/, ignore: true, trigger: true, load: true, file: true, test: true },
      { pattern: '**/**', ignore: false, trigger: true, load: true, order: 1 },
      { pattern: '**/__tests__/**/*.[jt]s?(x)', ignore: true, trigger: true, load: true, file: true },
      { pattern: '**/?(*.)+(spec|test).[tj]s?(x)', ignore: true, trigger: true, load: true, file: true }
    ],
    captureConsoleLog: true,
    tests: [
      { pattern: '/node_modules/', regexp: /\/node_modules\//, ignore: true, trigger: true, load: true, test: true, file: false },
      {
        pattern: '\\./dist/|\\./build/|\\./coverage/|\\./git/|/\\..+/',
        regexp: /\.\/dist\/|\.\/build\/|\.\/coverage\/|\.\/git\/|\/\..+\//,
        ignore: true,
        trigger: true,
        load: true,
        test: true,
        file: false
      },
      { pattern: '/tmp/jest_rs', regexp: /\/tmp\/jest_rs/, ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '\\./coverage', regexp: /\.\/coverage/, ignore: true, trigger: true, load: true, test: true, file: false },
      { pattern: '**/__tests__/**/*.[jt]s?(x)', ignore: false, trigger: true, load: true, test: true, order: 2 },
      { pattern: '**/?(*.)+(spec|test).[tj]s?(x)', ignore: false, trigger: true, load: true, test: true, order: 3 }
    ],
    runAllTestsInAffectedTestFile: false,
    updateNoMoreThanOneSnapshotPerTestFileRun: false,
    compilers: {},
    logLimits: { inline: { depth: 5, elements: 5000 }, values: { default: { stringLength: 8192 }, autoExpand: { elements: 5000, stringLength: 8192, depth: 10 } } },
    preprocessors: {},
    maxConsoleMessagesPerTest: 100,
    autoConsoleLog: true,
    delays: { run: 0, edit: 100, update: 0 },
    workers: { initial: 0, regular: 0, recycle: false },
    teardown: undefined,
    hints: {
      ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore|c8 ignore/',
      ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
      commentAutoLog: '?',
      testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' }
    },
    automaticTestFileSelection: true,
    runSelectedTestsOnly: false,
    mapConsoleMessagesStackTrace: false,
    extensions: {},
    env: { type: 'node', params: {}, runner: '<homeDir>/.nvm/versions/node/v18.19.1/bin/node', viewportSize: { width: 800, height: 600 }, options: { width: 800, height: 600 }, bundle: true },
    reportUnhandledPromises: true,
    slowTestThreshold: 75,
    lowCoverageThreshold: 80,
    runAllTestsWhenNoAffectedTests: false,
    configCode: 'auto.detect#1359811502'
  },
  packageJSON: { dependencies: undefined, devDependencies: { '@types/jest': '^29.5.12', jest: '^29.7.0', 'ts-jest': '^29.1.2', 'ts-node': '^10.9.2', typescript: '^5.4.4' } },
  fs: { numberOfFiles: 31 },
  debug: [
    '2024-04-07T15:15:27.854Z workers [bzmp8] Test executed: should verify useless task: 44 * 77 = 3388\n',
    '2024-04-07T15:15:27.854Z workers [5tol2] Test executed: should verify useless task: 50 * 8 = 400\n',
    '2024-04-07T15:15:27.854Z workers [5tol2] Test executed: should verify useless task: 50 * 9 = 450\n',
    '2024-04-07T15:15:27.854Z workers [r5ve9] Test executed: should verify useless task: 46 * 81 = 3726\n',
    '2024-04-07T15:15:27.854Z workers [r5ve9] Test executed: should verify useless task: 46 * 82 = 3772\n',
    '2024-04-07T15:15:27.854Z workers [p1j6c] Test executed: should verify useless task: 38 * 53 = 2014\n',
    '2024-04-07T15:15:27.854Z workers [p1j6c] Test executed: should verify useless task: 38 * 54 = 2052\n',
    '... and much much more''
  ]
}

[smcenlly]

Wallaby's worker mechanism is a little different to jest and so our automatic configuration does not (and cannot) apply the jest setting you are using.

Jest internally has two default maxWorkers for running your tests, a watch mode default and a non-watch mode default. Wallaby combines these two concepts with initial and regular settings so that you get the best of both worlds, a fast startup time (Wallaby is usually faster to do a full test run than jest) and an incremental run, which assigns less worker processes to keep your system responsive.

You will need to explicitly configure the workers setting in Wallaby. You can do this in a configuration file, or in your package.json file. Please refer to our workers docs here and here for how to create a configuration file / specify in your package.json.

Please note, you may also use the workers.restart property to restart processes for each run; I think this will fix the memory problem.