Random tests fail, always date related

[jkyoutsey]

Different, random tests fail each time I run all tests. Every time, those tests have some sort of date logic in them. This happens on both windows and mac systems and happens for different developers on the team.

For Example: One run this one fails: ​​​​ProductExpirationDisplayComponent Should show expired message Next run it's a different test: ​​​​QASessionComponent show/hide percentage or counts percentage in shortMode complete hold in past​​​​

The diagnostics report is far too big for me to include on GitHub comments. I'll be happy to email it.

{
  editorVersion: '1.63.2',
  pluginVersion: '1.0.322',
  editorType: 'VSCode',
  osVersion: 'darwin 21.1.0',
  nodeVersion: 'v14.15.1',
  coreVersion: '1.0.1206',
  checksum: 'MWM5MWM3NjY3ZWY1ZTQ5ZDNiNGFjYjljMDFlZWNlOWQsMTY2MjE2MzIwMDAwMCww',
  config: {
    diagnostics: {
      angular: {
        workspace: {
          '$schema': './node_modules/@angular/cli/lib/config/schema.json',
          version: 1,
          newProjectRoot: 'projects',
          projects: {
            mms: {
              root: '',
              sourceRoot: 'src',
              projectType: 'application',
              prefix: 'mms',
              schematics: { '@schematics/angular:component': { style: 'scss', changeDetection: 'OnPush' }, '@ngrx/schematics:component': { style: 'scss', changeDetection: 'OnPush' } },
              architect: {
                build: {
                  builder: '@angular-devkit/build-angular:browser',
                  options: {
                    outputPath: 'dist/mms',
                    index: 'src/index.html',
                    main: 'src/main.ts',
                    polyfills: 'src/polyfills.ts',
                    tsConfig: 'src/tsconfig.app.json',
                    assets: [ 'src/favicon.ico', 'src/assets', 'src/web.config', 'src/manifest.json', 'src/browser.html' ],
                    styles: [ 'src/styles.scss', 'src/assets/scss/dragula/dragula.css', 'src/assets/libs/hopscotch/css/hopscotch.css', 'src/assets/scss/site.scss' ],
                    scripts: [ 'src/assets/libs/hopscotch/js/hopscotch.js', 'src/assets/libs/bitmovin-loader.js' ],
                    allowedCommonJsDependencies: [ 'lodash', 'ng2-dragula', 'subsink', 'hammerjs/hammer', 'file-saver', 'dragula' ],
                    vendorChunk: true,
                    extractLicenses: false,
                    buildOptimizer: false,
                    sourceMap: true,
                    optimization: false,
                    namedChunks: true
                  },
                  configurations: {
                    devdeploy: {
                      budgets: [ { type: 'anyComponentStyle', maximumWarning: '6kb' } ],
                      fileReplacements: [ { replace: 'src/environments/environment.ts', with: 'src/environments/DO_NOT_IMPORT.ENVIRONMENT.DEVDEPLOY.ts' } ],
                      outputHashing: 'all',
                      sourceMap: false,
                      namedChunks: false,
                      extractLicenses: true,
                      vendorChunk: false,
                      buildOptimizer: true,
                      serviceWorker: true,
                      ngswConfigPath: 'src/ngsw-config.json'
                    },
                    production: {
                      budgets: [ { type: 'anyComponentStyle', maximumWarning: '13kb' } ],
                      fileReplacements: [ { replace: 'src/environments/environment.ts', with: 'src/environments/DO_NOT_IMPORT.ENVIRONMENT.PROD.ts' } ],
                      optimization: true,
                      outputHashing: 'all',
                      sourceMap: false,
                      namedChunks: false,
                      extractLicenses: true,
                      vendorChunk: false,
                      buildOptimizer: true,
                      serviceWorker: true,
                      ngswConfigPath: 'src/ngsw-config.json'
                    },
                    staging: {
                      budgets: [ { type: 'anyComponentStyle', maximumWarning: '13kb' } ],
                      fileReplacements: [ { replace: 'src/environments/environment.ts', with: 'src/environments/DO_NOT_IMPORT.ENVIRONMENT.STAGING.ts' } ],
                      optimization: true,
                      outputHashing: 'all',
                      sourceMap: false,
                      namedChunks: false,
                      extractLicenses: true,
                      vendorChunk: false,
                      buildOptimizer: true,
                      serviceWorker: true,
                      ngswConfigPath: 'src/ngsw-config.json'
                    }
                  },
                  defaultConfiguration: ''
                },
                serve: {
                  builder: '@angular-devkit/build-angular:dev-server',
                  options: { browserTarget: 'mms:build', proxyConfig: 'proxy.conf.json' },
                  configurations: { production: { browserTarget: 'mms:build:production' }, staging: { browserTarget: 'mms:build:staging' } }
                },
                'serve-local': { builder: '@angular-devkit/build-angular:dev-server', options: { browserTarget: 'mms:build', proxyConfig: 'proxy.conf.local.json' } },
                'extract-i18n': { builder: '@angular-devkit/build-angular:extract-i18n', options: { browserTarget: 'mms:build' } },
                test: {
                  builder: '@angular-devkit/build-angular:karma',
                  options: {
                    main: 'src/test.ts',
                    polyfills: 'src/polyfills.ts',
                    tsConfig: 'src/tsconfig.spec.json',
                    karmaConfig: 'src/karma.conf.js',
                    styles: [ 'src/styles.scss' ],
                    scripts: [],
                    assets: [ 'src/favicon.ico', 'src/assets', 'src/web.config', 'src/manifest.json' ],
                    codeCoverageExclude: [ '**/*.mock.ts', '**/*.module.ts' ],
                    fileReplacements: [ { replace: 'src/environments/environment.ts', with: 'src/environments/DO_NOT_IMPORT.ENVIRONMENT.UNITTESTING.ts' } ]
                  },
                  configurations: { production: { karmaConfig: 'src/karma.conf.prod.js' } }
                },
                lint: { builder: '@angular-eslint/builder:lint', options: { lintFilePatterns: [ 'src/**/*.ts', 'src/**/*.html' ] } }
              }
            }
          },
          defaultProject: 'mms',
          schematics: { '@ngrx/schematics:component': { styleext: 'scss', changeDetection: 'OnPush' } },
          cli: { defaultCollection: '@ngrx/schematics', analytics: '7773f977-d812-420b-bf64-2152c4ebec57' }
        },
        main: '// This file is required by karma.conf.js and loads recursively all the .spec and framework files\n' +
          "import 'zone.js/testing';\n" +
          "import { getTestBed } from '@angular/core/testing';\n" +
          "import { BrowserDynamicTestingModule, platformBrowserDynamicTesting } from '@angular/platform-browser-dynamic/testing';\n" +
          '\n' +
          "// Unfortunately there's no typing for the `__karma__` variable. Just declare it as any.\n" +
          "const karma: any = window['__karma__'];\n" +
          '\n' +
          'declare const require: {\n' +
          '\tcontext(\n' +
          '\t\tpath: string,\n' +
          '\t\tdeep?: boolean,\n' +
          '\t\tfilter?: RegExp\n' +
          '\t): {\n' +
          '\t\t<T>(id: string): T;\n' +
          '\t\tkeys(): string[];\n' +
          '\t};\n' +
          '};\n' +
          '\n' +
          '// Prevent Karma from running prematurely.\n' +
          'if (karma) {\n' +
          '\tkarma.loaded = () => {};\n' +
          '}\n' +
          '\n' +
          '// First, initialize the Angular testing environment.\n' +
          'getTestBed().initTestEnvironment(BrowserDynamicTestingModule, platformBrowserDynamicTesting(), {\n' +
          '\tteardown: { destroyAfterEach: true }\n' +
          '});\n' +
          '\n' +
          '// Then we find all the tests.\n' +
          '\n' +
          '// And load the modules.\n' +
          '\n' +
          '\n' +
          '// Finally, start Karma to run the tests.\n' +
          'if (karma) {\n' +
          '\tkarma.start();\n' +
          '}\n'
      }
    },
    testFramework: { version: 'jasmine@3.10.0', configurator: 'jasmine@2.1.3', reporter: 'jasmine@2.1.3', starter: 'jasmine@2.1.3', autoDetected: true },
    env: { kind: 'chrome', type: 'browser', params: {}, viewportSize: { width: 800, height: 600 }, options: { width: 800, height: 600 }, bundle: true },
    files: [
      { pattern: 'src/**/*.spec.ts', ignore: true, trigger: true, load: true },
      { pattern: 'src/polyfills.ts', ignore: true, trigger: true, load: true },
      { pattern: 'src/test.ts', ignore: true, trigger: true, load: true },
      { pattern: 'src/main.ts', ignore: true, trigger: true, load: true },
      { pattern: 'src/index.html', ignore: true, trigger: true, load: true },
      { pattern: 'src/test.base.ts', ignore: true, trigger: true, load: true },
      { pattern: 'src/test.wallaby.ts', ignore: false, trigger: true, load: true, order: 1 },
      { pattern: 'src/**/*.+(ts|js)', load: false, ignore: false, trigger: true, order: 2 },
      { pattern: 'src/**/*.+(css|less|scss|sass|styl|html|json|svg)', instrument: false, load: false, ignore: false, trigger: true, order: 3 }
    ],
    tests: [ { pattern: 'src/**/*.spec.ts', load: false, ignore: false, trigger: true, test: true, order: 4 } ],
    filesWithNoCoverageCalculated: [],
    runAllTestsInAffectedTestFile: false,
    updateNoMoreThanOneSnapshotPerTestFileRun: false,
    addModifiedTestFileToExclusiveTestRun: true,
    compilers: {},
    preprocessors: {},
    maxConsoleMessagesPerTest: 100,
    autoConsoleLog: true,
    delays: { run: 0, edit: 100, update: 0 },
    workers: { initial: 0, regular: 0, recycle: false },
    teardown: undefined,
    hints: {
      ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore/',
      ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
      commentAutoLog: '?',
      testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' }
    },
    automaticTestFileSelection: true,
    runSelectedTestsOnly: false,
    mapConsoleMessagesStackTrace: false,
    extensions: {},
    reportUnhandledPromises: false,
    throwOnBeforeUnload: true,
    slowTestThreshold: 75,
    lowCoverageThreshold: 80,
    loose: true,
    configCode: 'auto.detect#1313788472'
  },
  packageJSON: {
    dependencies: {
      '@angular-devkit/core': '^12.2.13',
      '@angular/animations': '^12.2.13',
      '@angular/cdk': '~12.2.12',
      '@angular/common': '^12.2.13',
      '@angular/compiler': '^12.2.13',
      '@angular/core': '^12.2.13',
      '@angular/forms': '^12.2.13',
      '@angular/material': '^12.2.12',
      '@angular/material-moment-adapter': '^12.2.12',
      '@angular/platform-browser': '^12.2.13',
      '@angular/platform-browser-dynamic': '^12.2.13',
      '@angular/router': '^12.2.13',
      '@angular/service-worker': '^12.2.13',
      '@ctrl/tinycolor': '^3.4.0',
      '@ngrx/data': '^12.5.1',
      '@ngrx/effects': '^12.5.1',
      '@ngrx/entity': '^12.5.1',
      '@ngrx/store': '^12.5.1',
      '@ngrx/store-devtools': '^12.5.1',
      '@types/hammerjs': '^2.0.40',
      'blob-util': '^2.0.2',
      'configcat-js': '5.6.0',
      'core-js': '^2.6.1',
      'file-saver': '^2.0.5',
      hammerjs: '^2.0.8',
      'http-server': '^14.0.0',
      lodash: '^4.17.21',
      mocha: '^7.1.2',
      'mocha-teamcity-reporter': '^3.0.0',
      moment: '^2.29.1',
      'ng-swipe': '^1.0.5',
      'ng2-dragula': '^2.1.1',
      'ngx-color-picker': '^11.0.0',
      'ngx-cookie-service': '^13.0.0',
      'ngx-infinite-scroll': '^10.0.1',
      'reflect-metadata': '^0.1.13',
      rxjs: '^7.4.0',
      subsink: '^1.0.2',
      tslib: '^2.0.3',
      'zone.js': '~0.11.4'
    },
    devDependencies: {
      '@angular-devkit/build-angular': '~12.2.13',
      '@angular-eslint/builder': '12.6.1',
      '@angular-eslint/eslint-plugin': '12.6.1',
      '@angular-eslint/eslint-plugin-template': '12.6.1',
      '@angular-eslint/schematics': '12.6.1',
      '@angular-eslint/template-parser': '12.6.1',
      '@angular/cli': '^12.2.13',
      '@angular/compiler-cli': '^12.2.13',
      '@angular/language-service': '^12.2.13',
      '@ngrx/schematics': '^12.5.1',
      '@types/jasmine': '~3.8.0',
      '@types/jasminewd2': '~2.0.8',
      '@types/lodash': '^4.14.149',
      '@types/node': '^12.11.1',
      '@types/tinycolor2': '^1.4.3',
      '@typescript-eslint/eslint-plugin': '4.28.2',
      '@typescript-eslint/parser': '4.28.2',
      'angular2-template-loader': '^0.6.2',
      cypress: '^5.6.0',
      'cypress-file-upload': '^4.1.1',
      'cypress-log-to-output': '^1.1.2',
      'cypress-teamcity-reporter': '^3.0.0',
      eslint: '^7.26.0',
      'eslint-plugin-import': '2.25.2',
      'eslint-plugin-jsdoc': '37.0.3',
      'eslint-plugin-prefer-arrow': '1.2.3',
      'jasmine-core': '~3.8.0',
      'jasmine-marbles': '^0.9.0',
      'jasmine-reporters': '^2.3.2',
      'jasmine-spec-reporter': '~5.0.0',
      karma: '~6.3.7',
      'karma-chrome-launcher': '~3.1.0',
      'karma-cli': '~1.0.1',
      'karma-coverage': '^2.0.3',
      'karma-jasmine': '~4.0.0',
      'karma-jasmine-html-reporter': '^1.7.0',
      'karma-spec-reporter': '0.0.32',
      'karma-teamcity-reporter': '^1.1.0',
      'ng-mocks': '^12.5.0',
      'ngx-spec': '^2.1.4',
      prettier: '^2.4.1',
      'start-server-and-test': '^1.11.3',
      'ts-node': '~7.0.1',
      typescript: '~4.3.5',
      'wallaby-webpack': '^3.9.16'
    }
  },
  fs: { numberOfFiles: 2779 },
 debug: [...]
}

[ArtemGovorov]

Different, random tests fail each time I run all tests. Every time, those tests have some sort of date logic in them. This happens on both windows and mac systems and happens for different developers on the team.

It sounds like there may be some (unwanted) dependencies between some of the tests, that are revealed when running Wallaby (and because it uses multiple processes with smart test distribution among them, different tests may fails because of the shared state).

• Ideally, you may need to find that shared state/dependency that breaks the tests, but if ng test consistently works without any issues for the project, then you may configure Wallaby to use 1 worker process (similar to ng test) to make Wallaby run all tests in certain order and ignore that shared state/dependency issue. Note that in this case Wallaby will not be running your tests in parallel so this may affect the initial run time. To make Wallaby to use 1 worker process you may add the following to your package.json:

...
"wallaby": {
    "autoDetect": true,
    "workers": {
       "initial": 1,
       "regular": 1
    }
}

• What kind of test failures are you getting, why exactly are the random date related tests are failing?

[jkyoutsey]

I reviewed these tests pretty thoroughly. I made sure they were not re-using any mutated objects, etc. I made sure that state was completely reset every test. And I even set the package.json configuration you provided. And the same tests still failed in wallaby, but not in ng test.

In all cases the failures are related to testing if a date is in the future or the past.

It is across three specific files that these fail. And If I put in a space in the file and then save so they re-run, everything in that file will suddenly pass.

[jkyoutsey]

For example, here is a form validator function and test that fails intermittently only in Wallaby. There is no state. This is a pure function, and the test has no dependencies.

import { ValidatorFn, FormGroup, ValidationErrors } from '@angular/forms';

export const reportByTopicDatesValidator: ValidatorFn = (control: FormGroup): ValidationErrors | null => {
    const start = control.get('startDate');
    const end = control.get('endDate');

    // We must not clear this error!
    if (end.errors?.matDatepickerParse) {
        return null;
    }

    end.setErrors(null);

    // end.value is 00:00:00 in the current time-zone based on how the MatDatePicker control works.
    // We need the end date to be the last moment of the day.
    if (end.value) {
        end.value.setHours(23, 59, 59); // don't set millis
    }

    if (start.value && start.value > new Date()) {
        const startDateInFutureError = { startDateInFuture: true };
        start.setErrors(startDateInFutureError);
        start.markAsTouched();
        return startDateInFutureError;
    }

    if (end.value && start.value && start.value >= end.value) {
        const datesBackwardsError = { datesBackwards: true };
        end.setErrors(datesBackwardsError);
        end.markAsTouched();
        return datesBackwardsError;
    }

    return null;
};

And here is the test:

    it('validates if end date is after start date', () => {
        const form = new FormGroup({
            startDate: new FormControl(today),
            endDate: new FormControl(tomorrow)
        });
        expect(reportByTopicDatesValidator(form)).toBeNull();
        expect(form.controls.endDate.touched).toBeFalsy();
        expect(form.controls.endDate.errors).toBeNull();
    });

[ArtemGovorov]

And If I put in a space in the file and then save so they re-run, everything in that file will suddenly pass.

This is interesting. It means that if the test file runs in isolation (from other test files), then its tests are passing. Can you please try the following:

• add // file.skip comment to any 2 of the 3 test files causing the issues (to make Wallaby not run the test files at all),
• restart Wallaby and see if you can reproduce the issue.

For example, here is a form validator function and test that fails intermittently only in Wallaby. There is no state. This is a pure function, and the test has no dependencies.

Thanks for sharing the example.

• Which assertion is failing in the test, this one: expect(reportByTopicDatesValidator(form)).toBeNull();? Can you add some console.log or //? comments to the reportByTopicDatesValidator function to find out why is it failing? Specifically are the end.value and start.value logged values expected or not?
• Where are today and tomorrow variables in the test are coming from? They seem to be declared in some our scope, are they used in other tests of the file?

One possible issue that I can see: because today and tomorrow variables are possibly re-used elsewhere in the test file, and the endDate value (possibly referencing tomorrow variable) is being mutated in your source code:

end.value.setHours(23, 59, 59);

and thus after the test executes, the tomorrow variable is modified (that may cause other test failures).

[smcenlly]

@jkyoutsey - were you able to review @ArtemGovorov's suggestions in the response above?

[jkyoutsey]

@jkyoutsey - were you able to review @ArtemGovorov's suggestions in the response above?

Not yet. But it does suggest some things that make sense. I will get to it, just not sure how soon. If you want to close the issue for now that's fine though.

[smcenlly]

OK - closing the issue for now. Feel free to reply when you're ready (if you need) and we'll re-open.