Wallaby stop reporting test results on IDE after couple of runs randomly (until restart)

dzsodzso63 commented 4 years ago

Issue description or question

Wallaby stop reporting test results on IDE, No results change in VSCode, no visible color annotation, text, and the Wallaby.js Tests output shows an outdated output The only place I can see the test has run, is the Wallaby.js Console output (After changing a test file)

[Info]  2020-07-04T08:42:10.698Z workers [9gfie] Run 1 test(s), skipped 1 test(s)
[Info]  2020-07-04T08:42:10.699Z workers [9gfie] Sandbox is responsive, closing it
[Info]  Finished executing 1 affected test(s) in 1.02 second(s)
[Info]  2020-07-04T08:42:10.699Z project Test run finished
[Info]  2020-07-04T08:42:10.699Z project Processed console.log entries
[Info]  2020-07-04T08:42:10.699Z project Processed loading sequences
[Info]  2020-07-04T08:42:10.699Z project Processed executed tests
[Info]  2020-07-04T08:42:10.718Z project Processed code coverage
[Info]  2020-07-04T08:42:10.720Z project Processed code trace
[Info]  2020-07-04T08:42:10.742Z project Test run result processed and sent to IDE

If I run Wallaby.js: Start command, it works again

Wallaby diagnostics report

{ editorVersion: '1.46.0',
  pluginVersion: '1.0.212',
  editorType: 'VSCode',
  osVersion: 'darwin 19.3.0',
  nodeVersion: 'v10.21.0',
  coreVersion: '1.0.920',
  config:
   { files:
      [ { pattern: 'img/**/*.+(css|less|scss|sass|html|json|svg|png)', load: false, ignore: false, trigger: true, instrument: true, order: 1 },
        { pattern: 'packages/**/*.+(css|less|scss|sass|html|json|svg|png)', load: false, ignore: false, trigger: true, instrument: true, order: 2 },
        { pattern: 'packages/**/*.js', load: false, ignore: false, trigger: true, instrument: true, order: 3 },
        { pattern: 'config/**/*.js', load: false, ignore: false, trigger: true, instrument: true, order: 4 },
        { pattern: 'packages/**/*.test.js', ignore: true, trigger: true, load: true, instrument: true } ],
     tests: [ { pattern: 'packages/**/*.test.js', load: false, ignore: false, trigger: true, test: true, order: 5 } ],
     env:
      { type: 'browser',
        kind: 'chrome',
        params: { runner: '--headless --disable-gpu --deterministic-fetch --window-size=1000,1200' },
        viewportSize: { width: 800, height: 600 },
        options: { width: 800, height: 600 },
        bundle: true },
     testFramework: { version: 'jasmine@3.5.0', configurator: 'jasmine@2.1.3', reporter: 'jasmine@2.1.3', starter: 'jasmine@2.1.3' },
     compilers: { '**/*.js': [Function], '**/*.?(lit)coffee?(.md)': [Function] },
     screenshot: true,
     trace: true,
     diagnostics: {},
     filesWithNoCoverageCalculated: [],
     runAllTestsInAffectedTestFile: false,
     updateNoMoreThanOneSnapshotPerTestFileRun: false,
     preprocessors: {},
     maxConsoleMessagesPerTest: 100,
     autoConsoleLog: true,
     delays: { run: 0, edit: 100, update: 0 },
     workers: { initial: 0, regular: 0, recycle: false },
     teardown: undefined,
     hints:
      { ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore/',
        ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
        commentAutoLog: '?',
        testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' } },
     automaticTestFileSelection: true,
     runSelectedTestsOnly: false,
     extensions: {},
     reportUnhandledPromises: false,
     throwOnBeforeUnload: true,
     slowTestThreshold: 75,
     lowCoverageThreshold: 80,
     loose: undefined,
     configCode:
      'const wallabyWebpack = require(\'wallaby-webpack\');\nprocess.env.BABEL_ENV = \'test\';\nprocess.env.NODE_ENV = \'test\';\nconst webpackConfig = require(\'./config/webpack.config\');\n\n// removing babel-loader, we will use babel compiler instead, it\'s more performant\nwebpackConfig.module.rules = webpackConfig.module.rules.filter(\n  rule => rule.use !== \'babel-loader\'\n);\n\n// removing unnecessary plugins\nwebpackConfig.plugins = webpackConfig.plugins.filter(\n  plugin =>\n    !(\n      plugin.constructor &&\n      [\'UglifyJsPlugin\', \'CleanWebpackPlugin\', \'WebpackBarPlugin\'].indexOf(\n        plugin.constructor.name\n      ) > -1\n    )\n);\n\ndelete webpackConfig.entry;\ndelete webpackConfig.output;\ndelete webpackConfig.devtool;\n\nconst wallabyPostprocessor = wallabyWebpack(webpackConfig);\n\nmodule.exports = function (wallaby) {\n  return {\n    files: [\n      {\n        pattern: \'img/**/*.+(css|less|scss|sass|html|json|svg|png)\',\n        load: false,\n      },\n      {\n        pattern: \'packages/**/*.+(css|less|scss|sass|html|json|svg|png)\',\n        load: false,\n      },\n      { pattern: \'packages/**/*.js\', load: false },\n      { pattern: \'config/**/*.js\', load: false },\n      \'!packages/**/*.test.js\',\n    ],\n    tests: [{ pattern: \'packages/**/*.test.js\', load: false }],\n    postprocessor: wallabyPostprocessor,\n    env: {\n      type: \'browser\',\n      kind: \'chrome\',\n      params: {\n        runner:\n          \'--headless --disable-gpu --deterministic-fetch --window-size=1000,1200\',\n      },\n    },\n    bootstrap: function () {\n      window.__moduleBundler.loadTests();\n    },\n    testFramework: \'jasmine\',\n    compilers: {\n      \'**/*.js\': wallaby.compilers.babel(),\n    },\n    screenshot: true,\n    trace: true,\n  };\n};\n' },
  packageJSON:
   { dependencies:
      { '@trendmicro/react-dropdown': '^1.4.0',
        'babel-polyfill': '^6.26.0',
        classnames: '^2.2.5',
        'core-js': '^2.5.1',
        'create-react-class': '^15.6.0',
        d3: '^3.4.4',
        'd3-array': '^1.2.0',
        'd3-axis': '^1.0.8',
        'd3-format': '^1.2.0',
        'd3-scale': '^1.0.6',
        'd3-selection': '^1.1.0',
        'd3-time': '^1.1.0',
        'd3-time-format': '^2.0.5',
        'element-closest-polyfill': '^1.0.0',
        'file-loader': '^1.1.5',
        'flag-icon-css': '^2.9.0',
        'google-libphonenumber': '^2.0.19',
        i18next: '^14.0.1',
        intl: '^1.2.5',
        'is-in-viewport': '^2.4.2',
        jquery: '^2.2.4',
        'jquery-ui': '^1.10.4',
        'js-cookie': '^1.5.1',
        'karma-spec-reporter': '0.0.32',
        lazyloadxt: '^1.1.0',
        leaflet: '^1.3.1',
        'leaflet.gridlayer.googlemutant': '^0.7.0',
        'leaflet.locatecontrol': '^0.63.0',
        'leaflet.markercluster': '^1.4.1',
        'lodash.debounce': '^4.0.8',
        'lodash.filter': '^4.6.0',
        'lodash.omit': '^4.5.0',
        'lodash.pick': '^4.4.0',
        'lodash.throttle': '^4.1.1',
        moment: '^2.10.2',
        'npm-modernizr': '^2.8.3',
        'postcss-color-function': '^4.0.1',
        'prop-types': '^15.5.10',
        'query-string': '^5.0.0',
        react: '^16.13.1',
        'react-color': '^2.13.8',
        'react-custom-scrollbars': '^4.2.1',
        'react-dates': '^21.8.0',
        'react-dom': '^16.13.1',
        'react-dotdotdot': '^1.3.1',
        'react-easy-edit': '^1.9.0',
        'react-i18next': '9.0.10',
        'react-redux': '^7.2.0',
        'react-router-dom': '^5.2.0',
        'react-select': '^1.3.0',
        'react-telephone-input-simple': 'Formlabs/react-telephone-input.git#v4.0.6',
        'react-tooltip': '^3.3.0',
        redux: '^4.0.5',
        'redux-create-reducer': '^1.1.1',
        'redux-thunk': '^2.2.0',
        rekapi: '^1.6.0',
        reselect: '^3.0.1',
        skrollr: '^0.6.26',
        'skrollr-stylesheets': '^1.0.0',
        'slick-carousel': '^1.3.13',
        'spin.js': '^2.0.0',
        'storybook-react-router': '^1.0.4',
        underscore: '^1.7.0',
        webpack: '^3.11.0',
        webpackbar: '^3.1.5' },
     devDependencies:
      { '@storybook/addon-actions': '^3.2.0',
        '@storybook/react': '^3.2.3',
        '@wdio/allure-reporter': '^6.1.14',
        '@wdio/cli': '^6.1.17',
        '@wdio/dot-reporter': '^6.1.14',
        '@wdio/jasmine-framework': '^6.1.17',
        '@wdio/local-runner': '^6.1.17',
        '@wdio/spec-reporter': '^6.1.14',
        '@wdio/sync': '^6.1.14',
        acorn: '^6.4.1',
        autoprefixer: '^6.7.7',
        axios: '^0.19.2',
        'babel-core': '^6.25.0',
        'babel-eslint': '^7.2.3',
        'babel-loader': '^7.1.1',
        'babel-plugin-istanbul': '^4.1.4',
        'babel-plugin-module-resolver': '^4.0.0',
        'babel-plugin-transform-class-properties': '^6.24.1',
        'babel-plugin-transform-object-assign': '^6.22.0',
        'babel-plugin-transform-object-rest-spread': '^6.26.0',
        'babel-plugin-transform-react-remove-prop-types': '^0.4.6',
        'babel-plugin-transform-wpcalypso-async': '^0.1.1',
        'babel-preset-env': '^1.6.0',
        'babel-preset-react': '^6.24.1',
        chromedriver: '^83.0.0',
        'clean-webpack-plugin': '^0.1.16',
        commander: '^3.0.2',
        'convert-svg-to-png': '^0.5.0',
        'css-loader': '^0.28.0',
        'dynamic-public-path-webpack-plugin': '^1.0.3',
        enzyme: '^3.3.0',
        'enzyme-adapter-react-16': '^1.1.1',
        eslint: '^6.8.0',
        'eslint-config-airbnb': '^15.1.0',
        'eslint-config-airbnb-es5': '^1.2.0',
        'eslint-config-prettier': '^2.6.0',
        'eslint-plugin-import': '^2.7.0',
        'eslint-plugin-jsx-a11y': '^5.1.1',
        'eslint-plugin-prettier': '^2.3.1',
        'eslint-plugin-react': '^7.1.0',
        eventify: '^2.0.1',
        'exports-loader': '^0.6.4',
        express: '^4.16.1',
        'express-ws': '^3.0.0',
        'extract-text-webpack-plugin': '^2.1.0',
        'git-rev-sync': '^1.9.1',
        husky: '^0.14.3',
        'i18next-scanner': '^2.10.3',
        'imports-loader': '^0.7.1',
        inquirer: '^7.0.0',
        'is-docker': '^1.1.0',
        'jasmine-ajax': '^4.0.0',
        'jasmine-core': '^3.5.0',
        'jasmine-enzyme': '^3.6.1',
        'json-format': '^1.0.1',
        karma: '^5.0.5',
        'karma-chrome-launcher': '^3.1.0',
        'karma-coverage': '^1.1.1',
        'karma-firefox-launcher': '^1.0.1',
        'karma-jasmine': '^3.1.1',
        'karma-webpack': '^3.0.5',
        'lint-staged': '^4.3.0',
        napa: '^3.0.0',
        'node-sass': '^4.14.1',
        'node-sass-asset-functions': '^0.1.0',
        npx: '^10.2.2',
        'postcss-loader': '^2.0.9',
        precss: '^2.0.0',
        prettier: '^2.0.2',
        'react-test-renderer': '^16.13.1',
        'redux-mock-store': '^1.5.4',
        'sass-loader': '^6.0.6',
        'string-template': '^1.0.0',
        'style-loader': '^0.18.2',
        'text-loader': '0.0.1',
        'uglify-js': '^2.8.23',
        'uglifyjs-webpack-plugin': '^0.4.3',
        'vendor-copy': '^1.3.1',
        'wallaby-webpack': '^3.9.15',
        'wdio-chromedriver-service': '^6.0.3',
        'wdio-intercept-service': '^4.1.2',
        'webpack-bundle-analyzer': '^2.8.3' } },
  fs: { numberOfFiles: 1636 },
  debug:
   [ '2020-06-30T13:12:58.546Z project Changes did not trigger any tests\n',
     '2020-06-30T13:13:03.623Z fs File changed in editor: packages/dashboard/js/common/ui/PrinterDiagram.js\n',

...

     '2020-07-03T15:25:07.221Z postprocessor Webpack compilation started\n',
     '2020-07-03T15:25:07.523Z fs File changed in editor: packages/dashboard/js/common/ui/styles/CartridgeIcon.css\n',
     '2020-07-03T15:25:07.574Z postprocessor Webpack compilation finished\n',
     '2020-07-03T15:25:07.579Z postprocessor Emitting 6 files\n',
     '2020-07-03T15:25:07.582Z project Test run cancelled, re-queueing run data\n',
     '2020-07-03T15:25:07.582Z project Test run finished\n',
     '2020-07-03T15:25:07.582Z project Test run data re-queued\n',
     '2020-07-03T15:25:07.638Z project Running postprocessor\n',
     '2020-07-03T15:25:07.638Z testTask Test files from affected: 0, from deleted or manually requested: 0, from recently changed: 0, from loaded by: 5, from failing: 0\n',
     '2020-07-03T15:25:07.666Z postprocessor Webpack compilation started\n',
     '2020-07-03T15:25:07.752Z postprocessor Webpack compilation finished\n',
     '2020-07-03T15:25:07.753Z postprocessor Emitting 6 files\n',
     '2020-07-03T15:25:07.754Z project Postprocessor execution finished\n',
     '2020-07-03T15:25:07.754Z project Test run started; run priority: 2\n',
     '2020-07-03T15:25:07.762Z testTask Test files from affected: 0, from deleted or manually requested: 0, from recently changed: 0, from loaded by: 5, from failing: 0\n',
     '2020-07-03T15:25:07.794Z workers Starting test run, priority: 2\n',
     '2020-07-03T15:25:07.794Z workers Distributing tests between 5 workers\n',
     '2020-07-03T15:25:07.795Z workers Running tests in parallel\n',
     '2020-07-03T15:25:07.795Z chromeRunner Starting sandbox [worker #0, session #8ys5j]\n',
     ... 3629 more items ] }

ArtemGovorov commented 4 years ago

Thanks for providing the details.

The only place I can see the test has run, is the Wallaby.js Console output

With the message like above in Wallaby Console, the issue is likely happens in the editor.

Can you please:

stop Wallaby,
restart VS Code,
open VS Code Developer Tools (Developer: Toggle Developer Tools command in VS Code),
start Wallaby and make a couple of changes until it stops updating the editor,
share the output from VS Code Dev Tools Console tab.

dzsodzso63 commented 4 years ago

@ArtemGovorov Thx for the tip!

Here's the result (2 errors in the console):

1.

mainThreadExtensionService.ts:65 [[object Object]]Cannot read property 'document' of undefined
$onExtensionRuntimeError @ mainThreadExtensionService.ts:65
_doInvokeHandler @ rpcProtocol.ts:402
_invokeHandler @ rpcProtocol.ts:387
_receiveRequest @ rpcProtocol.ts:303
_receiveOneMessage @ rpcProtocol.ts:230
(anonymous) @ rpcProtocol.ts:105
fire @ event.ts:587
fire @ ipc.net.ts:453
_receiveMessage @ ipc.net.ts:733
(anonymous) @ ipc.net.ts:592
fire @ event.ts:587
acceptChunk @ ipc.net.ts:239
(anonymous) @ ipc.net.ts:200
t @ ipc.net.ts:28
emit @ events.js:203
addChunk @ _stream_readable.js:295
readableAddChunk @ _stream_readable.js:276
Readable.push @ _stream_readable.js:210
onStreamRead @ internal/stream_base_commons.js:166

2.

mainThreadExtensionService.ts:66 TypeError: Cannot read property 'document' of undefined
    at d._isTimelineViewer (/Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/dist/index.js:15)
    at d.<anonymous> (/Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/dist/index.js:15)
    at d.emit (events.js:203)
    at u.<anonymous> (/Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/dist/index.js:15)
    at u.emit (events.js:203)
    at u._documentUpdates (/Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/wallaby/client.js:14)
    at WebSocket.<anonymous> (/Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/wallaby/client.js:14)
    at WebSocket.emit (events.js:203)
    at Receiver.ontext (/Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/wallaby/node_modules/ws/lib/WebSocket.js:841)
    at /Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/wallaby/node_modules/ws/lib/Receiver.js:536
    at /Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/wallaby/node_modules/ws/lib/Receiver.js:368
    at /Users/dzso/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.213/wallaby/node_modules/ws/lib/PerMessageDeflate.js:249
    at afterWrite (_stream_writable.js:484)
    at onwrite (_stream_writable.js:475)
    at InflateRaw.afterTransform (_stream_transform.js:96)
    at Zlib.processCallback (zlib.js:571)

The moment it throws these, it stops working until restart.

(I am not 100% sure but it happens usually when I click on a preview tab where there's no code)

I hope this helps

dzsodzso63 commented 4 years ago

@ArtemGovorov I played with it, and now I can reproduce 100%:

Open a split tab w/ a test case opened
Open Wallaby.js: Show Last Screenshot
Modify the test case, everything ok
Click on the image
Modify the test case, nothing happens

I can see the. error on the console, and wallaby ui stops working

ArtemGovorov commented 4 years ago

@dzsodzso63 Thanks for providing more details! The issue is fixed and the fix is published in the latest Wallaby VS Code extension v1.0.214.

wallabyjs / public

Wallaby stop reporting test results on IDE after couple of runs randomly (until restart) #2471

Issue description or question

Wallaby diagnostics report