wallabyjs / public

Repository for Wallaby.js questions and issues
http://wallabyjs.com
759 stars 45 forks source link

Using wallaby.js with apollo-universal-starter-kit #2368

Closed dimtim closed 4 years ago

dimtim commented 4 years ago

Issue description or question

I'm trying to use Wallaby.js in project based on Apollo Universal Starter Kit but get errors with automatic configuration. Can you please suggest configuration file that can be used with this project.

Wallaby diagnostics report

{ editorVersion: '1.43.0',
  pluginVersion: '1.0.181',
  editorType: 'VSCode',
  osVersion: 'darwin 18.7.0',
  nodeVersion: 'v10.15.0',
  coreVersion: '1.0.863',
  config:
   { diagnostics: {},
     testFramework: { version: 'jest@24.8.0', configurator: 'jest@24.8.0', reporter: 'jest@24.8.0', starter: 'jest@24.8.0', autoDetected: true },
     filesWithCoverageCalculated: [],
     micromatch: true,
     files:
      [ { pattern: '/node_modules/', regexp: /\/node_modules\//, ignore: true, trigger: true, load: true },
        { pattern: 'dist/|build/|coverage/|/\\.cache/|/\\.idea/|/\\.vscode/|/\\.git/|/\\.gitlab/',
          regexp: /dist\/|build\/|coverage\/|\/\.cache\/|\/\.idea\/|\/\.vscode\/|\/\.git\/|\/\.gitlab\//,
          ignore: true,
          trigger: true,
          load: true },
        { pattern: '**/**', ignore: false, trigger: true, load: true, order: 1 },
        { pattern: '**/__tests__/**/*.[jt]s?(x)', ignore: true, trigger: true, load: true },
        { pattern: '**/?(*.)+(spec|test).[tj]s?(x)', ignore: true, trigger: true, load: true } ],
     tests:
      [ { pattern: '/node_modules/', regexp: /\/node_modules\//, ignore: true, trigger: true, load: true, test: true },
        { pattern: 'dist/|build/|coverage/|/\\.cache/|/\\.idea/|/\\.vscode/|/\\.git/|/\\.gitlab/',
          regexp: /dist\/|build\/|coverage\/|\/\.cache\/|\/\.idea\/|\/\.vscode\/|\/\.git\/|\/\.gitlab\//,
          ignore: true,
          trigger: true,
          load: true,
          test: true },
        { pattern: '**/__tests__/**/*.[jt]s?(x)', ignore: false, trigger: true, load: true, test: true, order: 2 },
        { pattern: '**/?(*.)+(spec|test).[tj]s?(x)', ignore: false, trigger: true, load: true, test: true, order: 3 } ],
     filesWithNoCoverageCalculated: [],
     runAllTestsInAffectedTestFile: false,
     compilers: {},
     preprocessors: {},
     maxConsoleMessagesPerTest: 100,
     autoConsoleLog: true,
     delays: { run: 0, edit: 100, update: 0 },
     workers: { initial: 0, regular: 0, recycle: false },
     teardown: undefined,
     hints:
      { ignoreCoverage: '__REGEXP /ignore coverage|istanbul ignore/',
        ignoreCoverageForFile: '__REGEXP /ignore file coverage/',
        commentAutoLog: '?',
        testFileSelection: { include: '__REGEXP /file\\.only/', exclude: '__REGEXP /file\\.skip/' } },
     automaticTestFileSelection: true,
     runSelectedTestsOnly: false,
     extensions: {},
     env: { type: 'node', params: {}, runner: '/usr/local/bin/node', viewportSize: { width: 800, height: 600 }, options: { width: 800, height: 600 }, bundle: true },
     reportUnhandledPromises: true,
     slowTestThreshold: 75,
     lowCoverageThreshold: 80,
     loose: true,
     configCode: 'auto.detect#1026773938' },
  packageJSON:
   { dependencies: { inquirer: '^6.2.2', lodash: '^4.17.11', opencollective: '^1.0.3' },
     devDependencies:
      { '@alienfast/i18next-loader': '^1.0.15',
        '@babel/cli': '^7.0.0',
        '@babel/core': '^7.0.0',
        '@babel/plugin-proposal-class-properties': '^7.0.0',
        '@babel/plugin-proposal-decorators': '^7.0.0',
        '@babel/plugin-proposal-object-rest-spread': '^7.0.0',
        '@babel/plugin-syntax-dynamic-import': '^7.0.0',
        '@babel/plugin-transform-destructuring': '^7.0.0',
        '@babel/plugin-transform-for-of': '^7.0.0',
        '@babel/plugin-transform-regenerator': '^7.0.0',
        '@babel/plugin-transform-runtime': '^7.0.0',
        '@babel/preset-env': '^7.0.0',
        '@babel/preset-flow': '^7.0.0',
        '@babel/preset-react': '^7.0.0',
        '@babel/preset-typescript': '^7.0.0',
        '@babel/register': '^7.0.0',
        '@babel/runtime': '^7.0.0',
        '@loadable/babel-plugin': '^5.10.0',
        '@loadable/webpack-plugin': '^5.7.1',
        autoprefixer: '^9.5.0',
        'babel-core': '^7.0.0-0',
        'babel-eslint': '^10.0.1',
        'babel-jest': '^24.8.0',
        'babel-loader': '^8.0.6',
        'babel-plugin-dynamic-import-node': '^2.2.0',
        'babel-plugin-import': '^1.7.0',
        'babel-preset-minify': '^0.5.0-alpha.5a128fd5',
        'babel-preset-vue': '^2.0.2',
        'clean-webpack-plugin': '^1.0.1',
        'core-js': '^3.1.3',
        'css-loader': '^1.0.0',
        eslint: '^5.16.0',
        'eslint-config-airbnb': '^17.1.0',
        'eslint-plugin-import': '^2.17.2',
        'eslint-plugin-jest': '^22.5.1',
        'eslint-plugin-json': '^1.4.0',
        'eslint-plugin-jsx-a11y': '^6.2.1',
        'eslint-plugin-prettier': '^3.0.1',
        'eslint-plugin-react': '^7.12.4',
        'eslint-plugin-react-hooks': '^1.6.0',
        'file-loader': '^2.0.0',
        'fork-ts-checker-webpack-plugin': '^0.4.9',
        'hard-source-webpack-plugin': '^0.13.1',
        'html-webpack-plugin': '^3.2.0',
        'http-proxy-middleware': '^0.19.0',
        husky: '^1.1.2',
        'ignore-loader': '^0.1.2',
        'isomorphic-style-loader': '^4.0.0',
        jest: '^24.8.0',
        'jest-transform-graphql': '^2.1.0',
        'jest-transform-stub': '^2.0.0',
        lerna: '^3.4.0',
        less: '^3.0.2',
        'less-loader': '^4.1.0',
        'lint-staged': '^7.1.2',
        'mini-css-extract-plugin': '^0.4.1',
        'node-hmr-plugin': '^1.0.1',
        'node-sass': '^4.7.2',
        'npm-run-all': '^4.1.5',
        openurl: '^1.1.1',
        'postcss-loader': '^3.0.0',
        prettier: '^1.9.2',
        'qrcode-terminal': '^0.12.0',
        'raw-loader': '^0.5.1',
        'regenerator-runtime': '^0.13.2',
        'sass-loader': '^7.1.0',
        shelljs: '^0.8.1',
        'source-list-map': '^2.0.0',
        'style-loader': '^1.0.0',
        stylus: '^0.54.5',
        'stylus-loader': '^3.0.2',
        'ts-loader': '^4.5.0',
        tslint: '^5.11.0',
        'tslint-config-prettier': '^1.5.0',
        'tslint-plugin-prettier': '^2.0.0',
        typescript: '^3.4.5',
        'uglifyjs-webpack-plugin': '^1.3.0',
        'url-loader': '^1.0.1',
        'vue-loader': '^15.5.1',
        'vue-style-loader': '^4.1.2',
        'vue-template-compiler': '^2.5.22',
        'wait-on': '^3.2.0',
        webpack: '^4.33.0',
        'webpack-cli': '^3.3.3',
        'webpack-dev-middleware': '^3.1.3',
        'webpack-dev-server': '^3.1.5',
        'webpack-hot-middleware': '^2.22.3',
        'webpack-manifest-plugin': '^2.0.3',
        'webpack-merge': '^4.1.4',
        'webpack-node-externals': '^1.7.2',
        'webpack-sources': '^1.1.0',
        'webpack-virtual-modules': '^0.1.10',
        'whatwg-fetch': '^2.0.4' } },
  fs: { numberOfFiles: 1316 },
  debug:
   [ '2020-03-14T19:23:14.317Z config Detected Jest.\n',
     '2020-03-14T19:23:14.321Z config Configured Jest.\n',
     '2020-03-14T19:23:14.334Z project Wallaby Node version: v10.15.0\n',
     '2020-03-14T19:23:14.334Z project Wallaby config: <homeDir>/git/apollo-universal-starter-kit/auto.detect\n',
     '2020-03-14T19:23:21.643Z project File cache: <homeDir>/.vscode/extensions/wallabyjs.wallaby-vscode-1.0.181/projects/b712105d66b440cc\n',
     '2020-03-14T19:23:21.777Z uiService Listening port 51235\n',
     '2020-03-14T19:23:21.848Z project package.json file change detected, invalidating local cache\n',
     '2020-03-14T19:23:22.423Z workers Parallelism for initial run: 2, for regular run: 2\n',
     '2020-03-14T19:23:22.424Z workers Starting run worker instance #0\n',
     '2020-03-14T19:23:22.424Z workers Starting run worker instance #1\n',
     '2020-03-14T19:23:22.425Z workers Web server is listening at 62553\n',
     '2020-03-14T19:23:22.426Z project File cache requires some updates, waiting required files from IDE\n',
     '2020-03-14T19:23:22.903Z project Stopping process pool\n',
     '2020-03-14T19:23:22.999Z project Test run started; run priority: 3\n',
     '2020-03-14T19:23:23.003Z project Running all tests\n',
     '2020-03-14T19:23:23.162Z workers Starting test run, priority: 3\n',
     '2020-03-14T19:23:23.162Z workers Distributing tests between 2 workers\n',
     '2020-03-14T19:23:23.164Z workers Running tests in parallel\n',
     '2020-03-14T19:23:23.165Z nodeRunner Starting sandbox [worker #0, session #i80mb]\n',
     '2020-03-14T19:23:23.166Z nodeRunner Starting sandbox [worker #1, session #b5k85]\n',
     '2020-03-14T19:23:23.166Z nodeRunner Preparing sandbox [worker #0, session #i80mb]\n',
     '2020-03-14T19:23:23.166Z nodeRunner Preparing sandbox [worker #1, session #b5k85]\n',
     '2020-03-14T19:23:23.910Z workers Started run worker instance (delayed) #1\n',
     '2020-03-14T19:23:23.911Z nodeRunner Prepared sandbox [worker #1, session #b5k85]\n',
     '2020-03-14T19:23:23.911Z workers [worker #1, session #b5k85] Running tests in sandbox\n',
     '2020-03-14T19:23:24.050Z workers Started run worker instance (delayed) #0\n',
     '2020-03-14T19:23:24.051Z nodeRunner Prepared sandbox [worker #0, session #i80mb]\n',
     '2020-03-14T19:23:24.051Z workers [worker #0, session #i80mb] Running tests in sandbox\n',
     '2020-03-14T19:23:39.405Z workers Sandbox (active) [b5k85] error: undefined\n',
     '2020-03-14T19:23:39.503Z workers [b5k85] Sandbox is responsive, closing it\n',
     '2020-03-14T19:23:39.506Z project Test run finished\n',
     '2020-03-14T19:23:41.099Z workers [i80mb] Loaded unknown number of test(s)\n',
     '2020-03-14T19:23:41.100Z workers [i80mb] Test executed: should work\n',
     '2020-03-14T19:23:41.104Z workers Sandbox (active) [i80mb] error: undefined\n',
     '2020-03-14T19:23:41.153Z workers [i80mb] Sandbox is responsive, closing it\n' ] }
smcenlly commented 4 years ago

When using automatic configuration to run your project's tests, Wallaby will perform the same logic as if you were running jest from the command-line using npx jest. Right now, when you run your project's tests using npm test, lerna is actually running your tests for you on each individual project. In contrast, Wallaby runs on your entire project (e.g. entire mono-repo). The context for running your tests is therefore a little different with the current working directory being your mono-repo root with Wallaby and the current working directory being your package project root with lerna.

We took a look at the apollo-universal-starter-kit stable branch. We ran into a few problems with tests not passing from the command-line (using yarn test), but were not sure if this was a problem because we were using Windows, or there are other problems in the default stable branch that need to be fixed. We did make some progress towards getting this working but got stuck with graphql/apollo-starter specific problems that we are not familiar with solving.

Unfortunately we do not have an example configuration that we can provide to you for the apollo starter kit. To get Wallaby working with the apollo-universal-starter-kit we recommend first getting jest working from the root of the mono-repo in watch mode using npx jest --watch. You will need to create a jest.config.js file in the project root, similar to:

mono-repo root jest.config.js

module.exports = ({
  "projects": [
    "packages/*"
  ]
})

After configuring the root config, we ran into some other problems because the jest configuration for each package (e.g. packages\server\jest.config.js) is assuming that the current working directory is the package directory (e.g. packages\server) while running from the root, jest's current working directory is your mono-repo root. We needed to update your jest.config.js babel transform to fix this:

package jest.config.js transform change

  transform: {
    '\\.(gql|graphql)$': 'jest-transform-graphql',
    'locales[\\/]index.[jt]s': '<rootDir>/jest-transform-i18next',
-   '.*': 'babel-jest'
+   '.*': ['babel-jest', { root: __dirname }]
  },

After making this change, we ran into some other problems related to apollo-universal-starter-kit that we weren't exactly sure how to solve that are related to how graphql is being used by jest (please note that none of these problems were related to Wallaby itself). We then got an error running tests using npx jest from the project root. Unfortunately this is as far as we got as we simply aren't familiar enough with graphql or what the apollo starter kit is doing.

Network error: request to http://localhost/:undefined/graphql failed, reason: connect ECONNREFUSED 127.0.0.1:80

      at new ApolloError (node_modules/apollo-client/bundle.umd.js:92:26)
      at node_modules/apollo-client/bundle.umd.js:1587:34
      at node_modules/apollo-client/bundle.umd.js:2007:15
          at Set.forEach (<anonymous>)
      at node_modules/apollo-client/bundle.umd.js:2005:26
          at Map.forEach (<anonymous>)
      at QueryManager.broadcastQueries (node_modules/apollo-client/bundle.umd.js:2003:20)
      at node_modules/apollo-client/bundle.umd.js:1482:29

One more point to note (and it may have been related to our attempts to change configuration and get everything running) was that the tests seemed brittle/fragile and would sometimes fail for other reasons (network issues, port already in use, etc.). Sometimes such issues are caused by your tests/test framework expecting the process to be terminated after your tests execute while by default, Wallaby re-uses processes to decrease test times. If you run into these problems, you may configure Wallaby to use a single process, and to recycle the process between each test run. You may add Wallaby workers configuration to your package.json in your project root to fix this:

...
  "wallaby": {
    "workers": {
      "initial": 1,
      "regular": 1,
      "restart": true,
    }
  }
...

If you are able to resolve these problems using jest from the command-line then Wallaby should also work for you.

If/when you get npx jest working from the command-line, if you have problems with wallaby, please let us know and we can re-open this issue or else you may create a new issue.