cypress e2e testing does not exit on error in development server

[jdoubleu]

Version

3.4.0

Environment info

Environment Info:

  System:
    OS: macOS
  Binaries:
    Node: 11.6.0 - /usr/local/bin/node
    Yarn: Not Found
    npm: 6.7.0 - /usr/local/bin/npm
  npmPackages:
    @ascendancyy/vue-cli-plugin-stylelint: ^1.1.0 => 1.1.0 
    @fortawesome/vue-fontawesome: ^0.1.3 => 0.1.5 
    @vue/babel-helper-vue-jsx-merge-props:  1.0.0-beta.2 
    @vue/babel-plugin-transform-vue-jsx:  1.0.0-beta.2 
    @vue/babel-preset-app:  3.3.0 
    @vue/babel-preset-jsx:  1.0.0-beta.2 
    @vue/babel-sugar-functional-vue:  1.0.0-beta.2 
    @vue/babel-sugar-inject-h:  1.0.0-beta.2 
    @vue/babel-sugar-v-model:  1.0.0-beta.2 
    @vue/babel-sugar-v-on:  1.0.0-beta.2 
    @vue/cli-overlay:  3.3.0 
    @vue/cli-plugin-babel: ^3.2.0 => 3.3.0 
    @vue/cli-plugin-e2e-cypress: ^3.2.0 => 3.3.0 
    @vue/cli-plugin-eslint: ^3.2.1 => 3.3.0 
    @vue/cli-plugin-unit-mocha: ^3.2.0 => 3.3.0 
    @vue/cli-service: ^3.2.2 => 3.3.0 
    @vue/cli-shared-utils:  3.3.0 
    @vue/component-compiler-utils:  2.5.0 
    @vue/eslint-config-standard: ^4.0.0 => 4.0.0 
    @vue/preload-webpack-plugin:  1.1.0 
    @vue/test-utils: ^1.0.0-beta.27 => 1.0.0-beta.28 
    @vue/web-component-wrapper:  1.2.0 
    babel-helper-vue-jsx-merge-props:  2.0.3 
    babel-plugin-transform-vue-jsx:  4.0.1 
    eslint-plugin-vue: ^5.0.0 => 5.1.0 
    vue: ^2.5.21 => 2.5.22 
    vue-cli-plugin-element: ^1.0.0 => 1.0.1 
    vue-cli-webpack:  1.0.0 
    vue-eslint-parser:  2.0.3 
    vue-hot-reload-api:  2.3.1 
    vue-i18n: ^8.5.0 => 8.7.0 
    vue-loader:  15.5.1 
    vue-router: ^3.0.2 => 3.0.2 
    vue-style-loader:  4.1.2 
    vue-template-compiler: ^2.5.21 => 2.5.22 
    vue-template-es2015-compiler:  1.8.1 
    vue-test-chai: ^1.0.0-3 => 1.0.0-3 
    vuex: ^3.0.1 => 3.1.0 
  npmGlobalPackages:
    @vue/cli: 3.4.0

Steps to reproduce

1. Have cli-plugin-e2e-cypress installed in your project and configure a CI to run tests
2. Produce an error in one of your components which would break the build
3. Run e2e tests

What is expected?

When end-to-end tests are run inside a CI it should exit with an error code when there was an error building the app.

This way the CI fails and does not time out.

What is actually happening?

The WebpackDevServer shows errors, Cypress never starts and the CI might fail after a given timeout.

---

The cli-plugin-e2e-cypress waits for the serve command to finish before starting up cypress.

If there's an error in the build, the serve command never finishes, as the WebpackDevServer waits for changes to rebuild the app. However this will never happen in a CI and it might time out.

[Richard-Choooou]

I have this problem too.

[MatthiasGrandl]

Related, but shouldn't the process also exit, when all specs ran successfully? It just keeps running otherwise which isn't ideal. I think this should happen if the --headless parameter is passed.

[LinusBorg]

If that's your experience please open a separate issue. I haven't noticed anything like that

[larionov]

@LinusBorg We have this problem too. Maybe vue-cli-service serve shouldn't run in watch mode when run on CI? Or having an option to run vue-cli-service build && npx serve dist instead would make more sense?

Can I work on this?

As a workaround we run vue-cli-service build prior to test runs to prevent builds hanging, but that increases the build time obviously.

[Seraf]

Hello, we also have this problem sometimes on our CI testing the PR which hangs the build. Is there some work or someone working on this?

Thanks

[LinusBorg]

@Seraf I wanted to but didn't find the time. Go ahead if you want to dig into it.

[Seraf]

I'm sorry, I would really love to contribute but honestly, it's far beyond my skills :-( I think that a good way to solve it would be to add an option to the serve command to exit on failure.

Maybe I'm misreading but the serve command isn't exiting the process, right? Then adding an option to exit on an error and calling here with the parameter to exit would do the trick. Am I right?

[luxaritas]

This is also an issue with nightwatch - if there's an error, it just hangs (instead of exists) and doesn't run tests.