Open AleksanderBodurri opened 4 years ago
@bahmutov do you know what might be causing this?
Hard to say, I don't see why it fails...
I'm having the same issue, do you have any solution?
Same here, it also happens with Azure Devops
> $(yarn bin)/cypress run
# leads to error...
command: 'npm run cy:run',
@AleksanderBodurri Since you mention yarn
but the logs use npm
did you try adding quotes to the command like in https://github.com/bahmutov/start-server-and-test/pull/256/files? I got warnings locally about the npm binary being off until I added the quotes and it switched to yarn...maybe I should just switch back to npm...
Hi all, i'm experiencing similar issues: In my case i'm running Storybook from a yarn workspace and all works fine locally but In Circle CI it doesn't seem to be observing the "wait"
I can see if i watch the task in Circle CI that "Run Cypress tests" is actually started before Storybook finishes, and depending on how long it takes for Storybook to start up sometimes means Cypress errors:
I'm not sure what this means or why it happens: HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
Any help would be appreciated!
Thanks
"storybook": "yarn workspace docs storybook",
"cy:run": "cypress run",
"cy:pipeline": "start-server-and-test 'yarn storybook' http://localhost:6006 'yarn cy:run'"
version: 2.1
orbs:
node: circleci/node@3.0.1
cypress: cypress-io/cypress@1.16.1
jobs:
run-build:
executor:
name: node/default
steps:
- checkout
- run: yarn install
- run: yarn build
run-unit-tests:
executor:
name: node/default
steps:
- checkout
- run: yarn install
- run: yarn test -u
workflows:
all-the-things:
jobs:
- run-build
- run-unit-tests
- cypress/install:
yarn: true
- cypress/run:
yarn: true
build: yarn build
requires:
- cypress/install
start: yarn cy:pipeline
yarn run v1.22.4
$ ********************* 'yarn storybook' http://localhost:6006 'yarn cy:run'
********************* parsing CLI arguments: [ 'yarn storybook', 'http://localhost:6006', 'yarn cy:run' ] +0ms
********************* parsed args: { services: [ { start: 'yarn storybook', url: [Array] } ], test: 'yarn cy:run' } +6ms
1: starting server using command "yarn storybook"
and when url "http://localhost:6006" is responding with HTTP status code 200
running tests using command "yarn cy:run"
********************* single service "yarn storybook" to run and test +0ms
********************* starting server with command "yarn storybook", verbose mode? true +1ms
********************* starting waitOn http://localhost:6006 +6ms
********************* wait-on options { resources: [ 'http://localhost:6006' ], interval: 2000, window: 1000, timeout: 300000, verbose: true, strictSSL: true, log: true } +0ms
waiting for 1 resources: http://localhost:6006
making HTTP(S) head request to url:http://localhost:6006 ...
HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
$ yarn workspace docs storybook
$ start-storybook --docs -p 6006 -s ./public --ci
info @storybook/react v6.0.21
info
making HTTP(S) head request to url:http://localhost:6006 ...
info => Loading static files from: /root/project/docs/public .
HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
info => Loading presets
info => Loading presets
info => Loading config/preview file in "./.storybook".
info => Loading config/preview file in "./.storybook".
info => Adding stories defined in ".storybook/main.js".
info => Loading custom manager config.
info => Using default Webpack setup.
0% compiling
...
100%webpack built 446c4c3db0b389426589 in 9591ms
╭────────────────────────────────────────────────────╮
│ │
│ Storybook 6.0.21 started │
│ 8.7 s for manager and 11 s for preview │
│ │
│ Local: http://localhost:6006/ │
│ On your network: http://192.168.240.3:6006/ │
│ │
╰────────────────────────────────────────────────────╯
making HTTP(S) head request to url:http://localhost:6006 ...
HTTP(S) result for http://localhost:6006: { status: 200,
statusText: 'OK',
headers:
{ 'x-powered-by': 'Express',
'access-control-allow-origin': '*',
'access-control-allow-headers': 'Origin, X-Requested-With, Content-Type, Accept',
'accept-ranges': 'bytes',
'content-type': 'text/html; charset=UTF-8',
'content-length': '4466',
etag: 'W/"1172-v0SOHIZiw7G/ApxDDlPnHVfUjxs"',
date: 'Fri, 11 Sep 2020 21:04:45 GMT',
connection: 'close' },
data: '' }
wait-on(565) complete
********************* waitOn finished successfully +14s
********************* running test script command: yarn cy:run +1ms
$ cypress run
Cypress could not verify that this server is running:
> http://localhost:6006
We are verifying this server because it has been configured as your `baseUrl`.
Cypress automatically waits until your server is accessible before running tests.
We will try connecting to it 3 more times...
We will try connecting to it 2 more times...
We will try connecting to it 1 more time...
Cypress failed to verify that your server is running.
Please start this server and then run Cypress again.
Exited with code exit status 1
CircleCI received exit code 1
@PaulieScanlon
Try prefixing your server with http-get://
instead of http://
"cy:pipeline": "start-server-and-test 'yarn storybook' http-get://localhost:6006 'yarn cy:run'"
This was not the issue I was having but hopefully this helps.
I believe my issue is caused by flaky tests that for some reason do not display as actual failures when run with start-server-and-test
.
I never had any tests fail locally until I started trying to run cypress multiple times in parallel. When running 3 cypress processes with headless chrome in parallel, some tests ended up being extremely flaky, usually failing in at least one of those runs.
I'm not 100% sure that the failing local tests are related to the flaky CI builds but it does seem odd. I'm still trying to figure out why those particular tests flake locally when cypress is run multiple times in parallel (they seem to never fail when only running cypress once).
@AleksanderBodurri Thanks so much for the response. I did see the http-get
in the README and have run a few tests with it. I've asked in the Storybook Discord channel if anyone knows if Storybook uses webpack-dev-server
but so far the tests seem to be just as flakey using http-get
as they do without.
In both cases when the tests fail i see the same error message from wait-on
HTTP(S) error for http://localhost:6006 Error: connect ECONNREFUSED 127.0.0.1:6006
When i watch the Circle CI tasks both the Start
task and Run Cypress tests
task start at the same time which would suggest the wait-on
part isn't quite working as expected.
I only have the one .spec
at the moment and i'm not running multiple processes... just trying to work out where this issue is. Is it with start-server-and-test
, Storybook, CircleCI or Cypress
I've opened an issue on the Storybook repo to see if that sheds any light on the matter.
Here's a short screen capture. The tests do pass but it's only because Storybook just about finishes loading before the last time Cypress attempts to run the test: start-sever-and-test-v1
Thanks again!
Not sure if this will help anyone but for my particular issue i had to understand a few key things. I'll start with my package.json
"storybook": "yarn workspace docs storybook",
"cy:test": "start-server-and-test 'yarn storybook' http-get://localhost:6006 'yarn cypress open'"
"devDependencies": {
"cypress": "^5.1.0",
"start-server-and-test": "^1.11.3"
}
When i run cy:test
locally from my terminal everything does and has always worked fine. start-server-and-test
does indeed wait for Storybook to start on 6006
before it runs cypress open
The problem was this wasn't observed when run through Circle CI
My Circle config.yml
now looks like this:
- cypress/install:
name: cypress-install
yarn: true
- cypress/run:
name: start-storybook-and-test
yarn: true
requires:
- cypress-install
start: yarn storybook
wait-on: 'http://localhost:6006'
By adding wait-on
to config.yml
i'm able to correctly wait for Storybook to boot up and as the docs explain you can wait-for-server-to-respond and have the Cypress orb kick off the cypress run
command without having to add anything additional to your package.json
or config.yml
because that's kinda what - cypress/run
does for you.
So i guess the key takeaways here are there are the two different methods for using this.
start-sever-and-test
circleci/node@4.0.0
orb does this for me.Boogy time!
I'm not sure if I should open a new issue or tag onto this one but I've been having the this issue for a while now. Locally my tests run stable and never fail. On CircleCI it fails 100% of the time.
My CircleCI config
version: 2.1
jobs:
# The install job for all dependencies so we can reuse/depend on it in other jobs
install_dependencies:
docker:
- image: circleci/node:lts
steps:
- checkout
- restore_cache:
keys:
- v10-dependencies-{{ checksum "yarn.lock" }}{{ checksum "server/yarn.lock" }}
- v10-dependencies-
- run:
name: Install dependencies for app
command: yarn
- run:
name: Install dependencies for server
command: cd server && yarn
- run:
name: Adding debugging deps
command: sudo apt-get install tree # for debugging circle builds
- run:
name: Report file structure
command: tree -Ca
- run:
name: Node version
command: node -v
- run:
name: npm version
command: npm -v
- run:
name: Yarn version
command: yarn --version
- save_cache:
paths:
- .yarn
- .cache
key: v10-dependencies-{{ checksum "yarn.lock" }}{{ checksum "server/yarn.lock" }}
- persist_to_workspace:
root: ~/
paths:
- .yarn
- .cache
# Running all tests
testing:
docker:
- image: circleci/node:lts
steps:
- checkout
- attach_workspace:
at: ~/
- run:
name: Installing dependencies from cache
command: yarn
- run:
name: Installind dependencies for server from cache
command: cd server && yarn
- run:
name: Installing Cypress dependencies
command: sudo apt-get install xvfb libgtk-3-dev libnotify-dev libgconf-2-4 libnss3 libxss1 libasound2
- run:
name: Testing our code
command: yarn test
- store_artifacts:
path: src/tests/utils/videos
- store_artifacts:
path: src/tests/utils/screenshots
workflows:
version: 2
running_jobs:
jobs:
- install_dependencies # First we install dependencies
- testing: # Second we test our code
requires:
- install_dependencies
The CI will fail when running the tasks
1: starting server using command "npm run test:e2e:server"
and when url "[ 'http://localhost:3000' ]" is responding with HTTP status code 200
running tests using command "npm run test:e2e:run"
See full run here: https://app.circleci.com/pipelines/github/dominikwilkowski/bronzies/129/workflows/24aa0111-6b86-4ac7-a607-0e5357528844/jobs/261
It will crash (?) the server via
Error: server closed unexpectedly
at ChildProcess.onClose (/home/circleci/project/node_modules/start-server-and-test/src/index.js:69:14)
at ChildProcess.emit (events.js:314:20)
at maybeClose (internal/child_process.js:1021:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)
I tried the different orbs and other things.
@dominikwilkowski I know your comment was long ago, but just for future developers reading this thread, I had a very similar issue and, in my case, upcon closer inspection I found and error ENOSPC: System limit for number of file watchers reached
.
Then I successfully fixed the issue following the advice from this stack overflow answer
Sorry to revive after so long. I'm pretty sure I have a good idea of why this issue was occurring, at least in my case. I was writing tests that did not take full advantage of cypress' retry-ability mechanism. I was writing tests that looked like this:
cy.get('.foo').find('.bar').contains('baz')
Heres the scenario:
A view exists like this
<div class="foo">
<div class="bar">
qux
</div>
</div>
A cypress command causes some event that triggers the view to update. After the update, the view should look like
<div class="foo">
<div class="bar">
baz
</div>
</div>
Note that the divs after the update are not the same DOM elements as the divs before the update.
Cypress then runs cy.get('.foo').find('.bar').contains('baz')
. If the page view doesn't change quickly enough (as is sometimes the case in CI containers where system resources are not as high as our local devices), the cy.get('.foo').find('.bar')
part of the query matches against the elements from step 1. Since these elements will never contain 'baz'
, even if the contains('baz')
is retried after the view updates, cy.get('.foo').find('.bar').contains('baz')
will fail.
For some reason the details of the test failures were being silenced. Not sure exactly why that was happening, but I haven't seen the same flakiness since I refactored my tests to avoid the scenario above.
The biggest thing that helped me fix this issue was a snippet from the Cypress docs here https://docs.cypress.io/guides/core-concepts/retry-ability#Merging-queries
Specifically
Use cy.contains Tip: instead of cy.get(selector).should('contain', text) or cy.get(selector).contains(text) chain, we recommend using cy.contains(selector, text) which is retried automatically as a single command.
Documentation is once again undefeated 😄
Goto https://github.com/bahmutov/start-server-and-test/issues/250#issuecomment-1079451745 to see how I resolved this
Running Cypress will intermittently fail in CircleCI. I haven't been able to identify a pattern that causes it, my guess is it's some kind of race condition.
Running
yarn cy:ci
locally works perfectly every time.in package.json:
in .circleci/config.yml:
circle logs:
complete log:
It seems that everything works fine until the
cy:run
script runs. I have no idea why it fails randomly. Any ideas?