mfrachet / cypress-audit

⚑ Run Lighthouse and Pa11y audits directly in your E2E test suites
https://mfrachet.github.io/cypress-audit/
MIT License
353 stars 44 forks source link

cy.task('lighthouse') failed with the following error: > connect ECONNREFUSED 127.0.0.1:61216 #62

Closed Lukaszli closed 3 years ago

Lukaszli commented 3 years ago

Hi, I did all the steps from readme and I am using plain JS but still I get the error like:

cy.task('lighthouse') failed with the following error:

> connect ECONNREFUSED 127.0.0.1:61216

chrome_Qr7t4JPgGn

my package.json:

    "cypress": "^6.3.0",
    "cypress-audit": "^0.3.0",
    "cypress-file-upload": "^4.1.1",
    "cypress-sql-server": "^1.0.0",

Maybe I am missing some necessary step.

I have

const { lighthouse, pa11y, prepareAudit } = require('cypress-audit');

and

on('before:browser:launch', (browser = {}, launchOptions) => {
    prepareAudit(launchOptions);
  });

and

 lighthouse: lighthouse(), // calling the function is important
    pa11y: pa11y(), // calling the function is important

in my plugins/index.js

And I have imported audit in support/commands:

import 'cypress-audit/commands';
mfrachet commented 3 years ago

I'll try to check, maybe the browser is not on the expected port 😊

DaveGold commented 3 years ago

We have this in our nx project as well if I log the launchOptions I see this:

'--proxy-server=http://localhost:7108', '--disable-web-security', '--allow-running-insecure-content', '--proxy-bypass-list=<-loopback>', '--remote-debugging-port=7114', '--remote-debugging-address=127.0.0.1', '--remote-debugging-pipe'

So my impression is that this has to do with some kind of remote-debugging-port But for chrome I have a fixed one on windows "C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222"

Can someone explain why we get ECONNREFUSED 127.0.0.1:7708 => why is this port random? and has it somehow to do with chrome remote debugging?

mfrachet commented 3 years ago

In this library, I'm trying to check on which browser cypress is connecting, in order to avoid opening a new one that may lose some internal states (like an authenticated user). Maybe the way I'm doing it doesn't work for "manually configured" systems and I will need to find a way to solve that problem 😊

DaveGold commented 3 years ago

I have found something.. i am running cypress with Chrome.. I changed to Electron 87 and the test worked image

I did not.. it skipped the test

DaveGold commented 3 years ago

In this library, I'm trying to check on which browser cypress is connecting, in order to avoid opening a new one that may lose some internal states (like an authenticated user). Maybe the way I'm doing it doesn't work for "manually configured" systems and I will need to find a way to solve that problem 😊

I am not sure how manually my Angular nx project is configured.. browser running default on localhost:4200. It is in the resolved Cypress config: baseUrl: 'http://localhost:4200/',

mfrachet commented 3 years ago

I don't really know why, in your situation, it doesn't work as expected. But in the library, I try to resolve the chrome instance port by browsing in a cypress underlying configuration objects and attempt to resolve the --remote-debugging-port option in this file: https://github.com/mfrachet/cypress-audit/blob/master/index.js#L7

I think this enters in conflict with the NX config you mentioned πŸ€”.

sleepingmonk commented 3 years ago

Forgive my ignorance if I'm really missing something, but why not (also?) allow the test to run on a desired url or use the config baseUrl? Can it only run on localhost IP?

it("should pass the audits", function () {
    cy.visit('my-test-url');
    cy.lighthouse();
    cy.pa11y();
  });

And what about headless? I'm trying to use this after deploying to test environments with CI.

This is not a JS project. I just want to test pages, regardless of how they're built.

I didn't see anything in the docs to help me solve this. If that's not what this package should do, I apologize for misunderstanding.

vrknetha commented 3 years ago

I am getting the same error

image

My package.json

"cypress": "^6.3.0", "cypress-audit": "^0.3.0",

edgewords commented 3 years ago

I am getting the same with Chrome Version 88.0.4324.104 (Official Build) (64-bit) on Windows 10. "cypress": "^6.3.0", "cypress-audit": "^0.3.0"

Simple test: describe('Performance Tests', function () { it('should check performance - standard options', function () { cy.visit(''); //the url i point at is hosted on the internet, not on my local machine cy.lighthouse(); //Performance cy.pa11y(); //Accessibility }); });

image

When I try electron, it says the browser is not supported.

mfrachet commented 3 years ago

Can somebody create a project on the side to replicate that error? I understand y'all in trouble, but it's hard for me to just figure things out without a reproducible project 😬

Also, Lighthouse is supposed to run on chrome only. I guess it's normal that it doesn't work on electron.

@sleepingmonk , you need to pass the full URL in this case. Cypress basePath is not managed for now. Feel free to create another issue and describe what you exactly expect so that I can take this into consideration and challenge your ideas :)

Thank you for your help

edgewords commented 3 years ago

@mfrachet Hi Marvin, I have created a very simple demo that shows the problem I am having. https://github.com/edgewords/CypressAuditErr I'm using Windows 10 & Chrome 88. I run the single spec in /integration/performance with the Test Runner or CLI and get the error I sceenshotted above. Thanks, Tom

vrknetha commented 3 years ago

Also, it happens to be once i upgrade to Cypress: 6.3.0. Here is my repo

https://github.com/vrknetha/cypress-lighthouse

mfrachet commented 3 years ago

I'm wondering: are y'all using chrome 88 and running the tests on an external URL? Can you try in headless mode and tell me if the test pass?

Lukaszli commented 3 years ago

Yep I tried in headless and the test passed when it should not - it expected all results to be 100%

edgewords commented 3 years ago

I just ran in chrome headless & the test worked for me. Failed as expected as results were below 100% I'm using chrome 88 & external URL, so headless OK for me, GUI chrome get the connection timed out error.

edgewords commented 3 years ago

Although headless does work, I get this message at the start of the test when it is first connecting:

Timed out waiting for the browser to connect. Retrying... Warning: Cypress failed to connect to Chrome via stdio after 1 minute, 0 seconds. Falling back to TCP... Connecting to Chrome via TCP was successful, continuing with tests.

The test then runs & I get the metrics back, but at the end of the test I also get:

Warning: Cypress failed to connect to Chrome via stdio after 1 minute, 0 seconds. Falling back to TCP... The browser never connected. Something is wrong. The tests cannot run. Aborting...

Don't know if that is useful or not...

mfrachet commented 3 years ago

Thank you for your feedbacks.

I've updated Cypress on 6.4 and it looks working now. Here's the changelog: https://github.com/cypress-io/cypress/releases and here's the potential bugfix:

Reverted a change to how the Chrome DevTools Protocol is established. Instead of using stdio and then falling back to TCP, Cypress now only uses TCP to try to connect to Chrome DevTools Protocol. Addresses #14819.

Can you confirm that it works for you when using Cypress 6.4?

DaveGold commented 3 years ago

With cypress 6.4 it is working. Although in my case an extra tab was opened and closed while doing the the task. Is this normal?

mfrachet commented 3 years ago

It is 😊 . it's the tab opened by lighthouse

edgewords commented 3 years ago

All working for me as well with cypress 6.4.0 in headless & normal modes. Thanks Marvin, this is a really great plugin, I love that I can now do my GUI, API, AND Performance testing all in one tool. Nice one :)

mfrachet commented 3 years ago

Happy we managed to find something to fix this πŸ˜… .

I'm closing this issue, feel free to reopen or comment if necessary πŸ‘πŸ»

Thanks for your help 😊

swathisham commented 3 years ago

seeing the same issue again with cypress 8.5 and chrome 94 image @mfrachet

christianmartinnies commented 2 years ago

Same issue over here until i switched over to Edge. Hope, that may help someone out there… :)

addicus commented 2 years ago

I'm also having this issue on cypress 9.5.0 chrome 99

rgangal commented 2 years ago

I am getting error "Unsupported browser. Use Chrome version v87.0 or above ". Chrome version I am using is 99 and cypress version : "9.5.1"

TroyonGuillaume commented 2 years ago

I'm also having this issue on cypress 9.5.0 chrome 99

rarestomos commented 1 year ago

@mfrachet do you have or do you plan any update on the plugin? Doesn't seem to work anymore with the most recent Cypress versions.

guicamillo commented 1 year ago

I was facing the same issue and I was able to circumvent it by downgrading node from 18.x to 16.x.

Surely, it's not a long term fix, but it's a bandaid that unblocked me, at least for now.