Percy not capturing any snapshots (fails to find healthcheck)

stijnbernards commented 5 years ago

Hi

I'm trying to integrate Percy and Cypress for one of my projects. Sadly Percy doesn't seem to take any snapshots even though I'm calling cy.percySnapshot multiple times.

This is my current test:

describe('Magento tests', () => {
  beforeEach(() => {
    cy.visit(Cypress.env('staging-url'))
  })

  it('Should allow a product to be ordered', () => {
    cy.percySnapshot('Homepage')

    // Forcing due to sticky headers messing up the click area
    cy.get('nav.navigation > ul > li:first-of-type a')
        .click({ force: true })

    cy.percySnapshot('Category - Boxsprings')

    cy.get('.products.list.items.product-items')
        .find('li:first-of-type a.product-item-link')
        .click()

    cy.percySnapshot('Product page')
  })
})

This is my Cypress result:

UPDATE: Seems like percy is unable to capture screenshots on https websites. Is this correct? This is quite a showstopper for the current project I'm integrating it for.

Robdel12 commented 5 years ago

Hi there!

Percy can capture HTTPS websites. This seems a lot like there's CSP or CORS issues on the page blocking us from POSTing the DOM back to our node process. Can you try disabling Cypress web security and give it another shot?

stijnbernards commented 5 years ago

Hi, thank you for your response.

This does indeed seem to fix it. Would be nice if we're able to get error logging about this issue. I did see the part in the Percy Cypress docs about the websecurity. But I did not receive any error logging at all :(

GordonTester commented 5 years ago

I'm seeing an issue with Percy not taking screenshots within my Cypress project. I've tried to point the tests to both a https and non https site and each time the tests pass in Cypress but fail within the Percy Dash as it was unable to find any screenshots.

When I run via chrome (and the cy GUI opens locally) I can see that the percy agent healthcheck is failing for some reason...

Any advice?

I am running my test using a script, as such:

"cy:run:percy": "export TEST_SUITE=foo; PERCY_TOKEN=xxx PERCY_BRANCH=test percy exec -- cypress run",

I can see it running in the terminal, bookended by percy calls

[percy] percy has started / ended etc....

mapsandapps commented 5 years ago

Hi @GordonTester, I had the same issue and downgrading to version 1.0.2 fixed it for me.

Robdel12 commented 5 years ago

Hey all! We had an issue with the 1.0.3 release but it should be fixed with 1.0.4. Can you give it a try and let us know? https://github.com/percy/percy-cypress/releases/tag/v1.0.4

mapsandapps commented 5 years ago

@Robdel12 I did try 1.0.4, and I had the same issue as with 1.0.3.

Robdel12 commented 5 years ago

Can you share a little more about your setup? I'd be curious if there's any error logs or anything we can learn from what's going on there.

ericz82 commented 5 years ago

1.0.4 works for me. Looks like the healthcheck path fix did the trick!

mapsandapps commented 5 years ago

I think that perhaps it is because we are using webpack, and require.resolve is returning a number instead of a path, similar to this issue.

Robdel12 commented 5 years ago

I believe this is safe to close now. If anyone sees any more issues with the healthcheck failing please comment and we'll reopen 👍

amrwahdan3 commented 5 years ago

hello i am getting this issue. i dont get any snapshots and i get this error: 'Percy agent is not running. Skipping snapshots'

im running 3.3.2, my baseUrl in cypress.json is a https if that matters and i have the "chromeWebSecurity": false. my tests are pretty basic:

describe('desktop home page tests', function () {

    beforeEach('SetUp', function () {
        cy.visit('/?flagoverride=flag1::true,flag2::false')
    })

    context('Desktop_Homepage_LayoutCorrect', function () {
        it('The layout is correct.', function () {
            cy.percySnapshot('Desktop Homepage')
        })
    })
})

this is also is the log: [33m»[39m Warning: [33mhealth-check[39m is not a percy command.↵Did you mean [94mexec[39m? [y/n]:

it happens in chrome and electron browser.

thanks

Robdel12 commented 5 years ago

Hey @amrwahdan3! It sounds like you're running an old version of @percy/agent with a new version of the Cypress SDK. Could you do an npm ls @percy/agent and post what the results of that are? v0.2.0 is when the healthcheck command was implemented: https://github.com/percy/percy-agent/blob/master/CHANGELOG.md#020-2019-04-03

Edit: I looked at one of your builds and can confirm that you're running @percy/agent 0.1.15

amrwahdan3 commented 5 years ago

thanks @Robdel12 ! that did the trick i see snapshots being taken and uploaded now after updating to 0.8.3

james-hughes-ntt commented 4 years ago

I'm seeing the same issue that @amrwahdan3 was seeing.

[percy] stopping percy...
[percy] waiting for 0 snapshots to complete...
[percy] done.

I have the following setup:

@percy/agent

npm ls @percy/agent
cy-visual-tests@1.0.0 /Users/hughej/Projects/Personal/JavaScript/Cypress/cy-visual-tests
└─┬ @percy/cypress@2.3.1
└── @percy/agent@0.28.1

package.json dependencies

"dependencies": {
"@percy/cypress": "^2.3.1",
"cypress": "^4.9.0",
"cypress-image-snapshot": "^3.1.1",
"prettier": "^2.0.5"
}

cypress.json

{
"video": false,
"defaultTimeoutCommand": 5000,
"pageLoadTimeout": 10000,
"waitForAnimations": true,
"chromeWebSecurity": false
}

Any help would be greatly appreciated. Thanks!

amrwahdan3 commented 4 years ago

@james-hughes-ntt i updated @percy/agent to the newest version like @Robdel12 suggested and that ended up working.

james-hughes-ntt commented 4 years ago

Thanks. So after running npm ls @percy/agent it shows that I'm using @percy/agent@0.28.1 which appears to be the newest version. Not sure what else to try 🤷‍♂️

Robdel12 commented 4 years ago

I don't believe this is a health check issue (unless you're seeing that specific log in Cypress command log). What test is calling cy.percySnapshot? Are there any logs around that?

james-hughes-ntt commented 4 years ago

@Robdel12 I'm running percy exec -- cypress run Here's the log from running that:

percy exec -- cypress run

[percy] created build #18 ...
[percy] percy has started.

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    4.9.0                                                                              │
  │ Browser:    Electron 80 (headless)                                                             │
  │ Specs:      1 found (percy.spec.js)                                                            │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  percy.spec.js                                                                   (1 of 1)

  Visual Regression Testing with Percy and Cypress
    ✓ should take a percy snapshot (1295ms)

  1 passing (4s)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        1                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      0                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  0                                                                                │
  │ Video:        false                                                                            │
  │ Duration:     3 seconds                                                                        │
  │ Spec Ran:     percy.spec.js                                                                    │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

====================================================================================================

  (Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✔  percy.spec.js                            00:03        1        1        -        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✔  All specs passed!                        00:03        1        1        -        -        -  

[percy] stopping percy...
[percy] waiting for 0 snapshots to complete...
[percy] done.

Robdel12 commented 4 years ago

Just to be 100% sure, what is the code for that test?

james-hughes-ntt commented 4 years ago

Yeah good shout @Robdel12 . Pretty standard test.

describe('Visual Regression Testing with Percy and Cypress', () => {
    it('should take a percy snapshot', () => {
        cy.visit('https://www.example.com');
        cy.wait(1000);
        cy.percySnapshot;
    });
});

Robdel12 commented 4 years ago

Hey @james-hughes-ntt, thanks for that! Looks like you're missing the parens to invoke the function:

describe('Visual Regression Testing with Percy and Cypress', () => {
    it('should take a percy snapshot', () => {
        cy.visit('https://www.example.com');
        cy.wait(1000);
        cy.percySnapshot();
    });
});

james-hughes-ntt commented 4 years ago

@Robdel12 ah thanks mate, yeah that was it! So duh of me. I was looking at all the places except for the most obvoius - where percy is actually invoked! Thanks again appreciate the help

Robdel12 commented 4 years ago

Ahhh, no worries! This happens to me all the time when doing support too. I get too far off in the weeds and it turns out it was something obvious :p

Happy it was something easy rather than something like the health check failing :D

Uraz04 commented 3 years ago

Hey there. Seems like I'm having the same issue as @james-hughes-ntt and I've checked the parens etc within the code but all seems fine (to me that is)..the below is the output and I can see something about the certificate but not sure if that's the issue. Also, disabled websecurity for chrome in the cypress.json file.

cypress_snapshot@1.0.0 cy:percy percy exec -- cypress run

[percy] RequestError Error: self signed certificate in certificate chain | Wed Feb 03 2021 20:58:22 GMT+0000 (Greenwich Mean Time) [percy] stopping percy... [percy] waiting for 0 snapshots to complete... [percy] done. [percy] percy has started.

====================================================================================================

(Run Starting)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Cypress: 6.3.0 │ │ Browser: Electron 87 (headless) │ │ Specs: 3 found (percy.spec.js, percy2.spec.js, Snapshot/image.spec.js) │ └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

Running: percy.spec.js (1 of 3)

Visual Regression test with Percy and Cypress ✓ should take percy snapshot (4559ms)

1 passing (6s)

(Results)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Tests: 1 │ │ Passing: 1 │ │ Failing: 0 │ │ Pending: 0 │ │ Skipped: 0 │ │ Screenshots: 0 │ │ Video: false │ │ Duration: 6 seconds │ │ Spec Ran: percy.spec.js │ └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

Running: percy2.spec.js (2 of 3)

Visual Regression test with Percy and Cypress ✓ should take percy snapshot (3277ms)

1 passing (4s)

(Results)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Tests: 1 │ │ Passing: 1 │ │ Failing: 0 │ │ Pending: 0 │ │ Skipped: 0 │ │ Screenshots: 0 │ │ Video: false │ │ Duration: 4 seconds │ │ Spec Ran: percy2.spec.js │ └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

Running: Snapshot/image.spec.js (3 of 3)

0 passing (0ms)

(Results)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Tests: 0 │ │ Passing: 0 │ │ Failing: 0 │ │ Pending: 0 │ │ Skipped: 0 │ │ Screenshots: 0 │ │ Video: false │ │ Duration: 0 seconds │ │ Spec Ran: Snapshot/image.spec.js │ └────────────────────────────────────────────────────────────────────────────────────────────────┘

====================================================================================================

(Run Finished)

   Spec                                              Tests  Passing  Failing  Pending  Skipped

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ✔ percy.spec.js 00:06 1 1 - - - │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✔ percy2.spec.js 00:04 1 1 - - - │ ├────────────────────────────────────────────────────────────────────────────────────────────────┤ │ ✔ Snapshot/image.spec.js 2ms - - - - - │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ✔ All specs passed! 00:10 2 2 - - -

[percy] stopping percy... [percy] waiting for 0 snapshots to complete... [percy] done.

Robdel12 commented 3 years ago

Hey @Uraz04, it looks like you're running on a network with a self signed cert (this won't work with Percy)

[percy] RequestError Error: self signed certificate in certificate chain