search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
47.07k stars 3.19k forks source link

Cookies are broken with cy.origin #23165

Open denke8 opened 2 years ago

denke8 commented 2 years ago

Current behavior

I'm trying to test a WordPress plugin. The plugin does different things depending on if the domain is recognized or not.

For this reason I'm running the exact same test (except the last assertion) twice, once normally, and once wrapped in cy.origin. experimentalSessionAndOrigin is set to true

The test without the cy.origin works perfectly.

The one wrapped in cy.origin runs twice (test retry):

This is fairly repeatable

Desired behavior

No response

Test code to reproduce

describe('Manual plugin activation', () => {
  describe('w/ random domain', () => {
    beforeEach(() => {
      // This command brings up the appropriate docker containers, and sets WP up with a random domain,
      // which it adds to the hosts file so the browser can access it.
      // It then executes Cypress.config('baseUrl', baseUrl); to prepare for the tests
      // This seem to be working correctly, as the page does show up correctly in response to cy.visit()
      cy.resetTestingEnv({ randomBaseUrl: true })
    })

    it('should display signup CTA when autoconfig fails', () => {
      cy.origin(Cypress.config('baseUrl'), {}, () => {
        // cy.adminAuth()
        cy.visit('/wp-admin')

        // eslint-disable-next-line cypress/no-unnecessary-waiting
        cy.wait(500) // The login page clears/resets the input from js whan loaded, so need to wait

        cy.contains('Username').type(Cypress.env('CYPRESS_WP_ADMIN_USER'))
        cy.contains('Password').type(Cypress.env('CYPRESS_WP_ADMIN_PASS'))
        cy.contains('Log In').click()

        cy.contains('Dashboard').should('exist')

        // cy.activatePluginGui();
        cy.visit('/wp-admin/plugins.php')

        cy.get('#activate-plugin1').click()
        cy.contains('Thank you for installing plugin1 plugin!').should(
          'exist',
        )

        // cy.visitPluginSettings();
        cy.visit('/wp-admin/admin.php?page=plugin1')
        cy.contains('Create your account')
        .should('have.attr', 'href')
        .and(
          'include',
          'https://www.plugin1-staging.com/signup-wordpress',
        )
      })
    })
  })

  describe('w/ standard domain', () => {
    beforeEach(() => {
      // In this case it does the exact same as above with the exception that it sets a pre-defined domain
      // so it is recognized by the plugin. This domain does match the one given in cypress.config.js -> e2e.baseUrl
      cy.resetTestingEnv()
    })

    it('should display MS admin link when autoconfig succeeds', () => {
      // cy.adminAuth();
      cy.visit('/wp-admin')

      // eslint-disable-next-line cypress/no-unnecessary-waiting
      cy.wait(500) // The login page clears/resets the input from js whan loaded, so need to wait

      cy.contains('Username').type(Cypress.env('CYPRESS_WP_ADMIN_USER'))
      cy.contains('Password').type(Cypress.env('CYPRESS_WP_ADMIN_PASS'))
      cy.contains('Log In').click()

      cy.contains('Dashboard').should('exist')

      // cy.activatePluginGui();
      cy.visit('/wp-admin/plugins.php')

      cy.get('#activate-plugin1').click()
      cy.contains('Thank you for installing plugin1\'s plugin!').should(
        'exist',
      )

      // cy.visitPluginSettings();
      cy.visit('/wp-admin/admin.php?page=plugin1')
      cy.contains('Go to your plugin1 backend').should(
        'have.attr',
        'href',
        'https://admin.plugin1-staging.com/sites/145',
      )
    })
  })
})

Cypress Version (that I tested with and is broken)

10.3, 10.3.1, 10.4, 10.5, 10.6, 10.7, 10.8, 10.9, 10.11, 12.2, 12.5.1, 12.17.0, 13.2.0, 13.6.2, 13.7.2

Other

I suspect that changing Cypress.config('baseUrl'); might be a contributing factor in this issue

Not sure how to create a demo test setup of this complexity (multiple domains, etc.) so please let me know what other information or debug output to provide in order to track down the issue

AtofStryker commented 2 years ago

Hi @denke8. Thank you for opening an issue. Are you able to provide a small reproduction repository that we could run to verify behavior? My guess is the random base url is having an effect here. We typically don't recommend setting the cy.origin to your base url. Since top is already set to that url, it shouldn't be necessary. But you can have a different base url per test. Have you tried getting this to work without cy.origin?

denke8 commented 2 years ago

Hi @AtofStryker, Thanks for checking on this issue!

Are you able to provide a small reproduction repository that we could run to verify behavior?

Well, since I'm running against a WP installation, I'd need docker to be able to pull up:

My guess is the random base url is having an effect here. We typically don't recommend setting the cy.origin to your base url. Since top is already set to that url, it shouldn't be necessary. But you can have a different base url per test.

Thanks for the tip!

Have you tried getting this to work without cy.origin?

Based on your info above I have altered the test that it no longer sets Cypress.config('baseUrl', baseUrl);. Unfortunately the result is the exact same

AtofStryker commented 2 years ago

That is completely fine! As long as we have steps to set up the docker orchestration (or a docker-compose file) to get everything running and reproduce the issue.

denke8 commented 2 years ago

Hi @AtofStryker, thanks for the patience!

I created the reptoduction repo based on the official reproduction repo: https://github.com/denke8/experimental-origin-cookie STR, A, E are in the readme of the repo

Note, I managed to score a single successful run in Electron, but it too fails 90% of the time. Chrome seems to 100% reproduce the issue, and FireFox (I only dud a couple of runs) also seems to fail

Again, thanks for checking on the issue!

AtofStryker commented 2 years ago

Hey @denke8. Thank you for creating a reproduction! I am trying to give it a run, but am getting the following issues: Screen Shot 2022-08-15 at 4 23 36 PM.

I am on MacOS. Are you running on native linux?

denke8 commented 2 years ago

Hey @AtofStryker,

Yes, I am on ubuntu 22.04, not sure what other steps are required for MacOS (like verify, etc.)

Please confirm that you did run npm run prepare:containers successfully beforehand

Also, you can run that script manually too from the shell like scripts/reset_testing_env.sh --base foobar.loc for more debug information

I have also just seen that Cypress 10.5.0 got released, so I'll check that out tomorrow and report back

denke8 commented 2 years ago

I tried it with the latest Cypress version, 10.5.0, (Chrome 104) but unfortunately it did not fix the issue. I updated the repo to reflect this.

AtofStryker commented 2 years ago

Hey @denke8. I did run the npm run prepare:containers before launching cypress. I can try running running on my Linux Mint partition for a quick reprod, which I think is on mint version 21 (which should be Ubuntu 22).

denke8 commented 2 years ago

@AtofStryker Sounds great, let me know if I can help in any other way .... video, debug output, etc.

denke8 commented 2 years ago

@AtofStryker @rachelruderman Could you please clarify if any additional information is needed from me in this issue?

Asking because the issue "timeline" states that the "Needs Reproduction" label was removed 5 days ago, but it still shows the flag as active for me. Not sure what's happening.

rachelruderman commented 2 years ago

Hi @denke8 , thank you for checking in and above all, thank you for your patience!

let me know if I can help in any other way .... video, debug output, etc.

These are always super helpful. You can find instructions here on how to print your debug logs

the "Needs Reproduction" label was removed 5 days ago

This was an oopsie, I just added it back on. I'm working on reproducing your issue, I hope to get back to you tomorrow 🙏

denke8 commented 2 years ago

Hi @rachelruderman! No problem at all, thanks for checking on this issue!

It seemed like a better approach not to mix in the regular output, so here is the debug log printed to stderr: stderr.txt

And the stdout output was the following

Please note that there are a total of 3 test (attempts) in the log, in the following order:

Hope this helps

export DEBUG=cypress:* && npm run cypress:run 2>./stderr.txt

> cypress-test-tiny@1.0.0 cypress:run
> cypress run --browser chrome

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        10.6.0                                                                         │
  │ Browser:        Chrome 104 (headless)                                                          │
  │ Node Version:   v16.17.0 (/usr/bin/node)                                                       │
  │ Specs:          1 found (manualActivation.cy.js)                                               │
  │ Searched:       cypress/e2e/**/*.cy.{js,jsx,ts,tsx}                                            │
  │ Experiments:    experimentalSessionAndOrigin=true                                              │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  manualActivation.cy.js                                                          (1 of 1)

  Manual plugin activation
    w/ random domain
      (Attempt 1 of 2) should display signup CTA when autoconfig fails
      1) should display signup CTA when autoconfig fails
    w/ standard domain
      ✓ should display MS admin link when autoconfig succeeds (11780ms)

  1 passing (2m)
  1 failing

  1) Manual plugin activation
       w/ random domain
         should display signup CTA when autoconfig fails:
     AssertionError: Timed out retrying after 40000ms: Expected to find content: 'Plugins' but never did.
      at Context.eval (http://wp-plugin-e2e.loc/__cypress/tests?p=cypress/e2e/manualActivation.cy.js:111:10)

  (Results)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Tests:        2                                                                                │
  │ Passing:      1                                                                                │
  │ Failing:      1                                                                                │
  │ Pending:      0                                                                                │
  │ Skipped:      0                                                                                │
  │ Screenshots:  2                                                                                │
  │ Video:        true                                                                             │
  │ Duration:     1 minute, 52 seconds                                                             │
  │ Spec Ran:     manualActivation.cy.js                                                           │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

  (Screenshots)

  -  /home/denke/work/320ny/experimental-origin-cookie/cypress/screenshots/manualActi     (1280x720)
     vation.cy.js/Manual plugin activation -- w random domain -- should display signu               
     p CTA when autoconfig fails (failed).png                                                       
  -  /home/denke/work/320ny/experimental-origin-cookie/cypress/screenshots/manualActi     (1280x720)
     vation.cy.js/Manual plugin activation -- w random domain -- should display signu               
     p CTA when autoconfig fails (failed) (attempt 2).png                                           

  (Video)

  -  Started processing:  Compressing to 32 CRF                                                     
  -  Finished processing: /home/denke/work/320ny/experimental-origin-cookie/cypress/v   (10 seconds)
                          ideos/manualActivation.cy.js.mp4                                          

    Compression progress:  100%

====================================================================================================

  (Run Finished)

       Spec                                              Tests  Passing  Failing  Pending  Skipped  
  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ ✖  manualActivation.cy.js                   01:52        2        1        1        -        - │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘
    ✖  1 of 1 failed (100%)                     01:52        2        1        1        -        -
rachelruderman commented 2 years ago

Hi @denke8! I used a Linux VM to run your repro and I was able to reproduce the behavior you're seeing 🎉 Thank you for putting together a simple repro for us with clear steps in the README, it helps loads ❤

denke8 commented 2 years ago

Hi @rachelruderman!

I'm glad that it helps, and thank you for checking this issue out!

denke8 commented 2 years ago

I have updated the reproduction repo to Cypress 10.7, so you can work against the latest version. The problem is also present in 10.7

AtofStryker commented 2 years ago

Hey @denke8. We have some cookies fixes likely coming out soon that might address this issue. When Cypress releases next week, can you give it a shot and see if some of the cookie fixes solve you issue?

denke8 commented 2 years ago

Hi @AtofStryker!

Absolutely, I'll check against the next version as soon as it's out and report back / update the repo with the latest version if it still is not working.

Thank you for the info and all the work the team puts in!

denke8 commented 2 years ago

Hi @AtofStryker!

I have checked against the latest Cypress (10.8), unfortunately, it's still broken. I updated the description above and the reproduction repository

AtofStryker commented 2 years ago

@denke8 I'm sorry to hear that. Hopefully I can take a look at this issue again soon to hopefully determine a root cause.

denke8 commented 2 years ago

Reproduction updated to Cypress v10.9, unfortunately it did not fix the issue

denke8 commented 2 years ago

Reproduction updated to Cypress v10.11, unfortunately it did not fix the issue

@AtofStryker Did you happen to have a chance to take a look?

AtofStryker commented 2 years ago

@denke8 Thank you for being patience with us. Sorry I haven't been able to take a look. But I do have some bandwidth available in the coming days so I might be able to take a look and since if I can run again on my ubuntu machine. You deserve an answer and I'm going to work through getting you one.

AtofStryker commented 2 years ago

@denke8 Been super busy with getting a lot of the origin work ready for Cypress 12, but I still haven't forgotten. Hoping to take a look maybe Sunday or Monday if I can!

denke8 commented 2 years ago

@AtofStryker Thanks, appreciate the help!

denke8 commented 1 year ago

Hi @AtofStryker,

I have updated the reproduction repo to 12.2.

It still does not work, but now WP flat out states that cookies are not supported by the browser image

goska-malaga commented 1 year ago

It looks like I run into similar issue on cypress 12.3 (previously I tried cy.origin on 9.6 and it worked) I am getting communication error that shows when cookies are expired (not sure if that is any helpful info that website is build on vaadin framework https://vaadin.com/), then I tried this company admin portal and it is also on WordPress so I run to same result as shown on the screenshot. I do however run to this issue on the login page and one site blocks the login button on the wordpress page I actually didn't use correct login details but I get the same error anyway so it looks like it is easy to replicate. I got in fact impression that it worked just once as I approach to implement it and then as soon I started adding cy.session it broke and I keep trying to investigate it since last week.

And my steps are quite basic e.g. cy.origin('https://portal.lobster.de/', () => { cy.visit('https://portal.lobster.de/'); filling out fields and clicking the button

same steps works on other domains like aws

I find this interesting in replication step: cy.wait(500) // The login page clears/resets the input from js whan loaded, so need to wait as it takes indeed a noticable amount of time (other words it is visible slow to use cy.origin) and I wonder if there is anything related that when using experimentalModifyObstructiveThirdPartyCode: true it blocked the js so the login page didn't load at all (and it was cached quite well so I could not get rid of it for some time)

after all I am quite disappointed that feature that worked as experimental doesn't work when released and I cannot even run it on older version as the experimental flag no longer exists

denke8 commented 1 year ago

Hi @AtofStryker,

I have updated the reproduction repo to 12.5.1.

Would you happen to have any updates for this?

AtofStryker commented 1 year ago

Hey @denke8. Sorry for the delay here. My linux machine is currently bricked due to some issues with lightdm, so I am trying to give this a go on my mac. I keep running into issues with the reset_testing_env.sh script with getopt for illegal options?

I tried switching from zsh to bash to verify in the terminal and still have the same issue

any ideas?

denke8 commented 1 year ago

Hi @AtofStryker,

If I remember correctly I encountered the issue too, but the script did run correctly regardless. Please let me know if this is not the case

IIRC this error was only shown when the script was executed from under Cypress, but could not figure out the exact reason

AtofStryker commented 1 year ago

@denke8 I also seem to be running into it just through bash as well.

I tried setting failOnNonZeroExit: false for the exec command to see if I can sidestep the failure. We get further, but it looks like the WP server isn't set up and needs configuration?

denke8 commented 1 year ago

@AtofStryker This is very interesting. Yes, it seems it does not set up correctly in your machine. The only thing I can think of, is a userId mismatch. I am running this from the latest LTS ubuntu (22.04), so I'm using 33:33 as user and group when executing wordpress-cli. I don't think that this should cause any problems, because we are talking about interaction within two docker containers, but still:

https://hub.docker.com/_/wordpress please search for "Running as an arbitrary user"

Alternatively, I can create a VM image with a full linux install and docker if you want me to (and upload the disc file somewhere) ... not sure how else to proceed

AtofStryker commented 1 year ago

@denke8 the VM image might be the way to go since my personal linux machine is currently bricked and not sure what type of permission issues I am going to run into on my mac. I have a VMWare install on my machine and can run an ISO. Just shoot me a link if/when you're able to upload it!

denke8 commented 1 year ago

@AtofStryker I have created an .ova file that you should be able to import. Do you have a way I can privately send you the link?

AtofStryker commented 1 year ago

can you send it to support@cypress.io? Let me know once you send it and I can ask support for it!

denke8 commented 1 year ago

@AtofStryker

Sent, please look for an email with the title VM image - bug reproduction for #23165

wizzyto12 commented 1 year ago

I have the exact same issue on a website. When entering the website, it creates a cookie PHPSESSID, which is not preserved while the login form is submitted and therefor the CSRF token fails.

denke8 commented 1 year ago

Hi @AtofStryker I have updated the reproduction repo to the latest 12.17 version, but the issue still persists. Did you happened to have a chance to take a look?

AtofStryker commented 1 year ago

@denke8 I have the OVA file on hand but haven't had a chance to install the image and investigate. I'm hoping to have time on next rotation to dig in a bit more.

denke8 commented 1 year ago

The issue unfortunately is still present, updated the reproduction to the latest version, 13.2.0

kgartland-rstudio commented 10 months ago

I'm hitting this too. Is there a workaround?

denke8 commented 10 months ago

@kgartland-rstudio Not to my knowledge

@AtofStryker The reproduction repo has been updated to 13.6.2, the issue seems to persist.

lennerd commented 7 months ago

Having the same problem when testing a login feature with Keycloaks OAuth. Any updates?

denke8 commented 7 months ago

@AtofStryker I have updated the reproduction repo to 13.7.2. Would you happen to have any updates on this issue?

AtofStryker commented 7 months ago

@denke8 unfortunately no updates on the issue. I appreciate you updating the reproduction though!