cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
46.75k stars 3.16k forks source link

File watching does not execute actual code changes when site under test utilizes Service Workers. #702

Open brian-mann opened 6 years ago

brian-mann commented 6 years ago

Related to #686.

It's possible the application under test ends up using Server Workers to inadvertently cache the files Cypress uses to serve spec files. This prevents changes to the spec files from ever being updated.

Prior to the beginning of all tests we should clear the Service Worker cache and then add a new cy command and add this to lifecycle rules cy.clearServiceWorkerCache().

We could make this command print out all the cache keys it cleared.

Workaround:

before(function () {
  // run this once before all code
  return window.caches.keys().then(function(cacheNames) {
    return Promise.all(
      cacheNames.map(function(cacheName) {
        return window.caches.delete(cacheName);
      })
    );
  })
})
brian-mann commented 6 years ago

Additionally seeing problems with ServiceWorkers caching the /__ root client path which prevents Cypress from being served entirely.

brian-mann commented 6 years ago

Another potential lifecycle command we could add at the beginning of each test, and then subsequently at the end of all tests is something like: cy.clearServiceWorkerRegistrations()

The implementation for that might look something like:

window.navigator.serviceWorker.getRegistrations()
.then(function(registrations) {
  return Promise.map(registrations, function(registration) {
    return registration.unregister()
  })
})
brian-mann commented 6 years ago

We could even indefinitely prevent service workers from ever being registered by doing:

cy.visit("https://testy.gift/", {
  onBeforeLoad: function(win) {
    const promise = new Promise(function(resolve) {});
    return win.navigator.serviceWorker.register = function() {
      return promise;
    }
  }
})
brian-mann commented 6 years ago

We need to make a decision here. Currently ServiceWorkers end up generating very confusing behavior in Cypress which is hard for new (and even old) users to debug.

Proposal

  1. Prevent ServiceWorkers from ever being generated by default.
  2. Add two new commands above to enable clearing cache + registrations before each test + after all the tests finish
  3. Add a new config property: enableServiceWorkers: false by default (?)

When we enable lifecycle events this could potentially conflict...

HenrikJoreteg commented 6 years ago

so as a workaround as long as it doesn't cache the stuff at /__ we're good right? I could tweak the SW to reflect that.

brian-mann commented 6 years ago

Yes, you could. That is the only request of Cypress' that you are caching accidentally.

e-monkey commented 6 years ago

The workaround does not work for me:

Error: Failed to execute 'keys' on 'CacheStorage': Only secure origins are allowed (see: https://goo.gl/Y0ZkNV).

Because this error occurred during a 'before all' hook we are skipping all of the remaining tests.
    at Context.

The test files only update, when i close cypress or when i manually clear the cache through the Chrome settings.

Did not found a way to solve it or another workaround.

mxstbr commented 6 years ago

Prevent ServiceWorkers from ever being generated by default.

I want to be able to e2e test my serviceworker's behavior, so I'd hate to see this happen. I'd love to see an API to clear the caches before each test run though!

jennifer-shehane commented 6 years ago

@mxstbr What behavior of the service worker are you testing? We'd like to hear your more about your use case so we can properly address our side.

mxstbr commented 6 years ago

Caching of frontend assets and potentially web push notifications?

cdaringe commented 6 years ago

another hack i used was to have my web-app detect cypress. my CSP is already no-frame, so, in dev mode w/ the csp off, i can sniff for if (window !== window.parent) { ... } and just turn off service worker registration. window.Cypress is also a thing...

rcrichton commented 5 years ago

I solved the issues I was having by updating my navigation fallback config in sw-precache. This is what I added:

navigateFallbackWhitelist: [/^\/[^_]+$/], // fallback for anything that doesn't start with /__ which is used by cypress
mackelito commented 5 years ago

Still no workaround to clear service worker?

abigailcauchi commented 5 years ago

@mackelito I've managed to unregister all service workers before each test in one of my specs like this:

beforeEach(() => {
  if (window.navigator && navigator.serviceWorker) {
    navigator.serviceWorker.getRegistrations()
    .then((registrations) => {
      registrations.forEach((registration) => {
        registration.unregister()
      })
    })
  }
})
brandynprasad commented 5 years ago

@abigailcauchi This snippet helped me, thank you.

The application I'm working heavily relies on service workers. I noticed my cypress spec files would pass when run independently, but when I chose to run all specs the browser launched by cypress would fail to load some pages (resulting in failed assertions!).

After unregistering service workers in my before hooks, my specs are running much faster and passing consistently. Thank you again.

nkbt commented 5 years ago

Prevent ServiceWorkers from ever being generated by default.

Using ServiceWorkers for caching is only one use case scenario. They are effectively built-in proxy servers and we are using SW for example to add auth header for all outgoing requests and not using SW for caching at all.

Just my 2c.

johnpolacek commented 5 years ago

This one really threw me for a loop. Working on a Gatsby project, where you get service workers out of the box. After some time, adding pages and changing things in the project, I experienced the same thing as @brandynprasad where pages would not load. I switched to the default gatsby starter, installed cypress then made a simple navigation test. On the first run, the Cypress test runner rendered the top level page from my other project instead! After poking around, I tried the solution from @abigailcauchi and that did the trick.

When the test runner opens, it should not inherit any service workers from previous runs.

anupnitkkr commented 4 years ago

@brian-mann : what is the best way to integrate service work to improve running time performance in build . So once cached JS and html files will be served for all test cases

filiphric commented 4 years ago

@brian-mann how would you solve the situation with service workers when you are navigating from one site to another. in a scenario like this:

  1. go to homepage (e.g. www.example.com) - no SW
  2. click on redirect link
  3. navigate to app (e.g. app.example.com) - has SW

deleting SW with workarounds you are mentioning does not seem to delete SW from app. I tried to tie the code to window:before:load event too, but it does not seem to do the trick. Navigation does not stop or wait for SW to be deleted, which I guess might be the core problem

SleepWalker commented 4 years ago

I've disabled SW with the following snippet:

Cypress.on('window:before:load', win => {
  // disable service workers
  delete win.navigator.__proto__.serviceWorker;
});

source

koszonetdoktor commented 4 years ago

For me the solution was to check in the UI code if Cypress is running, and if yes, i just unregister the service worker, like this:

if (window.Cypress) {
    serviceWorker.unregister()
} else {
    serviceWorker.register()
}
jakobrosenberg commented 4 years ago

Is it possible to do this with Cypress?

Step 1. Clear service workers and cache
Step 2. Refresh window and reinstall service workers with a fresh cache
Step 3. Run tests

I've spend the last week trying to figure this out and I'm no closer. I've tried the following in a million different variations.

jbpallingayan commented 4 years ago

hello any updates on this.

rpong commented 4 years ago

Any plans regarding this? We are working on a PWA app and working around this is proving to be cumbersome as of the moment, we always have to manually clear Application Cache for our changes to take effect.

vrknetha commented 4 years ago

I have disabled the service workers when 'window.Cypress' but still it is not working. Anything on this yet.

mikerich commented 4 years ago

I came across this issue yesterday. I found I could get around it in Chrome by going to Dev Tools->Application->Service Workers and checking the 'Bypass for network' checkbox. Not ideal but it got me moving again so I could see changes I made to my test. Changing "watchForFileChanges" did not affect this issue.

Not an issue when running from the command line. Only from the test runner.

App is React/Gatsby/PWA using gatsby-plugin-offline (a workbox wrapper)

davidjeddy commented 4 years ago

Any plans regarding this? We are working on a PWA app and working around this is proving to be cumbersome as of the moment, we always have to manually clear Application Cache for our changes to take effect.

Same here. Just spent the better part of two days figuring out the root cause of our Cypress/Lighthouse tests failing. (using https://github.com/mfrachet/cypress-audit). We have to manually checkbox Bypass for network for the evaluations to work as expected. It appears the setting stays in the chrome instance controlled by Cypress, independent of the a users chrome session.

nicholaschiang commented 4 years ago

I'm using a service worker to handle authentication by intercepting fetch requests and appending a JWT in their Authentication headers (as @nkbt described). Without that service worker, the app doesn't know if it's logged in or not.

Currently during development, I always open up developer tools and check the "Allow service workers over HTTP" box like so: image

With Cypress, I can do the same thing when running cypress open and actually viewing the browser, but I don't know how to change the browser settings for Electron or headless Chrome when Cypress runs on my CI server.

Any suggestions for how to ensure that service workers are allowed over HTTP in the headless browser that cypress run uses?

nicholaschiang commented 4 years ago

Nevermind, I seem to have fixed my own issue by serving to localhost instead of 0.0.0.0 (Chrome seems to only be able to allow service workers over HTTP when they come from localhost).

RaviH commented 3 years ago

Hi, any update on this?

nicholaschiang commented 3 years ago

Related to #15670.

csvan commented 2 years ago

Related: https://github.com/cypress-io/cypress/issues/16192

kylemh commented 2 years ago

There doesn't seem to be a workaround for this for me...

Cypress.on('window:before:load', win => {
  if (window.navigator && 'serviceWorker' in window.navigator) {
    window.navigator.serviceWorker.getRegistrations().then(registrations => {
      registrations.forEach(r => r.unregister());
    });
  }

  // @ts-ignore
  delete win.navigator.__proto__.ServiceWorker;
  // @ts-ignore
  delete win.navigator.serviceWorker;
});

but the service worker still registers

KseniaCHG commented 1 year ago

Hello, I am a little junior and I've been struggling with this a lot, the solution is simple - clear cache in a browser that Cypress is using. Version Cypress 12.7.0 hope that helps :)

ohlori commented 10 months ago

I have encountered this just now. Cypress version 13.5.0.