mjhenkes
commented
2 years ago On first glance this appears to be a duplicate of https://github.com/cypress-io/cypress/issues/22353
Closed SadiqRahim closed 1 year ago
mjhenkes
commented
2 years ago On first glance this appears to be a duplicate of https://github.com/cypress-io/cypress/issues/22353
lmiller1990
commented
2 years ago I do not think this is duplicate of #22353, because that is only related to Component Testing, where we changed from 1 tab per run to 1 tab per spec. E2E changed from one browser launch per spec to a single browser launch, opening a new per spec, which is substantially faster.
Also, #22353 points out the entire run time is slower, but the individual spec run time is not - the time is getting lost on rendering the runner UI, parsing JS, etc. This screenshot seems to imply the actual spec execution is slower, which would indicate a different issue.
@SadiqRahim I'm quite curious here... your test takes 20 seconds to execute:
cy.window().then((appWindow) => {
let spreadHostElement = appWindow.document.querySelector(pageObj.forecastObj.hostElement)
let spread = appWindow.GC.Spread.Sheets.findControl(spreadHostElement)
let activeSheet = spread.getActiveSheet()
activeSheet.setActiveCell(4, 5)
activeSheet.startEdit(true, '10')
cy.get('.cc-forecasting.cc-ui-block').type('{enter}')
cy.wait(300)
})
cy.get(':nth-child(3) > .cc-ui-button').click({force: true})
cy.get('.cc-company-forecasting-save > .cc-ui-field > input').type('New forecast')
cy.intercept('POST', '/graphql', (req) => {
aliasMutation(req, 'SaveForecast')
})
cy.get('.cc-company-forecasting-save__actions > .cc-ui-button').click({force: true})
cy.wait('@SaveForecast')
cy.get('.is-active > .cc-ui-icon > .iconPlus').click({ force: true })
cy.wait(1500)Do you have any feeling on which part of this is slow? cy.get and click are basically instant. The only thing that looks like it would be taking time here is waiting for API requests. Are you able to clarify which part(s) of your test seem to be taking longer/account for the additional 5 seconds?
mjhenkes
commented
2 years ago @SadiqRahim, Is it just that test in question that is substantially slower or are other tests also slower? Could you comment out lines in that test to try to narrow down which step is taking the most time?
SadiqRahim
commented
2 years ago @lmiller1990 @mjhenkes thanks for looking at it. It slows down ridiculously at the cy.visit() command. I use the cy.visit() command in two different places on the code
I run the same code
Note that I have tried running them on electron and chrome and I see the same slowness on both the browsers
The migration instruction to v10 does not have any comments/notes about any lack of performance nor does it have any related fixes in the late releases. Thus I'm not quite sure what's going on.
lmiller1990
commented
2 years ago I completely misread your screenshots - I didn't fully grasp how much slower this was for you.
I think it's something on your end, but happy to help you debug - if cy.visit was really 6x slower, we'd see it in our test suite, or the entire product would be on fire.
The best way to start debugging is using our debug logs. Run your test with DEBUG=cypress:* yarn cypress ... and it will print lots of logs in the terminal. You could compare 9.x and 10.x and see if there's anything different. Especially pay attention to what is printed during cy.visit.
If you could do this and copy paste the logs here (they'll be huge, no problem) that will greatly help. Without a minimal reproduction or some logs, we can only really speculate.
mjhenkes
commented
2 years ago @SadiqRahim Have you been able to grab the debug logs? I'm fairly curious about what is going on with the slowness.
mjhenkes
commented
2 years ago SadiqRahim
commented
2 years ago @mjhenkes Thanks for looking at it. Sorry that I just got back from my holidays. Let me look at the issue #22528 and get back to you with more info.
SadiqRahim
commented
2 years ago @mjhenkes @AtofStryker @lmiller1990 My apologies for the late in reply. I did a quick debug only by visiting to see the difference just with the following code,
cy.visit('https://qa.cognitivecredit.com')
I ran the same line of code with 10v & < and 9v & >
The obvious diff I see between them are the time taken on cypress:server:request and cypress:network:cors and 10v & < running additional requests, which I do not see on 9.7.0v & below. Also, I'm able to see this issue only on our lower environments and one of the url is called on the code above.
This is replicable if you try with the pertinent versions. Something has evidently changed on the 10v and above. Appreciate any help on this pls as we're blocked to upgrade this to the latest cypress version,
AtofStryker
commented
2 years ago Hey @SadiqRahim. What operating system are you on? Is the issue reproducible just from running cy.visit('https://qa.cognitivecredit.com/') between v9 and v10?
SadiqRahim
commented
2 years ago @AtofStryker I'm trying to run it on my MacOS Monterey version 12.1. Yes its reproducible just from running that line of code between v9 and v10. Thanks for looking at this. Let me know should you need any more details pls.
AtofStryker
commented
2 years ago @SadiqRahim I was able to reproduce the issue and made a repository here
lmiller1990
commented
2 years ago I think the problem is our network proxy has a bottleneck when there is many XHR requests. This website's login page makes over 800 XHR requests, and ~500 CSS resources. Just to clarify, @SadiqRahim, is that intentional? It takes 30 seconds to finish even outside of Cypress - I think what's happening is Cypress is waiting for all network activity to stop, which takes a very long time.
Prod in browser:
Cypress:
SadiqRahim
commented
2 years ago @lmiller1990 Those are icons, images downloads. It appears to be because some of the popular browsers does show everything within the xhr panel nowadays particularly chrome I think. However, this area was not changed lately from our development point of view. Also no indication of such issue occurred on cypress 9v or below. The problem appears to be only on any of the 10v. So i'm unclear as to what has changed on 10v that this was loading faster on all other versions below 10v :(
public
commented
1 year ago +1. Cypress 10 (and 11) is more than 2x slower than 9 for us.
This is very frustrating because 9 uses Rosetta to make x64 build of Chrome it uses work on Apple Silicon which is extremely slow in local development. (e.g. devtools takes 30+ seconds to open.) So our options are currently either that every single CI build takes 30 minutes instead of less than 10 minutes, or our local development experience is incredibly slow.
lmiller1990
commented
1 year ago @public that sounds painful. This performance issue from 9->10 seems to only impact a subset of users (or sites? hard to say). In this issue, based on the debugging I've seen, this issue is greatly amplified for sites making large (several hundred) XHR requests on the initial load. I wonder if something in the proxy layer got changed between 9->10.
To debug a little more, can you share some info about the site you are testing? Are you primarily seeing this slowdown on a specific site (eg, the one you work on) or across all sites in general? If it's specific to 1 website, is it the kind of app that makes a large amount of XHR requests as soon as it's loaded? If so, this would be the same issue described in this issue, and help us narrow down the surface area for the problem.
public
commented
1 year ago @lmiller1990 We don't do a tremendous amount of XHRs or other requests within the page but we do quite a lot of tasks to setup specific test data in our backend.
Our test suite is very site specific so I can't tell you if it affects other sites or not. It is also somewhat difficult to really narrow down the problem as when running locally I only have Apple Silicon hardware to run on which is not a particularly fair test for Cypress 9 as I mentioned above.
lmiller1990
commented
1 year ago Hmm I had a similar problem on my M1, but my colleagues did not. I have a feeling my problem was something in the Rosetta compat layer (I am not sure if I messed something up during configuration). I was specifically seeing this problem when using Chrome, but it was nice and quick with Firefox. Are you seeing this problem on other browsers?
I am in the process of reformatting my M1 to see if that will fix the slowness I had there. Will report back, but I think you/mine M1 specific issue is unrelated to the issue described by the OP.
onthespotqa
commented
1 year ago @lmiller1990 so i don't think we have an unreasonable amount of requests in our app. But we do frontload a lot of data when you initially log in. We do use apis to create test data during the test.
i've noticed that the app is slow to load when running locally. Like on initial run the app takes over 30s to load the first page. Not on an M1 chip and generally using chrome/electron.
Where i noticed the slow down the most though was in ci. the time to complete doubled (~17 mins on avg to ~35 on avg with cypress 10). We did not change how we write tests. We got things down to a more manageable size in ci by increase the machine executor we use from circle's medium to large.
lmiller1990
commented
1 year ago Ok, if it is in CI it's definitely not related to M1 or Rosetta emulation.
Other thing to debug would be running both 9.x and 10.x side by side with DEBUG=cypress:*. You can narrow down the logs with more specific debugs, eg DEBUG=cypress:network* etc... I'm not really sure how much else I can help without access to your app, but the best advice I can offer is trying to figure out what part of Cypress is causing the bottleneck.
I don't think much changed in the network layer - it was mostly UI changes - but evidently something changed, somewhere, that's causing a significant bottleneck.
jueinin
commented
1 year ago i have this problems too。 when using cypress open local vite based page. It has about 200 js requests. in Chrome it takes about 5s to load. With cypress v11 it takes about 1 minute
lmiller1990
commented
1 year ago Thanks for posting @jueinin - can you try running your tests against the built version of your Vite app (not development) and see if the problem persists?
If so -- I think we are narrowing down on the issue, a bottleneck in the network layer when requesting JS modules. I do not know the network layer very well, but my understanding is we do some rewriting of JS modules in the rewriter package. I wonder if this is the problem?
If you can confirm it works better against your production build, it could be a dev-server with ES modules bottleneck - I could try build a binary without the rewriting logic for you, to test. Then, if it fixes the issue, we can find a more permanent solution. This could improve perf across the board.
jueinin
commented
1 year ago @lmiller1990 I have build this project to a index.esm.js. about 900KB, when load this file from vite in chrome disable cache mode, it takes about 60ms, when load this js from cypress it takes 150ms to 400ms. when enable cache(return 304 status code), in chrome it takes about 2-30ms, in cypress it takes about 60ms
lmiller1990
commented
1 year ago Right - I think Vite relies heavily on caching. I am fairly sure Cypress clears the cache each test to ensure a fresh, deterministic run, resulting in 2x-3x slowdown -- I guess this is the problem. I think that happens here. This only happens between specs - not tests - once your tests start, does everything proceed nice and quickly? Is the first test for each spec the only one that takes a long time?
Did you also have the same problem in Cypress 9? Were using using Vite in Cypress 9?
I wonder if there is way we could let users to change the caching behavior in Cypress to speed things up.
lmiller1990
commented
1 year ago I am fairly sure this is a caching issue. @SadiqRahim - I tried the original reproduction again - it's VERY slow (times out) on the first try. Subsequent loads are nice and quick. This is the same across Cy 9 and Cy 10, though.
Also I noticed the website constantly makes ongoing requests for lots of tiny CSS files - is that correct? Probably unrelated, but I let it sit there for a while and the number of requests keeps climbing.
@jueinin how about on your end - once it loads once, and you re-run (click the little re-run icon in the command log) is it quick the second time? This would confirm a caching issue.
This icon to re-run:
jueinin
commented
1 year ago @lmiller1990 it seems not cache to blame, when i click rerun, it's almost no different times to load. I am just running one test. it seems cypress will proxy and rewrite all js files? may be the problem is here? i noticed the CPU is about 30% when i refresh page in cypress, while in browser is about 8%.
jueinin
commented
1 year ago by the way, do you know how to set timeout of mocha testcase? in here it was set to false?
lmiller1990
commented
1 year ago @lmiller1990 it seems not cache to blame, when i click rerun, it's almost no different times to load. I am just running one test. it seems cypress will proxy and rewrite all js files? may be the problem is here? i noticed the CPU is about 30% when i refresh page in cypress, while in browser is about 8%.
There is some rewriting - I need to look into this part of the code base more, and find out exactly why.
by the way, do you know how to set timeout of mocha testcase? in here it was set to false?
There's a few ways to change the timeout, how about:
describe('some test', { defaultCommandTimeout: 10000 }, () => {
// ...
})You can also set them globally in your cypress.config.js.
Timeout is set to false - a test times out when a command fails to complete. There is a few different timeouts you can control, see here.
jueinin
commented
1 year ago @lmiller1990 could you please help with this issue? the performance is almost crazy!!!
and this issue seems more serious in Windows
lmiller1990
commented
1 year ago I will try and get some more eyes on this internally, especially those with more experience in the network layer than I.
kyleburnett
commented
1 year ago I suspect we might be running into the same issue. We were trying to do an upgrade between v6.6.0 and v11.2.0. I let Cypress automatically handle renaming the integration folder to e2e, converting cypress.json to cypress.config.js, etc. and that all went fine. Then I noticed that the build times for our Cypress jobs in CI went from less than 15 minutes to over 30 minutes.
Once you have a version to fix the issue reported above, I would be interested to test it out to see if it also addresses the performance in our upgrade scenario.
lmiller1990
commented
1 year ago I'm working on reproducing this. I've tried a bunch of the apps I work on, and on the Cypress monorepo, but the results are basically 1:1.
Cypress 9
1 Spec 10 times: ./run.sh 36.48s user 7.91s system 51% cpu 1:26.19 total
10 specs 1 time: ./run.sh 26.07s user 4.48s system 41% cpu 1:14.49 total
Cypress 10
1 Spec 10 times: ./run.sh 34.47s user 9.54s system 51% cpu 1:25.83 total
10 specs 1 time: ./run.sh 27.68s user 4.65s system 42% cpu 1:15.38 totalCan you provide some info about your project @kyleburnett? Do you run against a dev server (webpack/vite?) Does your project make a large amount of network requests?
I am going to keep trying out different apps until I find something that suffers from the same slowdown everyone is describing. FYI I am working on linux (which is what I suspect most people running Cypress on CI providers are using).
lmiller1990
commented
1 year ago Ok team, I think I had some luck reproducing this. I will record and get some numbers, will update the post as I go.
Edit: run mode similar on my local machine (linux, beefy 32GB ram, beast CPU) but slower on CI (see here).
Open mode looks suspect - but most of the problems here seem to be around CI. Here's what I was able to reproduce:
Pretty quick.
Look how much longer it takes! I will try reproduce this on CI.
I reproduced it by adding lots of tiny JS modules:
import * as mod1 from './mod-1'
import * as mod2 from './mod-2'
import * as mod3 from './mod-3'
import * as mod4 from './mod-4'
...
import * as mod200 from './mod-200'Each module is just export const foo = 'FOO'. It looks like somewhere between Cypress 9 and 10, something changed that makes loading lots of JS modules (and maybe other files) much slower.
This is a Vite app. I tried a Vanilla app -- no dev server -- and there's no difference in Cy 9 vs Cy 10. That Vanilla app is here.
jueinin
commented
1 year ago @lmiller1990 i have tested on my mac and windows. in mac that's OK. this problem may be Windows special?
lmiller1990
commented
1 year ago @jueinin maybe - one other user says they are on MacOS. It could be windows. I am able to (sometimes) reproduce slowness on Linux, as you can see above.
I can try windows, too, and see what happens. @kyleburnett what OS are you on? Locally and CI?
jueinin
commented
1 year ago 
it takes about 40s to load 300 js files. windows
lmiller1990
commented
1 year ago It's not just you, I am able to reproduce (finally). It's not a minimal reproduction, but we are getting there. This data is the same as the videos I posted above, but I ran it on CI, which is plain old Ubuntu. It's on GitHub Actions.
Notes:
yarn vite build.Cypress 9
Done in 13.51s.
real 0m13.656s
user 0m8.576s
sys 0m1.459s
Cypress 10
Done in 24.01s.
real 0m24.185s
user 0m9.431s
sys 0m2.132sI am still finding a minimal reproduction so we can actually FIX this, but it's pretty clear to me that, at least, loading lots of JS modules (many lots of
Please wait a bit more while I debug this, I will fix it.
kyleburnett
commented
1 year ago I have observed the slowdown in CI, and I have not done a performance comparison locally. I am running a Mac locally, so I will give that a try and report back. In CI, we run Cypress within an Ubuntu 22.04 container.
Here are a few other details:
jueinin
commented
1 year ago is there any way to disable network proxy feature?
jueinin
commented
1 year ago lmiller1990
commented
1 year ago @kyleburnett your situation sounds a little different to the others here. The fact it's so specific makes it difficult to give much more debugging advice, but let's see if fixing the general slowness I'm encountering helps on your end. It sounds like you make very few API requests, so I would not expect the network proxy to be the bottleneck. Weird.
@jueinin I don't think so, I think a lot of Cypress features and the general architecture relies on things going via the network proxy, some info here.
jueinin
commented
1 year ago related issues: https://github.com/cypress-io/cypress/issues/23214
VictorGosse
commented
1 year ago 👋 We are facing the same issue: lot of JS files loaded (~200) and each of them takes ~0.5sec to load when loaded in Cypress. If we load the same page outside of Cypress they are loading really fast (<~0.1sec). It makes local testing really complicated and frustrating. That said, if the page has been loaded at least once, refreshing the test makes it load faster (I guess because the files are in cache).
lmiller1990
commented
1 year ago Hello all, I'm still debugging this. I am chipping away this, bit by bit. In this reproduction, you can see that after exactly 138 JS module requests, it goes slow. The JS modules appear to be loaded in chunks of 6. It loads 6, waits for a (0.5s?) and then loads some more. This seems like a reproduction in line with what @VictorGosse is encountering.
Based on the below, it looks like it's not between Cy 9.7 (last version of Cy 9) and Cy 10.0.0 (first version of Cy 10) but a later regression. This also lines up with the OP:
We were on 9.5.4. Then did the upgrade to the latest version of cypress 10.3.1.
So, assuming this is the same bug for everyone involved, and there's only one bug, it was somewhere between 9.7.0 and 10.3.1.
I had to turn off cache to reproduce this. Makes sense - if the modules do not go through the Cypress network layer, they don't go through the bottleneck.
I will now try to find the commit that introduced this.
Must be in here:
git log --pretty=oneline v10.2.0...v10.3.0 tags/v10.2.0
Edit: yes, here is the offending commit! 172c36402127acbea0473c93760cc6435ccad471. The PR in question: https://github.com/cypress-io/cypress/pull/22462
lmiller1990
commented
1 year ago I found the point where the regression was introduced, here it is:
172c36402127acbea0473c93760cc6435ccad471. The PR in question: https://github.com/cypress-io/cypress/pull/22462
I will find out more about what that PR fixed and if there's another way to handle that fix that doesn't impact performance.
lmiller1990
commented
1 year ago After a certain number of requests this starts happening:
[cy:open:dev:576967]: GET /assets/mod-130.4b328c39.js 200 5.620 ms - -
[cy:open:dev:576967]: GET /assets/mod-131.20fe225b.js 200 6.070 ms - -
[cy:open:dev:576967]: GET /assets/mod-132.0411856e.js 200 10.253 ms - -
[cy:open:dev:576967]: GET /assets/mod-133.0ab33992.js 200 7.729 ms - -
[cy:open:dev:576967]: GET /assets/mod-134.37cfe664.js 200 6.695 ms - -
[cy:open:dev:576967]: GET /rune_delta.png 200 1041.509 ms - -
[cy:open:dev:576967]: Never received pre-request for request GET-http://localhost:5566/assets/mod-135.43411b4c.js after waiting 500ms. Continuing without one.
[cy:open:dev:576967]: ->>>>> pending
[cy:open:dev:576967]: GET /assets/mod-135.43411b4c.js 200 507.778 ms - -
[cy:open:dev:576967]: Never received pre-request for request GET-http://localhost:5566/assets/mod-136.99fc4cd4.js after waiting 500ms. Continuing without one.
[cy:open:dev:576967]: ->>>>> pending
[cy:open:dev:576967]: Never received pre-request for request GET-http://localhost:5566/assets/mod-137.4841f7c7.js after waiting 500ms. Continuing without one.I'll find out more.
lmiller1990
commented
1 year ago More info... it looks like @flotwig fixed this bug in https://github.com/cypress-io/cypress/pull/23227. The original issue the OP reported about his website loading slowly is nice and quick on the latest version of Cypress. @SadiqRahim can you try Cypress 11 (or Cypress 10.6, which has the fix, but always good to be on the latest).
I am still having some issues with the reproduction I put together, though, where I load 400 JS modules. Checking out the commit before #22462 fixes it. I also found I can remove this code and it fixes it:
I wonder if the logic around sweepInterval is incorrect. cc @BlueWinds - maybe you have some context. The original PR says:
It also sweeps unmatched prerequests more than 10s old
But looking at the code:
, this.requestTimeout * 2)requestTimeout is 500 so we sweep every 1 second. Is the code incorrect? I think it should be https://github.com/cypress-io/cypress/compare/fix/correct-sweep-interval?expand=1?. I tried adding that code and the regression I encountered goes away.
lmiller1990
commented
1 year ago Tentative PR https://github.com/cypress-io/cypress/pull/25209
lmiller1990
commented
1 year ago @VictorGosse @kyleburnett @jueinin can any of you try the pre-release binaries with my attempted fix here? https://github.com/cypress-io/cypress/commit/6b78454e0d78346729263e5ab0b77c856a43ce42#comments. Thanks!
I still think there is some other weirdness going on somewhere, for apps with many JS module requests, Cy 9 appears slightly quicker - I'm going to keep digging, but if you can at least verify if this does or doesn't help with the slowness you've encountered, that'd be great.
jueinin
commented
1 year ago sorry, due to time constraints, I have removed cypress and construct our e2e testing with plain mocha and vite. it's very fast!
so may be i can not reproduce this bug again.
Current behavior
We were on 9.5.4. Then did the upgrade to the latest version of cypress 10.3.1. Now the test execution time takes too long. This appears to occur when loading a page. Just the cypress package and cypress.config.js file changed. Screenshot attached with the differences on the same test code. Hopefully the information provided helps
Desired behavior
We expect to run faster or at the same speed
Test code to reproduce
it('Create new forecast', () => { cy.intercept('/transformation/').as('wait') cy.visit(
${global.url}/company/adient/forecasting/annual-iu62wfojz8iourjk74wn) cy.wait('@wait').its('response.statusCode').should('be.oneOf', [200, 307]) cy.wait('@wait').its('response.statusCode').should('be.oneOf', [200, 307]) cy.wait(1500)})
Cypress Version
10.3.1
Other
No response