search

happo / happo-e2e

Helpers for Happo's e2e integrations (Playwright, Cypress)
1 stars 0 forks source link

Happo e2e version 3.1.0 increases the execution time of the Happo Tests #46

Open sankar-refined opened 1 day ago

sankar-refined commented 1 day ago

Hi,

We have been using Happo as part of our visual regression testing tool. We have been using it for over a year now.

Recently I tried updating the happo-e2e package from version 2.4.1 to version 3.1.0.

I could see that the execution time for the happo tests have increased from 6 minutes to 21 minutes.

We are not sure of what is the problem, but here are our observations.

with 2.4.1, we get the output of [happo] > testname (20 seconds) we get 404 failed to fetch url errors, but they are printed as a single line (they are as designed and as expected)

But with 3.1.0, we get a more extensive output from Happo,

(here it starts the actual happo tests)

From my observation, There are too much information printed on the bitbucket console. and that increases the execution time for the Happo tests that we have in the pipeline. ( that 1 line printout in 2.4.1 prints out over 1400 lines of output on bitbucket in 3.1.0 version)

lencioni commented 1 day ago

Hello @sankar-refined! I'm sorry to hear that you are seeing some performance issues when updating your happo-e2e package. Going from 6 minutes to 21 minutes is quite the change! Thank you for reaching out! I'd like to help get this resolved for you.

It looks like there were a few releases between 2.4.1 and 3.1.0. Would it be possible for you to narrow down the issue to a more specific version? That will help us pinpoint the issue.

Also, are you changing anything else at the same time or have you already isolated the happo-e2e package. I just want to make sure that we are looking at the right thing before we dig too deeply.

sankar-refined commented 1 day ago

The issue starts version 2.6.0. It is from this version, we are seeing the performance issue.

lencioni commented 1 day ago

Thanks for narrowing that down! Here are all of the changes between 2.5.0 and 2.6.0: https://github.com/happo/happo-e2e/compare/v2.5.0...v2.6.0

It looks like it is just 2 small PRs:

I'm wondering if we can narrow it down to one of these specific changes. Do you happen to be using <canvas> at all? If not, then we can easily rule out #29. I'm wondering if this happened because of #31 and you have some stylesheets that were not previously correctly resolved?