search

plone / plone.app.robotframework

Provides Robot Framework compatible resources and tools for writing functional Selenium tests (including acceptance tests) for Plone CMS and its add-ons.
https://docs.plone.org/external/plone.app.robotframework/docs/source/index.html
11 stars 17 forks source link

Move ZODB and Zope setup and teardown to the layer setup and teardown #109

Open Rotonen opened 5 years ago

Rotonen commented 5 years ago

I did a bit of profiling, and contrary to a widely held belief, most of the wallclock time spent in the tests is not spent starting up or closing the browser (WebDriver), but rather at setting up and tearing down the plone site (ZODB, Zope) at every suite setup and suite teardown.

To me this'd seem like a big win to move those setups to the layer setup and layer teardown, as they're shared across all of the suites.

mauritsvanrees commented 1 year ago

This would still be very useful. I think I tried this years ago but failed.

One place where it is noticeable, is in the test console output, where every info/warning/error message is displayed lots of times. Currently I see only one (I opened issue #147 for that), so maybe we fixed enough deprecation warnings by now, or are simply not printing them. There used to be a time where this drowned out all other informative output.

gforcada commented 1 year ago

I don't have numbers, and anyway it is running locally, but I have the feeling that with playwright it is a bit faster: running one single robot test of plone.app.discussion takes around 10 seconds in my ~5 years old laptop. Running 4 of them ~20 seconds.

But it would be a nice improvement whenever the tests start to be more robust to look at performance improvements. 👍🏾

Rotonen commented 1 year ago

If the setups are moved around, one thing to keep in mind is that then the assumptions of the tests can be wrong: state builds up between tests on the site from the tests which are not read only.

A clear testing strategy of what to use browser tests for would be advisable. They're cumbersome and fragile and ultimately optimally mostly happen read only without a backend against simulated requests (so could move from whole stack to just testing Volto with the recorded responses from plone.restapi tests - or something in that rough direction).

A quick interim workaround could be to split the read only tests out.

gforcada commented 1 year ago

Hopefully the fragile part is being less and less of a problem with playwright based tests, one of the topics of the Midsummer sprint that is happening this week in Finland, see https://github.com/plone/Products.CMFPlone/issues/3813 for details

Rotonen commented 1 year ago

The fragility I saw back in the day is also on Plone: plenty of frontend race conditions which trigger by clicking too fast. That stuff is way easier to catch with unit tests on the frontend side of things.

I'm way out of the loop these days so hopefully things are better now.

mauritsvanrees commented 1 year ago

Last year I added some keywords in plone.app.robotframework that help against clicking too fast: Wait For Element, Wait For Then Click Element, Wait For Then Click Invisible Element, Wait For Elements. This seems to have gotten the tests a lot more stable, without requiring extra Sleep commands.