search

plone / plone.app.robotframework

Provides Robot Framework compatible resources and tools for writing functional Selenium tests (including acceptance tests) for Plone CMS and its add-ons.
https://docs.plone.org/external/plone.app.robotframework/docs/source/index.html
11 stars 17 forks source link

Try out robotframework-seleniumtestability for robot test stability #138

Closed petschki closed 2 years ago

mister-roboto commented 2 years ago

@petschki thanks for creating this Pull Request and helping to improve Plone!

TL;DR: Finish pushing changes, pass all other checks, then paste a comment:

@jenkins-plone-org please run jobs

To ensure that these changes do not break other parts of Plone, the Plone test suite matrix needs to pass, but it takes 30-60 min. Other CI checks are usually much faster and the Plone Jenkins resources are limited, so when done pushing changes and all other checks pass either start all Jenkins PR jobs yourself, or simply add the comment above in this PR to start all the jobs automatically.

Happy hacking!

petschki commented 2 years ago

@jenkins-plone-org please run jobs

petschki commented 2 years ago

didn't see #137 ... sorry ...

datakurre commented 2 years ago

Did it work? I saw JS errors in geckodriver log and did not resolve that yet.

petschki commented 2 years ago

buildout broke because of unpinned packages ... need to make a PR in buildout.coredev first ...

mister-roboto commented 2 years ago

@petschki thanks for creating this Pull Request and helping to improve Plone!

TL;DR: Finish pushing changes, pass all other checks, then paste a comment:

@jenkins-plone-org please run jobs

To ensure that these changes do not break other parts of Plone, the Plone test suite matrix needs to pass, but it takes 30-60 min. Other CI checks are usually much faster and the Plone Jenkins resources are limited, so when done pushing changes and all other checks pass either start all Jenkins PR jobs yourself, or simply add the comment above in this PR to start all the jobs automatically.

Happy hacking!

petschki commented 2 years ago

@jenkins-plone-org please run jobs

petschki commented 2 years ago

I also see many warnings about javascript errors, but the tests are getting green:

[ WARN ] Message: javascript error: Cannot read properties of undefined (reading 'when')
JavaScript stack:
TypeError: Cannot read properties of undefined (reading 'when')
    at eval (eval at executeAsyncScript (:552:26), <anonymous>:5:28)
    at eval (eval at executeAsyncScript (:552:26), <anonymous>:8:8)
    at executeAsyncScript (<anonymous>:552:47)
    at <anonymous>:567:29
    at callFunction (<anonymous>:446:22)
    at <anonymous>:460:23
    at <anonymous>:461:3
  (Session info: headless chrome=101.0.4951.64)
fredvd commented 2 years ago

Also for @mauritsvanrees

Interesting to see that all 3 pr jobs succeeded, we have patched the 3.8 job locally on jenkins with some tweaks this week but that doesn't seem to change the number of broken tests in this run.

I've started a separate run for only python 3.8 on the other PR, didn't see this PR until 15 minutes ago, it should give the same results.

I suggest we merge 137 after reverting the last commit and we update jenkins this afternoon with the other improvements.

Then we can update the other jenkins jobs through https://github.com/plone/jenkins.plone.org and a jenkins-job-builder run to add headlesschrome and PYTHONWARNINGS=ignore

If new robot test flukes/failures pop up we can then enable testability individually on those tests by adding the variable like https://github.com/plone/plone.schemaeditor/pull/91/files

petschki commented 2 years ago

so despite of JS errors, the tests are all green ... maybe we could even revert those extra wait keywords implemented here https://github.com/plone/plone.app.robotframework/pull/136 in favor of this plugin ...

fredvd commented 2 years ago

so despite of JS errors, the tests are all green ... maybe we could even revert those extra wait keywords implemented here #136 in favor of this plugin ...

The keywords Maurits added as a convience were already added in quite some places in 2021 when we had to fix many robot tests for the bootstrap 5 updates. I don't get the impression they slow down the current tests a lot it's not literal sleep commands, but working around scroll into viewport bugs in selenium2. (and other edge cases).

To make ourselves dependable on yet another .0.0 plugin that doesn't seem too stable and acts like a magic wand... Lets first wait for a flakey test where we can verify it does something/anything over multiple runs.

petschki commented 2 years ago

Fair enough ... I'll close this one as its only an experiment