search

plone / Products.CMFPlone

The core of the Plone content management system
https://plone.org
GNU General Public License v2.0
248 stars 188 forks source link

Port robot framework tests to Playwright #3813

Open gforcada opened 1 year ago

gforcada commented 1 year ago

STATUS

Report from Midsummer sprint 2023: playwright integration to plone.app.robotframework (via robotframework-browser) works! 🎉

All credits to @datakurre ✨

And even better, it not only works, but we don't have to have a sync day where all tests need to be converted at the same time, but rather we can move them incrementally!

TASKS

There are a plenty of things to do still, all arms and fingers are welcome! there are work items for everyone:

Please write your name next to a package if you want to work on porting them! 🙇🏾

PORTING TESTS

IMPORTANT NOTE: when porting tests from selenium to playwright, they have to be ported all tests from the same distribution in a single go, otherwise either the ported or the old ones will fail on CI.

Probably this is due to them sharing the same layer, if we keep the old test layer and create a new one for the playwright ones, all tests might pass.

RUN TESTS

To run tests locally:

DEBUGGING TESTS

By default playwright does not show the browser, as selenium used to do.

If you want to follow along though, add another environment variable ROBOT_DEBUG=true ✨

It will automatically stop (when using ROBOT_DEBUG) if a keyword fails 🤩 note though that if the keyword itself is wrong (an old selenium keyword) then it does not stop, it breaks the test and stops 😅

And as usual with @datakurre there is even more: VSCode integration right at your fingertips, run robot tests from within VSCode and run tests step by step 🤯

KEYWORDS STATUS

Notice that plenty of keywords that we were using for robot tests are no longer needed (the whole collection of Wait until... 💥 ) as playwright does implicit waits already 🚀

As soon as we have them running in CI we can see if we need to add them back, hopefully not 🍀 🤞🏾

robotframework-browser being much more younger than selenium has a smaller set of keywords available, see the p.a.discussion PR to get an idea on which replacements one can use.

On plone.app.robotframework we created quite a few custom ones. We either need to re-implement them with playwright based keywords, or drop them altogether, we need to see.

Exciting times! 🎉

datakurre commented 1 year ago

ROBOT_TRACE=true will generate file named tracing deep under parts/tests/..., which could then be opened with bin/rfbrowser -F path/to/tracing show-trace for Playwright trace viewer.

These flags are defined in https://github.com/plone/plone.app.robotframework/blob/e5d98a98483729231a23cb5007e56b386e7001a6/src/plone/app/robotframework/browser.robot#L43 and are just the first way to integrate these Playwright and robotframework-browser features that came to mind. (robotsuite does the magic for passing any ROBOT_-prefixed environment variable into robot run as a robot variable)

gforcada commented 12 months ago

@thet @petschki I see that on plone.app.event there is a disabled robot test.

On my way from Bilbao to Barcelona with train I was looking at it, and it's a bit tricky to get it to work again, given that the date widget is browser dependent... should we try it anyway, or should it be removed entirely? 🤔

stevepiercy commented 8 months ago

For those following this issue, I created another issue in plone/documentation where I would like to leverage your work for Plone 6 Documentation for a Google Season of Docs proposal that I am working on.

https://github.com/plone/documentation/issues/1611

1letter commented 2 weeks ago

@datakurre @gforcada what are the next steps if the tests are ported?

plone.themepreview is not core, is here the port really needed?

datakurre commented 2 weeks ago

plone.themepreview should be archived; I don't think it has been really used since very early Plone 5.It was used to "capture theme" with different devices through Sauce Labs, which uses Selenium.

mauritsvanrees commented 2 weeks ago

Please remind me: is './bin/rfbrowser init' only needed for playwright? Or can it also be useful for the old selenium based tests?

1letter commented 2 weeks ago

Please remind me: is './bin/rfbrowser init' only needed for playwright? Or can it also be useful for the old selenium based tests?

I think it's only needed for playwright based tests.

datakurre commented 2 weeks ago

Yes, it is only needed for playwright tests. Playwright is distributed through NPM and that command should imperatively install the required packages. Probably it also fetches the playwright compatible browser binaries.

mauritsvanrees commented 2 weeks ago

I am testing https://github.com/plone/plone.schemaeditor/pull/118.

I have a problem on my Mac: bin/test hangs on the second test.

On buildout.coredev branch 6.1 I checked out plone.schemaeditor, did bin/rfbrowser init. With bin/test --all -s plone.schemaeditor, the tests hang on the second test.

I tried each test scenario individually with for example bin/test --all -s plone.schemaeditor -t "Add a content type" and they all worked fine.

But running two together leads to a hang:

$ bin/test --all -s plone.schemaeditor -t "Add a content type" -t "Delete field"
...
  Set up plone.schemaeditor.testing.ROBOT in 0.011 seconds.
  Set up plone.app.robotframework.remote.AutoLoginRemoteLibrary:RobotRemote in 0.000 seconds.
  Set up plone.testing.zope.WSGIServer in 0.038 seconds.
  Set up plone.schemaeditor.testing.ACCEPTANCE in 0.000 seconds.
  Running:
    1/2 (50.0%)/Users/maurits/shared-eggs/cp312/protobuf-4.24.4-py3.12.egg/google/protobuf/internal/well_known_types.py:93: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
  _EPOCH_DATETIME_NAIVE = datetime.datetime.utcfromtimestamp(0)
/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/playwright.py:57: ResourceWarning: unclosed file <_io.TextIOWrapper name='/Users/maurits/community/plone-coredev/6.1/parts/test/test_fields/Scenario_Add_a_content_type/playwright-log.txt' mode='w' encoding='UTF-8'>
  process = self.start_playwright()
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/maurits/shared-eggs/cp312/grpcio-1.59.0-py3.12-macosx-14.2-x86_64.egg/grpc/_channel.py:1125: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  state.rpc_start_time = datetime.utcnow()
/Users/maurits/shared-eggs/cp312/grpcio-1.59.0-py3.12-macosx-14.2-x86_64.egg/grpc/_channel.py:226: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
  state.rpc_end_time = datetime.utcnow()
<frozen importlib._bootstrap>:530: DeprecationWarning: the load_module() method is deprecated and slated for removal in Python 3.12; use exec_module() instead
    2/2 (100.0%)

There it does nothing anymore.

I see a playwright log generated for the first test:

$ less .../parts/test/test_fields/Scenario_Add_a_content_type/playwright-log.txt
...
{"level":30,"time":"2024-09-16T10:56:32.535Z","pid":64385,"hostname":"MauBook19.local","msg":"Start of node method closeBrowser"}
{"level":30,"time":"2024-09-16T10:56:32.562Z","pid":64385,"hostname":"MauBook19.local","msg":"Removed page=8e96b647-b55f-43cc-bdab-d205d5b7d12c from context=4ce2a22d-b31b-44d0-ae80-89401eaad600 page stack"}
{"level":30,"time":"2024-09-16T10:56:32.935Z","pid":64385,"hostname":"MauBook19.local","msg":"End of node method closeBrowser"}
================= Original suppressed error =================
Error: Browser has been closed.
    at PlaywrightState.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9470:15)
    at PlaywrightServer.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9991:59)
    at PlaywrightServer.setTimeout (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:10237:57)
    at handleUnary (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/node_modules/@grpc/grpc-js/build/src/server.js:852:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
=============================================================
{"level":30,"time":"2024-09-16T10:56:32.944Z","pid":64385,"hostname":"MauBook19.local","msg":"Start of node method getBrowserCatalog"}
{"level":30,"time":"2024-09-16T10:56:32.944Z","pid":64385,"hostname":"MauBook19.local","msg":"End of node method getBrowserCatalog"}
================= Original suppressed error =================
Error: Browser has been closed.
    at PlaywrightState.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9470:15)
    at PlaywrightServer.getActiveBrowser (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:9991:59)
    at PlaywrightServer.setTimeout (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/index.js:10237:57)
    at handleUnary (/Users/maurits/shared-eggs/cp312/robotframework_browser-17.5.2-py3.12.egg/Browser/wrapper/node_modules/@grpc/grpc-js/build/src/server.js:852:17)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
=============================================================
{"level":30,"time":"2024-09-16T10:56:32.950Z","pid":64385,"hostname":"MauBook19.local","msg":"Start of node method getBrowserCatalog"}
{"level":30,"time":"2024-09-16T10:56:32.950Z","pid":64385,"hostname":"MauBook19.local","msg":"End of node method getBrowserCatalog"}

That sounds like teardown does not work correctly, and the second test cannot start. For the second test I only see an empty output.xml in the parts/test directory.

I have export ROBOT_BROWSER=headlesschrome in my bash profile, but see the same problem with ROBOT_BROWSER=chrome.

Any idea?

petschki commented 2 weeks ago

@mauritsvanrees I got this working on my Mac with the following version set:

robotframework-browser = 18.8.1
robotframework-assertion-engine = 3.0.3
robotframework-pythonlibcore = 4.4.1
grpcio-tools = 1.66.1

those versions are pinned here https://github.com/plone/buildout.coredev/blob/6.1/versions.cfg#L236-L239

NOTE: don't forget to rfbrowser init after upgrading those eggs

UPDATE: these packages can be updated too

robotframework-debuglibrary = 2.5.0
prompt-toolkit = 3.0.47
Rotonen commented 2 weeks ago

Tangent: to make test runs platform independent (same locally as in pipeline), it could be prudent to provide devcontainers.

https://containers.dev/

mauritsvanrees commented 2 weeks ago

Thanks @petschki, with those pins it works for me. I have merged your coredev PR.

Could you check if those versions work on 6.0 as well?

mauritsvanrees commented 2 weeks ago

@gforcada The Jenkins nodes may need some upgrade. "Host system is missing dependencies to run browsers." See https://github.com/plone/buildout.coredev/pull/948#issuecomment-2354981479

gforcada commented 2 weeks ago

Sorry for the delay on replies, I'm rather busy as of late, hopefully in 2 weeks the big project I work on will be deployed (and the more bugs and features will be asked 😆 ) and I will have some time 🤞🏾

Ok, I found some time, I bumped nodejs to v20.5.1

gforcada commented 2 weeks ago

Turns out that 20.5.1 is not enough 🙃

petschki commented 2 weeks ago

Log says

required: { node: '^18.18.0 || ^20.9.0 || >=21.1.0' }

petschki commented 2 weeks ago

And chromedriver seems to be outdated too. See https://jenkins.plone.org/job/pull-request-6.1-3.12/614/consoleFull ...

gforcada commented 2 weeks ago

So, finally, in a sleepless night I found the time to look at the problem 🎉

I installed nvm with node version 22, and enabled nvm to be reachable on jenkins jobs.

The long list of dependencies can be installed (if anyone hits the same problem) with npx playwright install-deps ✨

Oh, and indeed @petschki chromedriver was again outdated, fortunately that's easier to fix (and it is 😄 )

/cc @mauritsvanrees @1letter (I'm not sure in which other PR/issue we were also discussing CI problems of robot tests 😓 )

Seems that robot tests run again: https://jenkins.plone.org/job/plone-6.1-python-3.12-robot-chrome/561/console

mauritsvanrees commented 2 weeks ago

Thanks a lot! And I hope you sleep better next night. 😴💫

1letter commented 2 weeks ago

@gforcada Sorry, but an error happen on install rfbrowser

https://jenkins.plone.org/job/pull-request-6.1-3.10/360/console

+ bin/rfbrowser init
2024-09-19 04:55:01,402 [INFO    ] ==============================================================================================================
2024-09-19 04:55:01,402 [INFO    ] Installing node dependencies...
2024-09-19 04:55:01,403 [INFO    ] Couldn't execute npm. Please ensure you have node.js and npm installed and in PATH.See https://nodejs.org/ for documentation
2024-09-19 04:55:01,403 [INFO    ] ==============================================================================================================
2024-09-19 04:55:01,403 [INFO    ] Traceback (most recent call last):
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 382, in init
    _rfbrowser_init(skip_browsers, silent_mode, with_deps, browser)
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 155, in _rfbrowser_init
    _check_npm()
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 147, in _check_npm
    raise exception
  File "/home/jenkins/.buildout/eggs/cp310/robotframework_browser-18.8.1-py3.10.egg/Browser/entry/__main__.py", line 135, in _check_npm
    subprocess.run(
  File "/srv/python3.10/lib/python3.10/subprocess.py", line 503, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/srv/python3.10/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/srv/python3.10/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'npm'