Better access to playwright API

[floo51]

Problem

No playwright configuration

Can't pass options to browser or context, like ignoreHTTPErrors, locale, timezone. Every time we want to support a playwright feature, we need to implement a new flag and publish a new version

Can't reuse existing playwright tests

If I'm already using playwright as my e2e test framework and have some actual e2e tests, I would like to be able to use the same files and not have to maintain the same test for playwright, and for greenframe

Can't use page objects

Since I can't import local files in my greenframe tests, I can't implement the page object pattern on my scenarios. It's really a pity because this pattern is very common and makes tests so much more readable.

VM2 isolation is not needed in the CLI

We don't need VM2 isolation when executing tests locally, which is what greenframe-cli is doing. VM2 makes it impossible to use imports and such.

Distant mode still needs to work

For now, it only sends the scenario files set in the configuration file. But if we want to be able to plug greenframe to any pre existing test suite, we need to be able to have the full code, imported modules too.

Solution for local open and analyze

• Remove VM2
• Execute playwright programmatically directly from the CLI code, with all the context of the project we're in. (playwright can't be executed programmatically, we'll have to do a bash exec with npx run playwright)
• Add a greenframe init that proxies playwright init ?

This will break the functions we added to playwright page object. We should probably have a @greenframe/playwright that overrides the page object of playwright to add waitForNetworkIdle, scrollToElement, scrollByDistance, scrollToEnd, addMilestone, getMilestones.

Solution for distant mode

In order for us to execute distant playwright tests, we need to...

• resolve all local imports
• allow people to import "@playwright/test"
• allow people to use pure playwright test

My guess is that, if we transpile the playwright test locally before sending to greenframe servers, it should include local imports (and maybe even playwright and all other libs or playwright extensions used) We could even support typescript if we do that.

The analyzes running on greenframe servers should still use VM2 and disallow all external or node-builtin(fs,fetch...) module access.

We can probably do a cheaper solution by disallowing any import other than @playwright/test and @greenframe/playwright in the tests files. It should be good enough for people wanting distant analyzes.

Breaking changes

All previously written scenarios won't work anymore. They will need a few changes:

• import @playwright/test
• import @greenframe/playwright if any GF functions were used
• wrap the function in test()

This can probably be codmodded

[fzaninotto]

IMO, we can solve that problem in a backwards compatible way:

• in local mode, we accept two types of scenarios:
  • the ones that contain an import statement, which will be executed directly
  • the ones that don't, which will be wrapped with vm2
• in distant mode, we change nothing. We keep vm2, and the scenarios must be wrapped.

I don't think we should rewrite the local scenarios to make them work in distant mode, as there are many cases where this will break (e.g. import libraries that aren't installed, etc) and we won't be able to prevent hacks (where a nasty user writes a scenario designed to explore the distant server).

I'm not sure I understand that point:

Execute playwright programmatically directly from the CLI code, with all the context of the project we're in. (playwright can't be executed programmatically, we'll have to do a bash exec with npx run playwright)

What do you mean playwright can't be executed programmatically?

I see one additional problem: if we use pure playwright tests, the URL parameter of the open and analyze commands is useless - and that probably makes the CLI syntax not adapted.