Investigate other e2e test engines

[antgamdia]

Due to the number of the false-positive CI errors we usually get, it could be worthwhile to look at other test engines, like Michael's suggestion [1] , instead of continuing using Puppeteer. Besides, the API is pretty similar, and the change appears to be straightforward [2]

[1] https://playwright.dev/ [2] https://medium.com/@davert/puppeteer-to-playwright-migration-guide-6c86ea66e85e

Other options to consider:

[3] https://www.cypress.io/ [4] https://agouti.org/ - For a go-based integration test experience, together with ginko etc.

[absoludity]

https://agouti.org/ also looks very interesting (found from ginko website).

[castelblanque]

Cypress results so far

Initially looked very good, super easy to code steps for testing. Less verbose than current tests. Development env is really intuitive and fast.

A test for a login to Kubeapps looks as simple as this:

describe("Kubeapps CI auth", () => {
  it("Test UI login", () => {
    const hostname = Cypress.env('INTEGRATION_ENTRYPOINT');
    cy.visit(hostname);

    cy.get('cds-button').contains("Login via OIDC Provider").click();

    cy.get('.dex-container button').contains("Log in with Email").click();

    cy.get('input[id="login"]').type("kubeapps-user@example.com").should('have.value', "kubeapps-user@example.com");
    cy.get('input[id="password"]').type("password").should('have.value', "password");

    cy.get('#submit-login').contains("Login").click();

    cy.get('.dex-container button[type="submit"]').contains("Grant Access").click();
    // Breaks at this point with Kubeapps throwing a 403
  });
});

But Cypress uses iframes for performing the tests, and that breaks OIDC flow because we are changing hosts (Kubeapps -> Dex -> Kubeapps). Then the CSP directive is applied to the iframe, and Kubeapps shows a 403 when calling back. It is required to disable the chromeWebSecurity when running tests, but that doesn't solve the CSP issue.

No luck so far on completing a login flow. Tried with obtaining token through REST with DEX, no luck.

[castelblanque]

Playwright results so far

Syntax is similar to Pupeteer. Really easy to set up and build tests.

Pros

• Didn't need to adapt anything in source code of Kubeapps in order to make the tests work (e.g. adding special CSS selectors, etc.)
• Multiple browsers can be easily tested (chromium, webkit, firefox)
• Strictly checks for a single element to be found, not picking up the first one if multiple. E.g. selector .kubeapps-nav-link worked with old tests, whereas fails in Playwright due to returning two elements. On one side this is positive because it helps to avoids selecting the wrong element.
• Videos can be recorded!

Cons

• It seems that Playwright shares stuff with Puppeteer, so similar problems could appear.
• No big cons found so far

Built a POC of 4 tests in this branch. Work is in progress.

[castelblanque]

Finally Playwright has been chosen and E2E tests are implemented with it in #4185. It is yet unclear if this will be a final decision or e.g. Cypress will be a requirement in the near future.