We should run browser tests in Github Actions

[jeffgca]

Github actions seem to support puppeteer:

https://github.com/puppeteer/puppeteer/actions

This might be a simple first step towards having a complete-ish set of unit tests that run in current desktop browser runtimes.

@walkah - I think this is the most important thing to address wrt the current state of testing.

[jeffgca]

Down the road a commercial service like browserstack** might make sense but some puppeteer actions to at least test gecko and chromium feel like a good start

** https://www.browserstack.com/

[walkah]

I absolutely agree. I did some experiments with this over in the benchmark repo: https://github.com/fission-suite/benchmark (using Cypress because it's a tool I know well, but same idea). There are a few issues I've run into there:

• The redirect to the authlobby is explicitly rejected (from local) by cypress in firefox and requires a flag in chrome (app owned login wouldn't be subject to this)
• There's not a very good way to establish a fixture for a user in the browser, so currently it creates users on staging for every test run (not ideal). (Alternatively, there's not a good way to persist users between runs)
• With most of the relevant runners, you can manipulate tabs, but to automate tests for something like device linking we'd need multiple browser instances (with a communication channel between - which again gets us into requiring some level of infrastructure).

It's possible we can mock in a lot of this (although at what point are we actually getting valuable tests).

Generally, though, yes I agree - and I'm happy to help make it happen especially if anyone has some proposed solutions to the above!

[expede]

With most of the relevant runners, you can manipulate tabs, but to automate tests for something like device linking we'd need multiple browser instances (with a communication channel between - which again gets us into requiring some level of infrastructure).

Totally, this is a hard thing to test. Are there other flows that we can test and put the complex ones on the back burner (ROI)? Or do you think it's worth investing in the more robust system upfront?

[walkah]

I'm happy to test what we can test - and add as we go / figure out solutions (was more just recounting where I got with my experiments last time I touched it).

How do people generally feel about the tests hitting / creating accounts on our infrastructure (staging or otherwise)?

[expede]

E2E tests would be valuable (also will fail unpredictably, but that's also good to know). I'm okay with it hitting our infra, yeah. Prod okay for now; realistically, more likely will switch to staging over time, so start there? Thoughts?

[matheus23]

Just wanted to mention that we do have browser tests, e.g. this: https://github.com/fission-suite/webnative/blob/d69fce99533b312ef94544fbec3f6db7542c9fdb/tests/fs/concurrency.browser.test.ts

We're only running chromium in the github action, not firefox.

From reading the comments on this issue, it seems like what we want instead of these tests is thorough integration tests?

The tests we're currently running in puppeteer are essentially somewhat like unit tests, but running in the browser. And we've moved away from that, because puppeteer makes for a horrible environment for unit tests. But again, I'm all for full integration tests (not unit tests) with something like puppeteer, playwright or browserstack.

[walkah]

Yeah my understanding is more thorough integration / end-to-end tests (rather than in browser unit tests)