Proposal to get license for service called "AWS Device Farm"

[armstro]

John mentioned he can look into if any other LF project using such a service, Zowe team may piggyback on licenses bought for other project, and run Zowe cross browser test there @nkocsis @NakulManchanda

[jmertic]

@pono Can you review and see how to best integrate into the existing CI/CD environment?

[armstro]

Apologies - I messed up assignees and tried to fix

[jmertic]

Howdy!

So digging more into this after talking to our RE team - could we just run Selenium headless on the instance instead? Probably would be a bunch easier and remove the need for storing and managing additional credentials and services.

[NakulManchanda]

Hi @jmertic , As mentioned on OMP slack - zlc channel by @1000TurquoisePogs , we need a way to perform cross browser testing for four common browsers - firefox, chrome, safari, edge, maybe IE , and across different OS versions as well (Mac, Windows, Linux)

Linux we can use docker image, but even to run headless selenium to test safari, edge or IE for windows, mac we need host machines - they have different versions too - windows 10,7 , mac os catalina, mojave.

So its either we try to setup host machines manually and keep them upto date year after year, or use service like AWS Device farm, where we might need to manage only 1 license, and no other admin tasks needed. And this license can be shared with other projects depending on workload and licensing policy.

[jackjia-ibm]

@jmertic, a little background about what we have done with Open Source tools on browser testing.

Currently we are using Selenium headless testing with Gecko Driver, and the test cases are running daily against nightly builds. We also did try out Chrome Driver manually which works well. And I think we can test against multiple versions on Firefox/Chrome with some improvement. Those tests as Nakul mentioned are running in Ubuntu docker containers.

What we are trying to find out is how to test wider browser support we are claiming, including Safari/Edge/IE. Those browsers don't have headless support so far as I know.

[jmertic]

We can spin up a Windows Server instance in the existing LF RE infrastructure, which should open up testing for IE and Edge.

That does leave the gap of Safari - but I would think with Chrome and Safari both webkit based there's likely minimal gaps.

[1000TurquoisePogs]

Unfortunately Safari has trailed Chrome, such that in the past we've needed to pay special attention to it since things would not work quite right - Differences in CSS behavior being the hardest to plan for, but sometimes we've also been surprised with it missing JS features. I think it is getting better, but it's still not a drop-in replacement between the two.

[jmertic]

FWIW - AWS Device Farm doesn't help the Safari issue...

https://docs.aws.amazon.com/devicefarm/latest/testgrid/techref-support.html

[jackjia-ibm]

FYI - a support ticket https://jira.linuxfoundation.org/servicedesk/customer/portal/2/IT-19020 is created to help LF team track the progress.

[NakulManchanda]

Would be great if we can close ticket with LF,
We need a cross browser testing solution from them. Rocket current setup is using selenium grid, with mac and windows node, We can start with only windows node for now.

@jmertic @armstro , would appreciate if we can follow up with LF, opened this couple of months back after meeting between LF, bruce and @jackjia-ibm , they didn't propose any specific solution, Thanks!

Because not everybody has access to LF ticket, posting screen capture i got from Jack here:

[armstro]

@NakulManchanda @jackjia-ibm We discussed briefly at the ZLC today - I see latest date is Feb 18 - if Jack could please be clear the "ask" is for selenium grid we can see if OMP can accommodate.

[jackjia-ibm]

Thanks @armstro, I updated the ticket to address if Selenium Grid is the right option OMP can accommodate.

[jmertic]

Something else I came across that might be useful...

https://percy.io/

They have a free tier targeted at open source projects. Only supports Chrome/FF right now, but the visual review seems to be something that could hit what you are after.

[NakulManchanda]

Hi @jmertic ,

Purpose of pecy.io > Integrate, run, and review visual tests. Looked into it a little bit, this is to run visual test only, visual tests are additional tests we can write in our test suite

Purpose of Selenium But to drive browser automation and to get to different screen, and perform user actions in browser we still need Selenium. Its substantial effort to write these tests, and we already have tests written in ZLUX & Explorer projects in selenium, just looking to open source them with similar infrastructure.

Purpose of Selenium Grid Selenium grid is a smart proxy server that allows Selenium tests to route commands to remote web browser instances. Its aim is to provide an easy way to run tests in parallel on multiple machines. Rocket internally is using selnium grid as well.

There are no licensing requirements for downloading installing any of above stuff.

I think LF foundation ticket is enough to track this one now.

As original proposal of AWS device farm, was helping us to get our hands on different remote machines with different OS and browser versions without rolling our own, as whole idea is to connect to them using selenium grid. The only purpose for these remote machines to act as nodes to run our tests, and nothing else. So, they can die down, after we run our test using selenium grid.

[jmertic]

Got it - that's quite helpful context. Thanks!

[PeterFandelAtRocket]

Investigate another solution here: https://dev.to/mizchi/run-crossbrowser-e2e-testing-on-github-actions-671

[pdubz3]

Still gaps here but not a significant issue and will revisit if this becomes a problem.