Spike: Automation framework for DDEV Desktop Functional Testing

[mafriend]

End goal:

Find an automation framework that best suits functional testing for DDEV Desktop Timebox: 8hours

Questions to answer:

• Do these frameworks be used for both documentation and automation purposes?
• What languages do these frameworks support?
• Is it free or can we get by with only the free version?
• Do the frameworks provide support?

Needs:

• Scalable
• Low entry threshold
  • Anyone can begin contributing easily
• Well supported
• Maintainable

Wants:

• Something that could be used for more than one product
• Low dependency requirements
• Report tooling options

Frameworks to investigate so far:

• pickles doc — i think this is primarily for documentation?
  • http://www.picklesdoc.com/
• jest
  • https://jestjs.io
• cyprus
  • https://www.cypress.io/how-it-works/
• cucumber
  • https://docs.cucumber.io/
• bhat
  • http://docs.behat.org/en/v2.5/guides/1.gherkin.html
• bats -bash testing (for local and hosting)
  • https://github.com/bats-core/bats-core

[dclear]

@mafriend - i slotted this into current sprint. We can rollover, if needed. If you have time to do this and spin up issue for the next sprint, fantastic.

[mafriend]

Hello @cyberswat, @alkymst, and @dclear (and anyone else who is interested in providing feedback!), please review the below for a breakdown of framework specific benefits and drawbacks and let me know which drawbacks we are okay accepting! Spoiler alert: I'm leaning towards cypress with the cucumber processor (my thoughts/opinions are located at the bottom of the PR).

Spike notes Includes notes on all frameworks considered: https://docs.google.com/document/d/1nYYmdY3uesEanju3Xpw_4Hut6TtYlehKWgIZlwGspF4/edit?usp=sharing

Outcome We have two very good candidates for automation frameworks on our hands, each with their benefits and drawbacks. The common benefits between these languages are: they both provide documentation and automation functionality, they are both heavily supported in the community, and they both have lots of documentation on everything from installation to learning the framework.

• Cypress with the cucumber processor - This is an all-in-one tool which means we dont have to manage a test runner and an assertion library separately. That being said, cypress doesnt come with a "documentation" aspect meaning theres no good place to store gherkin like scenarios in the feature files and make them as easy to surface as cucumber does. That being said, I did find a npm package that we can use with cypress to use the gherkin syntax. One thing to watch out for is that this package is relatively new and I'm not sure how well supported it is. Another caveat to consider here is if we want to use the cypress dashboard that we will need to probably pay for a subscription. The dashboard has some very nice tools we can use for debugging, aggregating test results, running through Circle CI (this is the big important thing the dashboard offers), parallelizing tests, and getting full stack traces of errors. A basic subscription is for 3 users and is free, but if we want more than that, we will need to pay anywhere from 99 to 200 dollars per month.

Cypress also tests their own electron app with cypress tests: https://github.com/cypress-io/cypress/tree/develop/packages/desktop-gui/cypress/integration

• Cucumber with JS (or other language) - Cucumber is pretty widely known and supported framework that offers a lot of resources for easy adaption by developers (lots of resources on installing and learning tools to get those who are unfamiliar ramped up). Cucumber is also a great choice bc we have tons of flexibility when it comes to what languages we want to write the implementation steps in (java, js, ruby, go, php, or with specflow). A drawback to consider will be the implementation of selenium and webdriver. The desktop app doesnt currently run in a browser and cucumber is typically used for web browser testing. Knowing that, I was able to find documentation on how to run selenium tests against an electron app. So the initial set up of the automation suite might take a bit more effort, but once thats done and a test can run, we shouldn't experience those painpoints anymore.

My thoughts and feelings:

• I think cypress is going to offer us the most bang for our buck. The benefits seem to outweigh the drawbacks of this framework!
• I started reading up on cypress being used for unit as well as UI tests but couldnt find a ton of information out there about that. If Cypress could be used for both types of tests, that would be awesome and offer us a single-tool option for two different types of tests!
• Personally, I've never used cypress but am not opposed to learning something new! I would just need some time to ramp up on the framework and refresh my js knowledge.
• I do think the dashboard would be helpful but I would like to try out the free plan first to see if its actually something we would want. Though we may need it for circle CI
• My reservations around using cucumber: we may want to consider a proof of concept to make sure we can get the desktop app to open like a browser.
• Personally, I have the most automation experience with cucumber and java.

[cyberswat]

As of this morning, my vote is for the more traditional cucumber approach. Based on the research in this issue it sounds like Cypress has some potential, but looses points for being JavaScript specific and a less robust supporting community around topics like living documentation. Both of those items seem to translate to additional workload on the teams.

[mafriend]

That's a very valid point for using cucumber! We would have the ability to use a different language per project, but share a common framework/practice. Documentation and support around using selenium/webdriver with electron apps is pretty up to date and robust, which is a bonus! While I don't see cucumber fading into the background anytime soon, theres no guarantee about the level of support we could find with the cypress cucumber processor if anything started behaving poorly. I do really like the idea of being able to use one framework for unit, integration, and functional tests because that seems like less work overall, but I agree with you @cyberswat that the framework being language specific and less-robust support around living documentation could be additional work on team members.

I'm also much more familiar with the ins and outs of cucumber and would only need to brush up on whatever language we decided to pair with it 😎 Selenium conf in india just recently had a presentation on using selenium with an electron app! definitely something we could leverage if we decided to go with cucumber and selenium 🙆

[dclear]

Thanks for all of this work, Monica. To be clear, I am not a blocker here -- I trust whatever tool you smart people settle on, is the right choice.

[dclear]

@alkymst : can you review and weigh in? Thanks!

[alkymst]

I'm not sure it's an either or... However, I have the least real world experience with this.

My understanding, is the desktop app and web application for hosting present different requirements when it comes to what we need to test. It's not just functional or unit tests. We need the ability to fill forms, grab DOM elements, inspect there values, etc. As nice as the gherkin format is for functional tests, in reality I don't see it working for UI tests, filling forms, inspecting the DOM, or end-to-end tests to validate the UI. Snapshotting helps, but won't show you the true interactions. Cypress is solving a different problem all together. From their docs:

Cypress is not a general automation framework, nor is it a unit testing framework for your back end services. There are already great tools out there that do that. Rather, we specialize in one thing - creating a great experience while you write end to end tests for your web applications.

I feel like we need both types of tests.

1. Functional: Both testing the CLI responses through the App, that responses from the cli functions are returning the appropriate responses, that projects are getting created, etc. - Cucumber
2. End-toEnd: Actual interface testing - snapshots(if can actually record the session), testing the actual react data stores, performance measuring, rendering/UI issues, fill forms from the app and record responses. - cypress (possibly with Jest?)

The biggest thing I see when it comes to the UI is Cypress, or Jest run in the application loop allowing for a different level of testing that functional test do not really allow for. I get that it being JS limits it to only the app, but only the app has these issues we are trying to address. JS is the only way to actually test the interface as an actual user. As a developer you can run your end-to-end tests without having to trigger a CI process.

In addition, tests are really easy to write for a frontend developer.

Again, this is my lens as a frontend developer, and I'm totally onboard with whatever we use.

[mafriend]

Hey @alkymst !

Can you clarify what you mean when you say I don't see it working for UI tests, filling forms, inspecting the DOM, or end-to-end tests to validate the UI? With selenium webdriver (or something similar, we can use whatever programming language to do things like interact with the DOM, fill in forms in the UI, and validate certain UI elements are rendering.

I do agree with you that just using cucumber won't allow us for all levels of test coverage. Gherkin is intended to represent a very high level of acceptance criteria that help influence the other levels of testing and facilitate end-to-end UI testing. For example: We can take the gherkin from one scenario, automate it directly using a programing language to create the glue steps for the gherkin, and create an end-to-end UI test for that scenario. We will still need to write any necessary unit or integration tests though to make sure everything under the hood is working correctly. The end-to-end test in the above example may pass because the UI is unaffected, but there could still be an error when making an api call or storing something in the database (something that could be covered by a unit or integration test), so for the most complete and robust test coverage we'll need all levels of automated tests present.

Using your definitions above, I think I would categorize the tests thusly:

• End-to-End (how the customer would interact with the app): Testing the CLI responses through the App, that responses from the cli functions are returning the appropriate responses and the UI has changed appropriately and shows the correct state, that projects are getting created, fill forms in the app, interact with the interface a a user would, etc. - Cucumber
• Functional: Snapshots(if can actually record the session), testing the actual react data stores, performance measuring, rendering/UI issues, fill forms from the app and record responses. - cypress (possibly with Jest?)

The idea of this not being an "either/or" choice is an interesting one! I would caution though that we only use one framework for end-to-end tests and one for other levels of testing (like unit or integration level tests). I know both hosting and local have their own unit tests that they use. Hosting has mentioned using bats and local has mentioned wanting to pursue that framework as well. Should we consider using cypress for integration and unit level tests while cucumber takes care of end-to-end tests? To me, that seems like a lot of effort but that's from my perspective as a tester!

As a heads up, you can run cucumber tests locally without kicking off a CI process! This is configured via the editor and can be used against pretty much any environment. It just needs to be specified in the test runner on an individual basis.

[alkymst]

Hey @mafriend!

Can you clarify what you mean when you say I don't see it working for UI tests, filling forms, inspecting the DOM, or end-to-end tests to validate the UI? With selenium webdriver (or something similar, we can use whatever programming language to do things like interact with the DOM, fill in forms in the UI, and validate certain UI elements are rendering.

If this is the case, then that works for me. I was worried about our ability to interact with the DOM, but if we can do it with a selenium webdriver than that works.

The idea of this not being an "either/or" choice is an interesting one! I would caution though that we only use one framework for end-to-end tests and one for other levels of testing (like unit or integration level tests).

I agree 100%

Should we consider using cypress for integration and unit level tests while cucumber takes care of end-to-end tests? To me, that seems like a lot of effort but that's from my perspective as a tester!

If we go this way I would say it's the other way around:

• Cypress for end to end testing, or what a user sees
• Cucumber for integration and unit testing

As a heads up, you can run cucumber tests locally without kicking off a CI process! This is configured via the editor and can be used against pretty much any environment. It just needs to be specified in the test runner on an individual basis.

Ok sounds good

[dclear]

Note from planning call: Monica will make the decision / call and we can close this out (spinning up any required issues for next steps).

[mafriend]

Hey folks! I've given this much thought and after playing around and doing some more investigation with cucumber and selenium, I think we should try cypress with the preprocessor and here's my reasoning:

• Cucumber and selenium is really meant for more end-to-end testing of a product with a UI. That being said, I don't think hosting or local will get much use of of this approach beyond capturing the gherkin and using it to influence future testing and UI will be gaining powerful end-to-end testing tools, but not much else. For super solid automation coverage, we should make sure the whole test pyramid (or ice cream cone or pie or what have you!) is being considered and that leads me to point two --
• Cypress does allow for more variation in the types of tests you are writing meaning we could start writing unit, integration/functional, and end-to-end tests with the same language and tooling. Later on, if we are not fully satisfied writing unit test with cypress, other people have included jest in their automation tooling to supplement any additional needs for unit.
• A big downside to both cypress is and cucumber w/selenium is that it does not allow for CLI testing which hosting and local need the most. Unfortunately, I don't think we'll be able to share tooling right now between the three projects due to their functionality. Knowing that hosting will someday have a UI similar to local, the two UI projects will be able to share the automation tooling, which is cool! Hosting is already using bats and the local team has talked about starting to use that framework as well. That could be a good option for both teams given that we already have some familiarity with the framework within the company and we can help each other learn it's in's and out's and build up test suites.
• Holy bananas setting up cucumber and selenium webdriver can be a pain in the butt. Even just getting a demo set up for badcamp to run took way longer than expected. I quickly reremembered how frustrating it can be to attempt to implement things only to find out one version of a dependency needs to be updated but then once its updated, its no longer supported by another dependency.

I'm open to talk about this decision if others have questions (about testing, my thoughts, whatever!). Thank, folks!

[dclear]

this is completed; closing. Implementation will be handled under relevant issues / KRs.