Investigate Testing Framework

gashcrumb commented 5 years ago

Investigate possible test frameworks and methods that we can get tests in place to help us avoid regressions and provide us with more confidence when making changes to the codebase.

@seanforyou23 @riccardo-forina FYI

riccardo-forina commented 5 years ago

I stumbled upon this article that compares cypress against testcafe (a project I wasn’t aware of). The fact that cypress supports only chrome is a downer. Perhaps it’s worth doing a comparison of the two projects ourselves. What do you think?

https://medium.com/yld-engineering-blog/evaluating-cypress-and-testcafe-for-end-to-end-testing-fcd0303d2103

Il giorno mer 31 ott 2018 alle 19:13 Stan Lewis notifications@github.com ha scritto:

Investigate possible test frameworks and methods that we can get tests in place to help us avoid regressions and provide us with more confidence when making changes to the codebase.

@seanforyou23 https://github.com/seanforyou23 @riccardo-forina https://github.com/riccardo-forina FYI

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/syndesisio/syndesis-react-poc/issues/1, or mute the thread https://github.com/notifications/unsubscribe-auth/AA6-rCtcPxRmZd7wMtl3p2IvALxZU32Zks5uqehOgaJpZM4YEppu .

seanforyou23 commented 5 years ago

I just learned yesterday that cypress has plans to broaden the cross browser support. So going from Chrome, Chromium, Canary - to also include FF, Edge, IE11, and looking like Mobile Safari and Android as well .

https://docs.cypress.io/guides/references/roadmap.html#Upcoming-Features

Looking like FF will be the first, there's lots of churn here - https://github.com/cypress-io/cypress/issues/1096

Of course, I haven't seen mention of timeline so keeping that in mind. Good to have something to compare Cy with, I'll check that article out and start with Cypress, can compare once I've got the basics down with the first. Thanks for bringing this up!

gashcrumb commented 5 years ago

This kinda long writeup showed up in my feed this morning, though the main thing it raised for me is scoping,

One problem right now is of course, with no contracts it's hard to mock the backend. But then that leaves us with having to do full end to end testing, which QE I think already does. Should our focus then be limited to unit testing sub-components and containers?

seanforyou23 commented 5 years ago

One thing we can do with e2e testing that QE likely doesn't, is test for a11y issues. We did this for patternfly-next and it's proving very helpful.

https://github.com/patternfly/patternfly-next/pull/611 https://github.com/patternfly/patternfly-next/pull/636

I guess this isn't exactly the same as traditional e2e testing, but you get the idea. @gashcrumb you bring up a good point though. I suppose it would be good to sync with the QE team to better understand what their automated test scripts are doing so we don't duplicate efforts. WDYT?

gashcrumb commented 5 years ago

+1

@tplevko what do you think? Would anyone have time next week to kinda give us an overview and discuss?

riccardo-forina commented 5 years ago

Before I forget it, let me write down what I had in mind to address testing in this POC.

Components

Components are all about rendering some HTML based on the input data. They should have no state handling, only event callbacks (eg. an onSave prop that will be fired when the Save button gets clicked). I'd test them using Jest (provided out of the box by CRA) for performance reason. We should be testing for:

check if the data passed in is actually rendered in the HTML. Ideally done through selectors and hasText checks
check if the interactive elements are properly wired. Ideally done by passing down as props properly wired spies.
a11n testing (+1 @seanforyou23).
visual correctness of the component (eg. does it respond well to all the media breakpoints?). Ideally done manually through stories, I wouldn't spend much time trying to automate this because of: check the next point.
nice to have visual regression testing, ideally done through visual diffing against screenshots.

Containers

Here things get more interesting. Containers usually depend on something set up higher above in the app to work, like Contexts or the Router (or the Provider if we were using Redux). If we use Jest, this means that testing a container requires a bit of setup and boilerplate. I think it's still worth doing it since we want to be as thorough as possible testing containers because they hold the juice of the app.

It's a bit difficult to make a generic bullet point for containers, so I'll write some considerations taking the withMonitoredIntegrations container as an example.

So, we don't have to test if the API endpoint is actually called since that will be covered by the tests around the Rest container. We do have to test for edge cases, like checking that we pass to the render prop a valid (empty) data structure in case the API returns an error. Since this container merges the data about integrations with the data about monitoring, we should check that the merging happens correctly. The merge function should be unit testable (it is not right now, the merge logic is embedded in the render function, we should refactor it to be an exported function). We should set up a number of mock scenarios to test for a reasonable amount of possibilities: integrations without monitoring, no integrations with monitoring, integrations with no matching monitoring, no anything...

We should do this exercise for every single component. I wouldn't stretch so far to say that we should adopt a TDD approach, by writing down the specs and the tests first, and then going down writing the container. But I wouldn't mind see us doing it :)

Pages

These are a special kind of containers, but they are strictly tied to the app. I wouldn't bother testing these with Jest, I think something like Cypress (set up with mocks) would be way more useful here. Ideally, we should be testing for:

are redirects, if any, being followed?
is something being rendered? We shouldn't go too deep in this check, just check if some of the data that's in the mock that we know would have been rendered is actually rendered. Eg. in the integrations list page we should check for the number of integration list items, and perhaps the titles.
are links right and pointing where they should?

riccardo-forina commented 5 years ago

I would also mention that I think BDD would be very useful for Syndesis.

We already have stories. We should start from them and add the scenarios we can think of, and then write E2E tests.

There is some overlap with what QA does, but I think it wouldn't hurt us developers if we were involved in this.

gashcrumb commented 5 years ago

Awesome! You're the best, also

I would also mention that I think BDD would be very useful for Syndesis.

For sure!

There is some overlap with what QA does, but I think it wouldn't hurt us developers if we were involved in this.

Absolutely! I'd like to see what they do currently and see if there's anything we can combine efforts with etc if possible.

syndesisio / syndesis-react