Functionality Testing

[loganpowell]

It seems that some of the apps aren't actually functioning properly. The only one I've had complete success with is the Keechma app so far:

Quick summary of non-functional features

Framework	Routing	Tagging	Article	Profile	Other
Angular	home (logo)	article tag nav			Images broken
Angularjs		article tag nav
Redux	home (logo)	article tag nav			No "favorite"
Re-frame		article edit, tag nav
Keechma					Can't delete comments
Vue		article tag nav
Mobx		article tag nav
Svelte		article tag nav	Publish sticks, no favorite
Apprun		tag nav persists over feed		Profile Update
Dojo2	home (link)
Crizmas		article tag nav
Aurelia		article edit, display, tag nav	profile links		Whole app locks up with some clicking around pages
Elm		article edit, tag nav

I think there should be some kind of user acceptance testing for these if they're supposed to compare apples to apples. I may have missed some, ran out of time to really dig into them all

[Cameron-C-Chapman]

This is an awesome break down @loganpowell! Thanks for taking the time.

This does really point out an issue with our front end validation. For the most part we follow the instructions in the projects readme and make sure those work from scratch and the app runs. Then we sort of smoke test it for some basic functionality.

I've known this is a gap on the front end projects for awhile. This isn't nearly as big an issue with the backend implementations because there is an automated test suite we can run. I would love to have something like cypress setup to run on the front end implementations. The challenge I ran into when I was thinking that through was that we'd have to build out a spec that required a certain DOM structure or certain id's/classes and that would probably be a pretty significant effort.

Do you have any other thoughts on how we might improve this? Automated testing, better specs, etc. I'd certainly be on board getting improvements around this rolling.

[loganpowell]

It could be as simple as some user acceptance tests. Just a series of tasks for a human to run through to ensure all front-end functionality operates as expected. It would only take one person about an hour to run through what I did to test all the current apps, but I'm not sure if I covered everything. I figured I'd point out some hairy elbows that I found at least as a starting point for such tests if they seem valuable to your effort.

[anishkny]

How do we ensure manual acceptance tests get run regularly/on every commit?

I took a crack at writing Puppeteer based E2E tests for the React/Redux Frontend (against NodeJS Backend). This could be used as a starting point for more generic acceptance test bed.

[calaway]

Hello @Cameron-C-Chapman and @loganpowell!

I came to this repo because I was interested in building my own private Real World App(s) as an educational tool to learn the ins and outs of different languages and frameworks. I figured I would start by pulling down the end-to-end tests from this project to use as acceptance criteria for my own project, but I was surprised to find that they don't exist.

Cameron, I see that you marked this issue "help wanted" back in 2018. Is that still the case now? If so, I would be interested in building and contributing an end-to-end acceptance test suite for the project. If I did would I have the support of the community?

Assuming the answers to the above questions are yes, let's discuss some details.

I agree with you that I think Cypress is the right tool for the job. Regarding this concern you stated above:

The challenge I ran into when I was thinking that through was that we'd have to build out a spec that required a certain DOM structure or certain id's/classes and that would probably be a pretty significant effort.

I'm hopeful that this won't necessarily be a problem. If we follow the best practices for querying the DOM as set forth by the popular Testing Library, our "test should resemble how users interact with the code (component, page, etc.) as much as possible." As per Testing Library's creator, Kent C. Dodds, this means making your UI tests resilient to change by not testing implementation details. Is it okay to require standard verbiage on the UI, such as on headers or input placeholder text? If so, then I believe we can build tests that are (mostly) agnostic to implementation details.

Here are a list of questions I can think of, in loose order of importance regarding what information I would need to proceed.

1. As asked above, since this issue is over two years old, can you please confirm that an end-to-end acceptance test suite is still desired?
2. Would this have the support of the community, or at least the primary maintainers? I would hate to build something and then see a big backlash to it.
3. Is my plan to use Cypress acceptable?
4. Is my plan to query the DOM without testing implementation details acceptable? Is it okay to have a spec that requires specific verbiage on the page to query elements by?
5. In which repo should this test suite live? Does it belong in the same repo as the backend API test suite? If so, where is that?
6. Which of all the existing Conduit apps should be used as the prime example that I can start writing the tests against? I'm assuming https://demo.realworld.io, but please confirm that.
7. Is there already a pre-defined spec to use as acceptance criteria? I would need that in order to know exactly what tests to write. If not, who would be able to help me define that?
8. Would we want to implement this test suite in a CI pipeline for each of the different Conduit implementations? If so, I'm sure we can work out the details later. I just want to get a conversation started about it early.

I would plan to keep this project agile, submitting small PRs and keeping tight feedback loops. So if we don't have answers to all of these questions now, that's okay, just so long as I have enough information to get started.

Thank you in advance for providing advice and direction on this. I look forward to getting started!

[Cameron-C-Chapman]

Hi @calaway 👋

I've been pretty out of the loop on the realworld project for awhile, but I always thought the idea of a front end test suite that could be ran similar to the way the api tests work would be really beneficial. Not only for people using this repo as a learning tool, but also for the maintainers to be able to validate projects as they review them and for the developers to be able to confirm expected behavior.

@anishkny has taken over the maintenance of this project so any official decisions would need to go through him, but I don't see any reason you couldn't put a prototype together using one of the projects and everyone can think through how to apply it generically to all of the frontends.

[calaway]

Thank you so much for your reply, @Cameron-C-Chapman, and thank you for referring me to @anishkny.

@anishkny, would you be able to work with me to answer questions and provide guidance on this project? If not, is there someone else who is willing and able that you could point me to?

I went ahead and started a new repo for this: https://github.com/calaway/real-world-app-e2e-test-suite. Currently I have purposefully built just one test, which simply validates the header. That should be enough to take a look at my methodology and choice of tooling to either provide any feedback or give me the thumbs up to go ahead with more tests. The readme provides instructions if you want to pull it down and run it.

I still need answers to the questions I asked in my last post, but some aren't as pressing as others. The top thing I could use help on to proceed from here is question 7, what is the list of acceptance tests that we would want written?

Here's an example of my ideal test description in "given, when, then" format.

Sign in spec:

GIVEN
As an existing user

WHEN
When I visit https://demo.realworld.io
And I click "Sign in" in the nav bar
And I fill in "Email" with "test_user_1@mailinator.com"
And I fill in "Password" with "Passw0rd1"
And I click "Sign in"

THEN
Then I land on the route "/#/"
And I see a tab titled "Your Feed"

Let me know what you think of this. I look forward to working with you and hearing your guidance on how to proceed.

Thank you so much for your dedication to this valuable open source project! I really appreciate your help!

[Cameron-C-Chapman]

@calaway I think cypress is a perfect tool for this sort of testing and your setup looks great!

I know you're just building and testing it out at this point, but the only thing I notice is that this should be written with the idea that developers would use this to test their local apps, not pointing against the deployed realworld demo app. So the instructions would include what configurations to change to point to the right local url, etc.

Really with an e2e test suite like cypress, we could replace the api tests and eventually provide a default front end implementation, a default backend implementation, and allow someone to configure cypress to point at a local front end or backend depending on what they're working on. That might end up being more maintenance but a single e2e suite with swappable configurations could be pretty cool.

We don't have a list of scenarios, so you'd get the first crack at it. I would look at the api tests and the routes involved and mirror it after that to start. Get user, add user, get articles, add article, etc. It should be platform independent, so not per framework type, but just a generic scenario list.

[benjamin-thomas]

Hi,

So I was looking for Cypress tests and got here ;)

I was thinking of using this project to dive into different technologies I'm interested about....

To implement a backend, my plan is to use the current API spec here, as a guide to then implement framework specific tests.

For the UI, it'd be natural for a Cypress suite to fill the same role: validate general UI interactions, then use framework specific tools for learning (tdd style).

It turns out that Cypress has its own real-world app: https://github.com/cypress-io/cypress-realworld-app

However it seems very complex in comparison, so not as a good candidate to investigate and compare technologies. A nice project to refer to though.

Could you give a bit of guidance here ? I feel creating a simple cypress test suite as suggested by @calaway is perfectly reasonable and could be done right now.

Once the usage is established, you could make use of it to validate an implementation. But in the meantime it'd be just a reference for tinkerers. If the initial test suite turned out to be inappropriate, you could delete it and start over if needed.

Are you open to doing that and accepting PRs for it?

[mvila]

@benjamin-thomas Indeed, a test suite for the frontend would be a nice thing to have, but I think we should wait for the v2 specs before implementing it.

[benjamin-thomas]

@mvila I wasn’t aware a v2 spec was planned.

Where can I track this?

[mvila]

@benjamin-thomas The v2 project has not started yet. We still have some work to do before we can talk about it and involve the community.

[Christine-Pinto]

Hi there @mvila , does it make sense to add cypress ui test to v1 or should we wait for v2?

I'm an automation engineer and would like to help with the Cypress setup

[mvila]

Hi @ChrisZie,

I think it's better to wait for v2 which should provide a frontend reference implementation that is designed with testing in mind (e.g., relying on data-* attributes instead of CSS classes).

[Christine-Pinto]

Hi @mvila ,

I could set up a general Cypress test suite and prepare one minimal test to open the page. We could include that in the build or as a GitHub action. Do you build in a docker setup?

[geromegrignon]

Hey @ChrisZie,

I'm currently working on a new v1 version of the frontend demo app for maintenance effort and as a starting point for fix/features for v2. As I already did with the new backend implementation (https://github.com/gothinkster/node-express-prisma-v1-official-app/blob/main/.github/workflows/ci.yaml), I plan to use a GitHub Action for tests.

About tasks related to this request, as @benjamin-thomas and @calaway have already shown interest, I want to make sure no one is left behind.

If it's fine with all three of you, I can create a repository and give you a maintainer role so you can coordinate there to decide how to setup Cypress and create the test suites.

I don't have technical requirements and don't want to be a gatekeeper about additional dependencies usage. My only expectations are :

• using up-to-date dependencies
• to include documentation for further contributions and local usage
• considerations about clean code, maintainability, and readibility

[calaway]

Thanks for keeping me in the loop, but I think at this point I'll step back from this project. Feel free to reference or use the Cypress test suite repo I started. It isn't much, but I at least got most of the boiler plate starting code ready and set it up with Prettier and ES Lint. After that I just wrote one header test as a proof of concept.

Best of luck!

[benjamin-thomas]

Hey @geromegrignon

I think I'll be interested in contributing to developing a cypress test suite.

Can you explain to me what's up with v2 though? I don't see any activity on that.

Are you planning on creating the repo inside the gothinkster namespace? If so yes add me as a maintainer.

Initially, I though that a Cypress test suite had to exist for this project, so that's what I'd be interested in contributing (I will surely learn a few things along too).

I'm not sure I follow how this project is structured though. There are a bunch of scattered repos, some projects are unmaintained, multiple websites, etc.

So please do clarify if there are any specific objectives regarding this new repo.

As a side note, I think this project should be handled mono repo style. It'd probably make things easier regarding quality control, discoverability, etc.

But for a start, let's just say a test suite exists and it is expected to be ran before submitting a v1 project.

[geromegrignon]

The current version of the specs was defined 5 years ago. Expectations evolved and there are some bugs to fix too. The goal of this new version is to address them.

There is no activity yet about v2 as the new maintainer team had to deal with some issues with the project firstly.

While the first implementations were created and maintained by the maintainer team of the project, most of the current implementations are now created and maintained by the community.

Today the core repositories maintained by the core team are:

• stater-kit: a starter with assets
• realworld: documentation
• node-express-prisma-v1-official-app: the new backend implementation used for the demo (conduit.productionready.io)
• the ongoing frontend implementation to replace the current one used for the demo (demo.realworld.io)

Some of the implementations living inside the GoThinkster organization on GitHub are still maintained by the community.

Cypress (or any e2e related solution) will help us to test newly submitted frontend implementations without having to rely on manual tests to approve them.

A mono repo would not match our needs:

• contributors fork the starter kit
• anyone should be able to fork an implementation without caring about additional projects living in the same repository
• so there is only the documentation left

[benjamin-thomas]

Alright thanks for the background info :)

Let me know when that repo will be ready and I’ll start working on it.

[Christine-Pinto]

I'm currently working on a new v1 version of the frontend demo app for maintenance effort and as a starting point for fix/features for v2. As I already did with the new backend implementation (https://github.com/gothinkster/node-express-prisma-v1-official-app/blob/main/.github/workflows/ci.yaml), I plan to use a GitHub Action for tests.

I just started using Github action to execute cypress tests on pushes into a repo. They are amazing and so easy.

stater-kit: a starter with assets realworld: documentation node-express-prisma-v1-official-app: the new backend implementation used for the demo (conduit.productionready.io) the ongoing frontend implementation to replace the current one used for the demo (demo.realworld.io)

Would it make sense to add any cypress tests for any of these core repositories? @geromegrignon

[geromegrignon]

I just started using Github action to execute cypress tests on pushes into a repo. They are amazing and so easy.

GitHub actions are awesome!

Would it make sense to add any cypress tests for any of these core repositories?

For backend applications, we are already testing them with a Postman collection.

[geromegrignon]

I created a repo for the Cypress test suite: https://github.com/gothinkster/realworld-e2e-tests

@benjamin-thomas & @ChrisZie I gave you write permissions so you'll be able to push content on the repo.

I keep this issue open for general information about the project. I encourage everyone to open issues on the new repo for details about the test suite.

[Christine-Pinto]

Add the first set of tests - https://github.com/gothinkster/realworld-e2e-tests/pull/2. Do you want to link that PR to this issue? @geromegrignon

I still have to check how we can set up that these tests run when something is pushed into this repo.

[geromegrignon]

Hi @ChrisZie, I'll review the PRs next week. About links to this PR, I would prefer to update this issue with general information, keeping details in the e2e repo.

I'll open an issue about automation in https://github.com/gothinkster/realworld-e2e-tests to discuss expectations and solutions to automate tests.

[benjamin-thomas]

I created a repo for the Cypress test suite: https://github.com/gothinkster/realworld-e2e-tests

@benjamin-thomas & @ChrisZie I gave you write permissions so you'll be able to push content on the repo.

...

Got ya thanks, I'll be working on it soon (sorry for the late feedback)