Discussion: Testing Strategy

[qiyi-jiang]

Considerations:

The scenarios where front-end behavior, back-end assertions and assertions on 3rd-party testing clouds are to be in a single test case/flow and verified as a whole. e.g. MAGIC SIGN-IN

1. FE: user taps on magic-sign-in button, the page shows magic-sign-in form with email/phone input and submit button. 1.1 FE: user fills in the email/phone input and taps on submit button
2. BE: a new magic-sign-in request is created, an email/SMS is sent
3. 3rd-party: an email or SMS is received and the body includes the correct URL with the generated token

[qiyi-jiang]

Testing framework

Current setup: FE: Cypress, BE: RSpec

Observations

1. Seamlessly integrating FE Cypress and BE RSpec is seemingly impossible. In other words, a test case that is executed and carried out by the combo of FE Cypress and BE RSpec is not possible.
2. FE Cypress cannot take care of BE assertions.
3. BE RSpec can take care of FE behavior assertions using Capybara.
4. Both of them can integrate with most of the 3rd-party testing clouds, i.e. Mailinator via the API.

Conclusion BE RSpec with Capybara will be used for these all-in-one test cases.

[qiyi-jiang]

Cross-browser capabilities

SeleniumDriver, that is used by Capybara, supports a number of browser engines to test with: https://www.rubydoc.info/github/jnicklas/capybara/Capybara/Selenium/Driver

[qiyi-jiang]

Game Plan

1. Setup and configure Capybara and its dependencies on BE codebase.
2. Configure RSpec so a specific command could initiate or mount the FE (in testing env) before running test cases.
3. Configure RSpec/Capybara to selectively run on different types of browsers.
4. Figure out and setup the best testing cloud for testing email and SMS retrieval - for now Mailinator.
5. Write the first test case which is described in the Considerations: section of the issue.
6. Configure CI/CD part for the new setup.

[qiyi-jiang]

Priorities

No.2 - Configure RSpec so a specific command could initiate or mount the FE (in testing env) before running test cases. No.4 - Figure out and setup the best testing cloud for testing email and SMS retrieval - for now Mailinator.

These are to be addressed first - as they are the prerequisite technical hurdles to overcome for the whole plan to fall into place.

[qiyi-jiang]

Thoughts

By implementing this testing setup, we will have 3 different types of testing setups:

1. Cypress on the FE
2. standalone RSpec on the BE (only BE assertions)
3. all-in-one with RSpec/Capybara

I don't think we should get rid of FE Cypress setup in any cases. It is much more productive and atomic to write the FE-only test cases using FE Cypress rather than through all-in-one RSpec/Capybara. Moreover, Cypress has some advantages over Capybara when it comes to synchronous FE testing (i.e. the ability to listen to FE events when compared to Capybara's waiting strategy) Same with BE-only test cases.

In other words, we use 3 different setups depending on the nature/context of the test cases.

[fny]

I don't want to use Capybara/RSpec for testing because it requires that you have access to the database directly.

So in theory, these end to end tests should be able to run by using the API endpoints for set up and tear down. This way you'd actually be able to run the tests against a staging instance or even the production server.

This will also drive us to create API endpoints that will allow us to perform any number of needed operations in the app.

@ever-dev What do you think?

[ever-dev]

By means of end-to-end testing, it should test an application workflow from beginning to end. For example, for a user, create, update, delete, modify actions should be tested. But some of those behaviors are missing due to application spec, and also it's impossible to test all cases when the data is vary depending on the date and time.

The easiest way to overcome this is to set a proper value in the database before we test. For this, we might need to provide some APIs to set values in the database, only for testing purposes. In this case, we must make sure that these APIs are not available to call outside of the testing framework.

One thing to consider when we try to test on the production server, the data created for/by testing should not affect the others at all.

I'm not sure how Capybara/RSpec works but for Cypress, we are able to call actual API endpoints and get responses from them. I'll take a look at Capybara/RSpec and add my thoughts on that later.

Question: Is there any way to check third-party libraries? For example, how can we check whether the email is delivered to the user correctly?

[qiyi-jiang]

To-dos on BE

• at this stage, the context will be Location
  1. a POST endpoint that accepts a temporary token, find_or_generate a temp location (for that token) and a number of users for different roles within that location; responds with the generated user credentials
  2. a DELETE endpoint that accepts the temporary token, finds the location for that token, clears everything belonging to the location (a.k.a. the context), and deactivate/deprecate the token
  3. CI flow (in collaboration with Daniel) where a temporary token is generated and passed down to Cypress process as a parameter (ENV var maybe)

Until No.3, will be using a predefined/fixed token for the minimal blocking situation.

@ever-dev Here ^