Add tests and docs for tests

[liammulh]

EDIT: We have Vitest set up and working. https://github.com/numberscope/frontscope/pull/245 adds the ability to create E2E tests with Cypress. The original comments in this issue are stale. See https://github.com/numberscope/frontscope/issues/25#issuecomment-1379482635 for up-to-date status.

[liammulh]

Unit tests are added in https://github.com/numberscope/frontscope/pull/123.

[liammulh]

The ability to add integration tests is also done in https://github.com/numberscope/frontscope/pull/123.

[gwhitney]

Is there an example integration test? What are the relevant source files? Thanks.

[gwhitney]

As far as I can see, there do not actually exist any integration tests as of yet, and in addition, some of the unit tests are disabled because certain aspects of the testing are not working (as documented in the code). We need to get to the point that each source file has at least some tests properly running on its code.

[liammulh]

there do not actually exist any integration tests as of yet

Yes, this is true. I believe integration tests can be written using Vitest without any further configuration.

We need to get to the point that each source file has at least some tests properly running on its code.

I agree.

---

A couple of points I think we need to discuss:

1. We should think about the naming convention for integration tests and where they will be housed.
2. I think unit tests can be added as we touch the files in future PRs. Not sure how we'll want to add integration tests. It seems like perhaps this deserves its own issue. Thoughts?

---

I'm not actively working on this, so I'm going to un-assign myself.

[gwhitney]

Yes, this is true. I believe integration tests can be written using Vitest without any further configuration.

Is there any guidance of how to go about this? In my mind, an "integration test" is something like "bring up the frontscope in a browser, click on certain buttons, and it should produce a picture" -- i.e., something that uses the whole system running end-to-end. I presume vitest doesn't allow this sort of thing (as I presume it does not incorporate a browser that allows prescribed clicks etc.) but perhaps there are some sorts of integration tests for a package like frontscope that do not involve the use of a browser and simulated user interaction. Any thoughts on what an example of such might be? I mean, can I put a test on the Scope view that somehow invokes a sequence option and a visualizer option and invokes the create bundle operation and then checks something about the resuting Bundle? I would consider that an integration test even if there's no actual browser involved. So any pointers on how to do something like that would be welcome.

Ultimately, I think we will also need to do automated browser testing with simulated clicks as well -- but I think that will require something like https://www.browserstack.com/ and I think setting something like that up should be left to another issue.

[liammulh]

Okay, the definitions I have in my head are:

• integration test := a test where multiple modules are tested to see if they behave correctly together (this type of test is similar to a unit test, but instead of importing a single module, you import multiple modules and try to ensure some behavior is working correctly)
• end-to-end test := a test where you essentially try to mimic the behavior of a user (ideally this type of test ensures the parts of the tech stack exposed to the user are working)
  • N.B. this type of test works from the front end downward, as opposed to an integration test which could theoretically bypass the front end entirely.
  • This type of test usually involves a test framework that runs tests in a headless browser or something along those lines.

These definitions seem to comport with what is written in this blog post. (I think the author, Kent C. Dodds, is a fairly reliable source for info on testing JS.)

---

As for E2E testing for Vue, I think the Vue people want us to use Cypress. See https://vuejs.org/guide/scaling-up/testing.html#recommendation-2.

I believe Cypress can also be used for testing components.

---

Would the following be helpful in the docs somewhere?

Types of tests

• static := checking code for typos, lint errors, type errors, etc.
  • Use a spell checker, your editor, ESLint, TypeScript
• unit := checking a single module for correct output or correct side effects
  • Use Vitest
• component := checking a single user interface component for correct visual or behavioral logic
  • Use Vitest and/or Cypress
• integration := checking that multiple modules behave correctly together
  • Use Vitest and/or Cypress
• end-to-end (e2e) := checking (as closely as possible) that a user's actions cause the correct output and side effects as the actions cascade down the tech stack
  • Use Cypress

[liammulh]

perhaps there are some sorts of integration tests for a package like frontscope that do not involve the use of a browser and simulated user interaction

I think we might be able to test some of the TypeScript files this way. I will look into this further after I get some dinner.

[gwhitney]

OK, my apologies for confusing the terminology a bit between integration and end-to-end testing. It is a bit of a blurry line, though: take component testing, which is mentioned in the references you give as one important type of integration testing. How much does a component test for the Scope view differ from end-to-end testing? As far as I can see, pretty much only by not having navigation among the views, which in our case is a pretty minor aspect of the app.

So I think at the moment we should go as far as we can with vitest testing and worry about Cypress end-to-end testing as another issue. In particular, the Scope and BundleCard tests are skipped for now, but we should have tests for both and between them (or possibly as an additional integration test) we should have a test that loads enough of the system so that a bundle can be created, I guess by function calls rather than events, and we can check that it is actually drawing on canvas. I guess that vitest can be used to do that, but I admit I am a bit unclear on how to set up such a test.

[liammulh]

we should have a test that loads enough of the system so that a bundle can be created [...] and we can check that it is actually drawing on canvas [...] I guess that vitest can be used to do that

Maybe? The issue I was running into when I tried to create tests for those components makes me skeptical that Vitest + Vue Test Utils is enough for this. I'm also not clear on how exactly we'd set up such a test.

I think we should investigate if it would be easier to write this test using Cypress or Vitest. I don't want us to spend a ton of time trying to get this working using Vitest only to find out that it would have been a lot easier to do in Cypress. Also, I think Cypress is something we're going to want eventually, so I don't see the harm in investing some time into it.

I'll sink some time into this tomorrow morning and report back here with my findings.

[gwhitney]

Yes well the operative word was "guess" -- it was just that. If you have the inclination and time to investigate whether testing these sorts of things would be easier with Cypress, more power to you. I think it's good that we seem to be in agreement that testing them one way or the other is of significant value.

[liammulh]

Okay, I've just installed Cypress, and I think I see a solution to the problem I was having when trying to add tests for the components @gwhitney mentioned above:

If we add the p5 script in the wrapper, I think the issues I was having will go away. That being said, it's been a while since I was working on adding those tests, so maybe I'm mis-remembering the issue.

[liammulh]

I worked on my Cypress PR a bit last week.

From the Cypress docs:

Our users are typically developers or QA engineers building web applications using modern JavaScript frameworks.

Cypress enables you to write all types of tests:

• End-to-end tests
• Integration tests
• Unit tests

Cypress can test anything that runs in a browser.

I think it makes sense to rewrite the existing tests in Cypress and just abandon Vitest in favor of Cypress.

• Cypress does everything Vitest does and more.
• Cypress is mature and stable.
• Vitest is new and in beta.
• The existing tests shouldn't be too difficult to rewrite with Cypress.

Need I say more?

[gwhitney]

As usual, my position on something like testing is that if you are willing to do the work and Cypress seems suitable (more so than vitest), by all means proceed.

[gwhitney]

P.S. And indeed, if you are pursuing this then it would be great if you could assign yourself so that we can see that the last remaining issue in the first Project of the "roadmap" is being attended to :)

[liammulh]

This is an "epic" issue in the sense that it has multiple "sub"-issues:

Right now, we have two testing frameworks: Vitest and Cypress. I think the plan was to rewrite Vitest tests in Cypress. However, I think Cypress is mostly used for testing code running in the browser, but we also need to be able to test generic code that doesn't run in the browser. I think we can use Vitest for this.

• [ ] Add docs for creating component tests with Cypress
• [ ] https://github.com/numberscope/frontscope/issues/246
• [ ] https://github.com/numberscope/frontscope/issues/73
• [ ] Add docs for creating integration tests with Vitest

[gwhitney]

This should be brought to some satisfactory status with #420. Setting milestone.