Open jaredpalmer opened 3 years ago
@jaredpalmer related question - is there a way to expose the bundler you make for bundling code is TSDX? Then we could point cypress-react-unit-test at that bundler and be able to do component testing directly without spinning example playground.
You mean expose the build
and watch
commands in a node.js API that can be used by scripts?
I don't see why we can't, all we would need to do is abstract the current commands into functions and export them. Right @agilgur5?
@jaredpalmer part of the rationale for #407 is also to eventually expose a programmatic API (which is also something microbundle
has had for a bit).
But I don't think that's what @bahmutov had in mind however. Looking at cypress-react-unit-test
's CRA integration, it doesn't run react-scripts build
directly as a preprocessor, it runs the webpack
preprocessor with some configuration for CRA.
I'm also not sure there would be much difference in exposing that vs just using the dist/
directory and running tsdx watch
in a separate terminal.
@bahmutov could you please clarify and be specific as to what you mean? TSDX uses Rollup and has a fairly complex config there and some code on top of it (that I've mentioned a lot I'd like to rewrite them as Rollup plugins, but there's a ways to go get there)
Add to or make a new react template [...]
- We point cypress by default at the
example
on localhost:1234
Would prefer using the example
directory as opposed to a new directory -- this ensures you're testing what you're advertising.
New command or flag We abstract the above steps behind
tsdx e2e
command. This is nice because we then we wouldn’t install a dep.
sade
has subcommands, so one could use tsdx test e2e
instead. But I'm not a fan of adding more commands to core, as I've said elsewhere, this drastically increases the API surface and makes a breaking change in one command / dependency a breaking change to all of TSDX. So if Cypress makes a breaking change and TSDX upgrades, TSDX must make a breaking change. As it already has to do for Prettier, ESLint, Jest, Rollup, etc. This makes breaking changes more frequent, which is not a good DX.
Some TSDX users also don't like the bulkiness and inability to opt-out of certain pieces either, so adding another large dependency makes that more complicated. Have mentioned elsewhere that I'd much prefer to support things like @tsdx/cypress
(or @tsdx/mocha
, etc). Also supporting apps has been a frequent ask as well (e.g. #654)
however, will still leak into code as cypress is based on mocha/chai under the hood (and not jest). The describe and it API’s are the same, but cypress basically has its own custom assertion engine at this point.
Yea, Cypress is its own test runner and has its own assertions. I'm not a huge fan of Cypress because of this and a large configuration surface with directories, plugins, preprocessors, config, etc (which goes against TSDX's purpose), which I've mentioned in https://github.com/agilgur5/physijs-webpack/pull/16#issue-345957853. (Also an instrumentation issue I filed https://github.com/cypress-io/code-coverage/issues/102 was never responded to, though neither was one in Jest https://github.com/facebook/jest/issues/9233)
Use Jest + Playwright or Puppeteer I haven’t looked at the difference between what Cypress (without the dashboard) does vs. puppeteer or playwright in a while.
Also without the test runner and assertion bit of Cypress.
Previously the main difference between Cypress and Puppeteer was the API, usage, and configuration (including plugins, etc). Utilities like puppeteer-recorder
/ daydream
and jest-puppeteer
among many others were more prevalent with Puppeteer. At least that's what it was in my experience. @bahmutov would obviously be able to speak to it more.
Unlike Puppeteer, Playwright is cross-browser. Cypress was not previously cross-browser, but seems to now support a few browsers. Playwright is also a pretty new player too and I'm guessing used internally at Microsoft.
I wouldn't suggest using Puppeteer because of its lack of cross-browser support. Libraries by definition should support at least as many browsers as the apps that use them. If those run cross-browser tests, so should the libraries.
would work out of the box with our existing jest setup
That's why I would prefer it as well, though that's not entirely accurate. jest-playwright
specifically recommends a separate jest.e2e.config.js
. tsdx test
supports the --config
flag (as of #526) that could be used for that, however I'm not sure if there's anything that may need changing in the TSDX config to support this.
@jest-runner/electron
I'd be remiss if I didn't mention karma
and @jest-runner/electron
that both let you run your existing tests in a browser, instead of running browser tests with browser events. Can think @jest-runner/electron
like a replacement for JSDOM in that sense -- I've also experienced it running as fast as JSDOM.
Both can have their uses, but if your goal is to "just render" or to run unit tests in a browser, it is much faster to get started with @jest-runner/electron
than start using something like Cypress or even Playwright.
Unfortunately, when I last checked there was still no way of running unit tests in browsers other than Electron with Jest (@jest-runner/electron
itself was experimental). Karma is also its own test runner (but not assertions) and somewhat legacy, with no Jest integration. In recent months though karma-jest
was released, but it's not a Jest runner for Karma, it's a Karma plugin for Jest. Though theoretically one could import your Jest config into it with jest: require('./jest.config.js')
(though that would require #634 / #270 )
It also helps people fall into a pit of success with testing instead of overdoing unit tests.
In my experience, I've much more frequently seen too many e2e tests (over unit tests) or e2e tests that should really be unit tests as opposed to the opposite. Test Pyramid metaphor is a thing after all, though lack of testing is a much bigger problem in general. TSDX itself has issues with this (another reason for #407 ).
Between this and storybook and size limit and github actions, I think it may be time to consider some kind lazy initialization of features.
Yes that duplicates #253 which I've been looking to implement at some point (though tsdx add
sounds like a better name) and have each piece be an independent mrm
plugin or preset as well that doesn't necessarily need to be installed with TSDX (since things like, e.g. size-limit
don't rely on it).
It would be nice to make jest
dependencies and stuff optional and have support for mocha
(On projects already depending on it)
Current Behavior
(React) e2e testing must be setup by hand.
Desired Behavior
TSDX has an e2e testing solution for React libraries.
Suggested Solutions
Add to or make a new react template
example
on localhost:1234app
orfixture
or something (that's identical to theexample
) but specifically for e2e tests. I just did this in Formik and it's a really really nice development experience.New command or flag We abstract the above steps behind
tsdx e2e
command. This is nice because we then we wouldn’t install a dep. however, will still leak into code as cypress is based on mocha/chai under the hood (and not jest). The describe and it API’s are the same, but cypress basically has its own custom assertion engine at this point.Use Jest + Playwright or Puppeteer I haven’t looked at the difference between what Cypress (without the dashboard) does vs. puppeteer or playwright in a while. The beauty of using Puppeteer or Playwright is that we wouldn’t actually need to do anything because they are just node.js packages. would work out of the box with our existing jest setup. Only thing we might have to do is potentially write a GitHub action for the e2e test that has the proper binary installed.
Who does this impact? Who is this for?
React users. Even if you don't do anything other than use Cypress to check that your that your React library actually renders I still think its worth it. It also helps people fall into a pit of success with testing instead of overdoing unit tests.
Describe alternatives you've considered
Between this and storybook and size limit and github actions, I think it may be time to consider some kind lazy initialization of features. Perhaps a new command for
tsdx init storybook
tsdx init cypress
where these run in a way that evaluates the current project and morphs it accordingly.Additional context
Formik Cypress PR: https://github.com/formium/formik/commit/af425b4b0ccbc857353d58438466509678c7a85b TSDX Monorepo Example PR: https://github.com/jaredpalmer/tsdx-monorepo-playground/commit/440235886a91f7aa10d13f63a50b1d58adabd134