[💡 Feature]: Snapshot testing support for unit and component testing in the browser

christian-bromann commented 1 year ago

Is your feature request related to a problem?

Similar to Jest or Vitest it would be great if WebdriverIO could support snapshot testing for its unit and component testing runner. Some initial research has shown that:

to make this work we need to extend the expect matcher in the browser and send the object serialised to the Node.js world where a snapshot handler does the rest of the work (e.g. saving snapshots to files, doing the assertion)
Vitest has ported this feature into ESM but it is not easily consumable (created https://github.com/vitest-dev/vitest/issues/3016)

Describe the solution you'd like.

Allow things like:

render(
    html`<simple-greeting name="WebdriverIO" />`,
    document.body
)

const innerElem = await $('simple-greeting').$('>>> p')
await expect(innerElem).toMatchSnapshot()

Describe alternatives you've considered.

Allowing screenshot comparisons using https://github.com/mapbox/pixelmatch

Additional context

n/a

Code of Conduct

[X] I agree to follow this project's Code of Conduct

christian-bromann commented 1 year ago

To create image snapshot we might just want to re-use: https://github.com/americanexpress/jest-image-snapshot

sidferreira commented 1 year ago

Will it work with appium? Also, a cli argument to update snapshots?

christian-bromann commented 1 year ago

Will it work with appium?

Yes, a toMatchSnapshot or toMatchImageSnapshot matcher would take a snapshot from an element or browser and compare them with each other. Appium sessions can take screenshots so this feature would work there too.

Also, a cli argument to update snapshots?

Yes, that would be a requirement.

erwinheitzman commented 1 year ago

Christian, what is the goal here? I myself have done html diffing but it's very prone to change change (html).

Pix diff would be awesome.

Diffing an element with a snapshot I have never done to be honest as you are checking way too much, instead I recommend to only verify what is needed (a single attribute for example).

If we want to go for image comparisons I think that is very valuable, what do you think?

christian-bromann commented 1 year ago

I think DOM snapshotting can be valuable when used on web components and within a small subset of a DOM tree. You are right that it gets prone to change when the snapshot is very big. I am happy to print out e.g. a warning message if we see the snapshot being to big. The ultimate goal here is to be able to compose a lot of assertion into one snapshot check. The same goes for image snapshots where it can get prone to change or flakiness due to different render strategies in different browser.

AnthonyQuy commented 1 year ago

Hi, I'm interested with this ideal and would like to spend sometime on. There are some thought:

Is this always enable or have to be put in the config as plugin?
How do we config this feature? (Baseline file location, filename format)
Properly we need to handle dynamic element then might need option to ignore, override value of element (ex: expect(innerElem).toMatchSnapshot({ignore: subSelector}) )

christian-bromann commented 1 year ago

Is this always enable or have to be put in the config as plugin?

Ideally be always enabled.

How do we config this feature? (Baseline file location, filename format)

Please take a look at how Vitest or Jest allows the configuration of these parameters. If we can align to them that would be great.

Properly we need to handle dynamic element then might need option to ignore, override value of element (ex: expect(innerElem).toMatchSnapshot({ignore: subSelector}) )

Sounds like a neat idea +1

@AnthonyQuy please go for it 💯 have a look at Vitest and see if we can just re-use what they have. Our expect lib is based on Jest which makes this a probably difficult but let's see how much code we can re-use.

AnthonyQuy commented 1 year ago

@christian-bromann I'm trying to put Vitest toMatchSnapshot similar implementation to our expect. Seem they have an util to get the test object then build the expected filePath from the test filePath but we don't. Do you have any suggestion how we build the expected filePath ?

https://github.com/vitest-dev/vitest/blob/76607ead169733f27e241554bca01f10e81ea849/packages/vitest/src/integrations/snapshot/chai.ts#L63C15-L63C19

**The draft PR is linked

christian-bromann commented 1 year ago

Do you have any suggestion how we build the expected filePath ?

I am not quite sure what the expected filePath is but I would assume the ./__snapshots__/<test-filename>.snap right?

AnthonyQuy commented 1 year ago

yes similar but I need the spec file location to build it

christian-bromann commented 1 year ago

Gotcha, we could propagate it as part of an environment variable through the testrunner.

AnthonyQuy commented 1 year ago

same thought but a bit struggle to find the best practice. Like a global variable, an util object, or add to the browser object...

christian-bromann commented 1 year ago

See https://github.com/webdriverio/webdriverio/blob/fbc2bcdb65bf9f80edbe7d055733e788435e2679/packages/wdio-local-runner/src/worker.ts#L100-L103

AnthonyQuy commented 1 year ago

Great!! seem I'm overthinking... this should be good enough for me to continue! 👍 Thank you!

AnthonyQuy commented 1 year ago

Sorry, a bit busy recently! Quick update on this...

Properly the worker approach seems not enough to make this work as we gonna need the describe name and test name to create the snapshot. I'm moving with new approach to create a service wdio-snapshot-service that will be plugged in by default

AnthonyQuy commented 1 year ago

@christian-bromann can you help to release an alpha version for below PR, then I can test the service properly https://github.com/webdriverio/expect-webdriverio/pull/1375/commits/98c73c5e1a19a1b132c62643cba6f3faa216d783#diff-46a780c9e128671c56b215d26f78ad7369bbd3c83527e76ff462bbf8675c277c

AnthonyQuy commented 1 year ago

the draft PR for the service https://github.com/webdriverio/webdriverio/pull/11753

christian-bromann commented 1 year ago

can you help to release an alpha version for below PR, then I can test the service properly

I would prefer not to release anything that is not ready for prime. Any chance you can link the packages into each other so that you can develop in both without having to go through release cycles. For example to link my local expect-webdriverio package into my WebdriverIO development project I would do this:

cd /project/path/to/webdriverio
cd ./node_modules
mv ./expect-webdriverio ./expect-webdriverio-original
ln -s ../../expect-webdriverio .

This assumes that you have expect-webdriverio check out in the same parent directory:

/ root
    |--> webdriverio
    |--> expect-webdriverio

Overall this is coming nicely together, very excited about it!

christian-bromann commented 6 months ago

This can be closed as we have now support for this. Thank you @AnthonyQuy for your contributions and initiative on this. Your input has massively influenced how todays implementations looks like.

AnthonyQuy commented 6 months ago

Cheers!! Glad to be a part of this feature :D

webdriverio / webdriverio