Implement some sort of snapshot testing?

[etiennebacher]

The main issue with testing is that the output of render_docs() is dozens or even hundreds of files. One way to test that changes don't break existing websites would be to take screenshots and compare them with the original.

Each time the test runs, it generates a fake package, renders the docs, then call webshot2 to take a screenshot and compare it to the original. We can take several screenshots, for example to ensure that images are rendered in vignettes, or that we don't change by mistake the content of the "reference" page.

Potential problem: if the reference is generated on windows, will it make a difference to run the test on another OS?

@vincentarelbundock what do you think about this? did you run into some issues with tinysnapshot that could be problematic here?

[vincentarelbundock]

Hmm, yes. I get the impulse, but I'd be reluctant. Basically, tinysnapshot was a huge pain in the a** and I'm not really happy with how it came out, and I would basically never have started that project if I had known.

The main problem is the one you highlight, but I imagine webshot2 will be fickle as well, with different sizes and resolutions and slight differences in cropping. To me that sounds like an endless nuisance, with automatic tests failing on GH all the time, and local changes not matching remote changes.

Instead, I would probably focus on writing some custom testthat expectations that check many of the crucial aspects of the websites. I guess this is what pkgdown does, maybe? https://github.com/r-lib/pkgdown/tree/main/tests/testthat

[etiennebacher]

Yes, I see. It's technically easier to test some text files with snapshots, and restricting these tests to the sidebar, the index file, one vignette and one man page should already be helpful.

I'm letting this open for now, thanks for the comments