muodov
commented
1 week ago @freiondrej I will try to answer your questions here to consolidate the general test discussion from https://github.com/duckduckgo/autoconsent/pull/468 and https://github.com/duckduckgo/autoconsent/pull/490.
First of all, I really appreciate your enthusiasm. It is very inspiring to see external contributors like you and @seia-soto willing to donate their time to help with this project. Thank you!
As I mentioned in https://github.com/duckduckgo/autoconsent/pull/468#issuecomment-2255532560 and https://github.com/duckduckgo/autoconsent/pull/468#issuecomment-2293615088, there are different types of JS tests, and even more tools that you can use to write them. It can be difficult to figure out what you actually need, as every library claims to solve all your problems at once.
I think the best way is to start from the problem first, and focus on that. I outlined my view in https://github.com/duckduckgo/autoconsent/pull/468#issuecomment-2255532560, and if I had to pick the most impactful thing to do, it would be logic tests for DOM actions and the main Autoconsent class. The latter is not covered by unit tests at all, and for the former we have very clunky tests based on Playwright. Playwright is not a convenient tool for that. There's some hacky overhead: you need to make a fake HTML page, and encode JS snippets to eval in the page context.
It's been a while since I worked with Cypress, but as I understand it's more of a competitor to Playwright Test. That is, it's focused on e2e testing, emulating user actions, and is more suited for UI development. We have lots of e2e Playwright tests right now, which verify specific sites. If you check the implementation, you'll find that the logic is similar to your Cypress PR. They do the job well, and the only problem is flakiness due to the sites themselves changing all the time. That will apply to the Cypress tests too if they test against real websites. Similar to Playwright, it would also be awkward for logic tests, as testing individual functions and mocking is tricky. Finally, Playwright can run on Webkit (Safari) and Firefox, so I don't see value in replacing Playwright with Cypress at the moment.
The reason I suggested Web Test Runner or karma + mocha, is that they are more low-level. They essentially generate a web page that encapsulates all you your test code and the assertions. They automate browsers only to open that page, but that's it. You could also just point your own browser to a special URL, and it would work the same way. This gives two advantages for "unit" tests:
- you are not limited to Chromium. The only browser automation needed is opening a URL, so it's easier to test for web compatibility issues. This is important as DDG runs Autoconsent on all platforms
- there's no extra communication between the "test" world and the "page" world. Since everything is in the same (page) context, you have access to all the DOM APIs, and it's easier to mock/monkey-patch stuff.
Now, to make it more concrete, how about we do this:
- Introduce a new type of tests based on Web Test Runner
- Convert existing DOM click tests to that format.
I imagine the test would be a JS test that roughly does the following:
- Create an instance of DomActions
- Mock some methods with sinon, so that no actual DOM is required. For example, you could mock elementSelector() or querySingleReplySelector()
- assert that
click()(and potentially other methods) are working as expected with different arguments.
Alternatively, instead of a JS test you could explore HTML tests to test DomActions. You can probably get away with less mocks then.
Later on, if this works with DomActions, we can follow up with tests for the Autoconsent methods.
@freiondrej How does this sound? Let me know if you'd be interested to give it a try.
freiondrej
Some parts of the code are complex enough that they can benefit from unit testing (as opposed to e2e tests that verify rules). For example, internal logic of handling multiple CMPs, or prioritizing non-cosmetic rules. We have something resembling unit tests for rule executors, but it's hacky and not extensible.
First thing that comes to mind is using karma, but we can consider other frameworks as well.