The current debugging story of rules_webtesting is a bit lacking, as there seems to be no out-of-the-box tooling or suggestions for helping users debug their tests. Debugging is a critical part of developer experience, and I think we can do more to help users in this space.
The two most important thing a developer would need is:
Breakpoint part way through a test and inspect state in their test runner.
View the browser visually, open up DevTools, and click around to see the state of the page, look for errors encountered, maybe even use some framework specific tooling like React or Angular DevTools.
Point 1. is mostly handled by existing language support in Bazel, so I don't think there's anything specific there rules_webtesting needs to worry about. Point 2. on the other hand can be improved. Currently I think the only tool exposed here is the REMOVE: option which allows users to remove a --headless flag. This is not documented, requires manual setup, and each user/workspace will enable this slightly differently. I think if we add an option to easily disable the --headless flag, we'll greatly improve the debugging story for rules_webtesting.
I propose a new Bazel flag to enable this functionality, something like --@io_bazel_rules_webtesting//browsers:use_debug_browsers or just --define use_debug_browsers=True. This could be read at build time to switch the browser target to use a different metadata configuration which includes REMOVE:--headless. Users could add the flag to some --config debug, and that option is all that would be necessary to inspect a browser in the middle of a test. I'm not sure if REMOVE: is technically public API, but if this is the only major use case for it, we could arguably drop that and eliminate this heavily nuanced API.
I built a prototype of this strategy in one of my projects and I think it works out pretty reasonably. If we just did this by default with a canonical flag and some small documentation to explain it, the debugging story would be significantly improved out the box for rules_webtesting.
Using a flag isn't the only way of doing this. One other option I've seen used a lot is to have a macro like web_test_suite() generate a _debug target for each browser with the relevant flags enabled. This would be certainly better than the current situation, but I personally have forgotten to switch to the _debug target many times, and often don't get obvious errors pointing me in that direction. It also means that two targets are always generated, and we need extra effort to manage different tags between them in particular, which can be quite annoying and confusing (you probably don't want to run your _debug test on CI).
rules_nodejs already has a canonical --config debug flag this would easily be added to, so it would "just work" for users who added this new flag to that configuration. I'm less familiar with how other toolchains enable "debug mode", but if they require any additional Bazel flags, this would be quite easy to integrate with those systems via bazelrc files. As a result, I think a flag-based approach would be the most ideal direction here. Some short docs on the matter would help developers learn how to debug their test, without requiring each user/workspace to figure it out themselves.
If this seems like a good direction to take, I'm happy to put a PR together. It's not that much work and would allow me to upstream my changes and simplify some of my own infrastructure.
The current debugging story of
rules_webtestingis a bit lacking, as there seems to be no out-of-the-box tooling or suggestions for helping users debug their tests. Debugging is a critical part of developer experience, and I think we can do more to help users in this space.The two most important thing a developer would need is:
Point 1. is mostly handled by existing language support in Bazel, so I don't think there's anything specific there
rules_webtestingneeds to worry about. Point 2. on the other hand can be improved. Currently I think the only tool exposed here is theREMOVE:option which allows users to remove a--headlessflag. This is not documented, requires manual setup, and each user/workspace will enable this slightly differently. I think if we add an option to easily disable the--headlessflag, we'll greatly improve the debugging story forrules_webtesting.I propose a new Bazel flag to enable this functionality, something like
--@io_bazel_rules_webtesting//browsers:use_debug_browsersor just--define use_debug_browsers=True. This could be read at build time to switch the browser target to use a different metadata configuration which includesREMOVE:--headless. Users could add the flag to some--config debug, and that option is all that would be necessary to inspect a browser in the middle of a test. I'm not sure ifREMOVE:is technically public API, but if this is the only major use case for it, we could arguably drop that and eliminate this heavily nuanced API.I built a prototype of this strategy in one of my projects and I think it works out pretty reasonably. If we just did this by default with a canonical flag and some small documentation to explain it, the debugging story would be significantly improved out the box for
rules_webtesting.Using a flag isn't the only way of doing this. One other option I've seen used a lot is to have a macro like
web_test_suite()generate a_debugtarget for each browser with the relevant flags enabled. This would be certainly better than the current situation, but I personally have forgotten to switch to the_debugtarget many times, and often don't get obvious errors pointing me in that direction. It also means that two targets are always generated, and we need extra effort to manage different tags between them in particular, which can be quite annoying and confusing (you probably don't want to run your_debugtest on CI).rules_nodejsalready has a canonical--config debugflag this would easily be added to, so it would "just work" for users who added this new flag to that configuration. I'm less familiar with how other toolchains enable "debug mode", but if they require any additional Bazel flags, this would be quite easy to integrate with those systems viabazelrcfiles. As a result, I think a flag-based approach would be the most ideal direction here. Some short docs on the matter would help developers learn how to debug their test, without requiring each user/workspace to figure it out themselves.If this seems like a good direction to take, I'm happy to put a PR together. It's not that much work and would allow me to upstream my changes and simplify some of my own infrastructure.
/cc @erzel and @zellyn who brought this up earlier in https://github.com/bazelbuild/rules_webtesting/issues/324.