Closed kyliau closed 3 years ago
I don't think market share is 64% for Cypress!
@ankushpoly13 according to Cypress it's probably 94%!))
Thank you to @kyliau and the Angular team for opening this topic up and requesting input. That we are all able to voice our support and list our concerns is such an important part of the community and I really appreciate the RFC. I'm also learning a lot reading the comments on this RFC about other frameworks. Such a great conversation!
I have the same concerns as a few others others have commented. I'm sad to see that e2e won't come ready to go out of the box. One of the things I like about Angular and what I found so helpful when I started is that Angular is opinionated and that a fully functioning app worked immediately. And as I became more experienced using Angular and learned more about the ecosystem, I liked that I could change my tooling to suit my needs.
Angular is still an opinionated framework and tries to encourage best practices from things like file structure, templated code via schematics, and enabling strict mode by default in v12. Shouldn't the encouraging of best practices continue with default e2e testing (and linting tools too)??
I recommend that the Angular team should leave everything that saves time to do better from others. Angular has other strengths and those are the ones that should be further developed and focused on. In this context I think that the Angular Component Team is working on the migration to MDC Web. This brings many advantages. You have more time for more important things. What I'm getting at is that the same is true for e2e testing. You should rely on technologies such as Cypress that have gained industry acceptance and basically build Angular tooling (Harnesses, etc) with the Cypress API.
@SerkanSipahi Yes, and we rely on Protractor which have gained industry acceptance since 2013, it solve business needs and we don't understand why we need to rewrite all code base to another framework. We know that it was developed by @google and this was a main reason why we choose Protractor. We don't know who is the Cypress in the world (but we know who is Google and Angular), we investigated that it very limited for our needs.
And, also, within the same opinion could say like "khmm we evaluate that React (or something else) good, we will stop development on Angular and you can switch to React".
Can someone explain to me why is Cypress so supported when it's nothing more than overrated version of Selenium 1 (yes you read that correctly Selenium 1) but done in JS. When the community worked so hard to improve that and move away from it to things like DevTools protocol or WebDriver 3/4, why do we want to go back now to something that was horrible?
Please consider the following table before answering 🤔
PROs | CypressJS | CodeceptJS | Playwright | WebDriverIO |
---|---|---|---|---|
Quick & Easy to setup | Built by a large community of SDETs | Cross browser support | Very mature community | |
Fast to implement & debug | Supports multiple test libraries | Built and maintained by a large company (Microsoft) | WebDriver 7 is very fast | |
Relatively fast to execute simple tests | Single syntax | Full docker support | Full API testin support | |
Modern tool | Multiple reporters supported | Typescript focused out of the box | Large collections of plugins, examples and helpers | |
Considered a standard by FE developers | Active community and updates | Fast running | Very good documentation | |
Good for startups | Support for mobile web testing | Multi language support | Parallel run | |
Large community | Support for hybrid mobile testing | Bidirectional events | Easy debugging | |
API mocking support | Support for native mobile testing | Auto-waiting for elements | Full docker support | |
Adopted by a lot of large companies | Full docker support with multiple options | Stubbing and mocking through traffic capturing | Multiple reporters supported | |
Easy debugging | Supports iFrames | Multiple runners supported | ||
Multiple debugging options made available | Supports multi-tabs | Supports multi-tabs | ||
Parallel runs through multi-process | Supports BDD | Full support of iFrames | ||
Parallel runs through Worker threads | Reporters support | Supports multi-session browser tests | ||
Support for multiple browsers | Flakiness detection feature | Supports all browsers | ||
Large number of plugins and helpers made available by the community | Supports multiple runners (Mocha Jasmin Jest) | Supports BDD | ||
Full support for iFrames, multi tabs and multi browser sessions | Headless Browser with event-driven architecture | Has mobile support (Native, hybrid and web) | ||
Easy setup for services like Browserstack/Saucelabs | Partial mobile support | Fully event driven | ||
Full BDD support | Multi session support | Very stable and reliable | ||
SMART features (Autowaiting and more) | Supports multi-page and third-party implementations | Many maintainers and very mature community | ||
Easy to keep using the latest versions of packages | Doesn’t generate any files | Supports Shadow DOM | ||
Can be built to become a distributable package | Parallelisation support | Supports interacting natively with Angular and React elements | ||
Support Shadow DOM | Fully event driven | Extended support for DevTools Protocol | ||
Database support | Front-end performance metrics gathering | |||
Full API support (REST and GraphQL) | Continuously growing and improving with newer and newer features | |||
Flakiness detection through ReportPortal.io | Can natively talk to Puppeteer instances if needed | |||
Stubbing and mocking support | Easy debugging | |||
Fully event driven | Multiple options for Visual testing | |||
Multiple options for visual testing | Support for REST and GraphQL testing | |||
Supports single syntax for all libraries supported (WebdriverIO, Playwright, TestCafe, Puppeteer, Protractor etc) | Multi DB engine support | |||
Full integration with Browserstack/Saucelabs/ReportPortal/Selenoid | ||||
Easy syntax | ||||
Supports mocking and stubbing |
CONs | CypressJS | CodeceptJS | Playwright | WebDriverIO |
---|---|---|---|---|
It has issues scaling up | Requires getting used to the syntax | Fairly new | Can be slow if not setup correctly | |
Features offered for free in other tools, either cost money with Cypress or demands you to build it yourself | Does have a learning curve | API is still evolving so things you learn now may change next version | Slow if WDIO version used is lower than v6 | |
It has difficulties with uploads and downloads | It does require custom code if you want to have a helper allowing you to test kafka for instance | Small community for now | Does not have auto waits for elements | |
They are moving more and more features behind the subscription model | Not a framework but a wrapper | Does not support real devices but supports emulators | Retry mechanism needs to be built (however there are already a ton of examples of reliable stable mechanisms built by the community) | |
Limited support for iFrames | Supports only JS/TS | Has no support for IE11 or non-browser platforms | ||
No support for multi-tabs | Mocha only as a runner | Documentations and community are not as good as the other framework yet. | ||
Can not drive 2 browsers at the same time | Depending on library chosen and configuration tests can be slow (however they can also be very fast) | |||
Does not provide support of IE or Safari | ||||
Parallel runs require a machine with significant amount of resources | ||||
Does not support any type of mobile interaction | ||||
No Shadow DOM support | ||||
Can’t navigate to a different domain URL | ||||
API testing is inefficient if needed because Cypress always runs a browser, while API testing does not require one | ||||
Developers need to find a lot of workarounds to issues the tooling has | ||||
Flakiness detection is a paid feature instead of being tied to reporting (like many other frameworks out there) | ||||
Supports only JS/TS |
👋 everyone!
Here is Diego, a core member from the Selenium project. We'd like to thank the Protractor team and its contributors for their huge efforts over all these years 🤗 Protractor is one of the main reasons why the JS Selenium WebDriver bindings have been highly used during the last years.
To continue this amazing collaboration, we'd like to invite anyone who is interested to provide a solution like or similar to the one mentioned at FAQ#8
(@cnishina how can we support you?), to let us know how we can help at the Selenium project. Feel free to reach out to us through our Slack/IRC channel, we are here and there to help the community.
Reading through all the pros and cons of the different existing technologies, it seems that there is quite some variation in what the preferred technologies is for different teams. Any technology choice will leave the people that have chosen a different technology disappointed that angular has not chosen their technology as the default one.
My question would be how feasible it is to give the choice to the end-user, similar (but I presume much more complex to pull off) as with the choice between css/scss/… when starting a new angular project.
IMO it would be ideal that angular facilitates the different e2e testing tools, without limiting to a specific one. Probably this would necessitate some sort of collaboration between the angular team & some of the major tools. Where angular provides some generic scaffolding that each tool/technology can build upon/integrate with. This would then by default also future-proof the solution when new tools arise.
A similar concept would be useful for the whole jasmine/jest/… choice for unit testing.
Even without understanding specifics, I can imagine that this would probably be the most time-consuming and complex solution for the angular team; but it would probably also be the one that serves the community the most.
I have no clue of course about feasibility and/or budget considerations, just my 2 cents. Whatever the choice; thank you for asking the community for their input about this subject. It is enlightening to read the multiple viewpoints and opinions that have to be taken into account by the team.
@StanislavKharchenko
When it comes to that point, you can throw React in the garbage because they often release a new API which is supposed to be the new standard and declare the old one not good. Do you want to rewrite your React app every time? I don't understand the confusion about Protractor which is hardly or not at all supported anymore.
That's your opinion. I respect that. You could do what you want :) And this is not about changing a framework. It's an e2e test and not replace Angular for React!
And, also, within the same opinion could say like "khmm we evaluate that React (or something else) good, we will stop development on Angular and you can switch to React".
@LanderBeeuwsaert me and others have been saying this all along. Don't force a specific framework on developers, provide bindings for the most popular ones instead. Angular should have never tied itself to an e2e framework in the first place.
Angular should have never tied itself to an e2e framework in the first place.
Bullshit. I'm thankful to them for this.
Phrases "I strongly recommend" and "you should" look ridiculous from persons who are not even contributors to the Protractor code. Give some respect.
I agree that one of the biggest points of Angular is all-in-one package that just works. I mean starting from DI to material, to e2e testing. For me (and my clients) it means predictability and not wasting time looking around for some sh*t to make things work. Angular team spent time collaborating with MS on TypeScript, they also provided VS Code extensions. I'm sure they can work with MS on Playwright or stay with Protractor.
But they should keep to the idea "all-in-one and it works" (c) :)
I don't think that having a default (whatever that default would be) and giving choices that can be easily switched to (for those who have different needs outside of that default); are mutually exclusive. So I'm not sure the whole discussion about; "angular has to provide an all-in-one package" versus "angular needs to give developers the choice"; is even necessary.
If it would be easy to setup different/extra e2e tools; suppose that you could be able to set them up easily in parallel (we run cypress & playwright for example); it would possibly make the decision/migration/... around protractor also easier. Giving teams that use protractor the option to code their new tests with a different tool while not having to lose their investment into protactor. Not ideal maybe, but for a lot of teams probably better than a big migration-at-once approach.
"If it would be easy to setup different/extra e2e tools" is never easy ;) Mostly because you depend on something you don't really control.
Maybe. Any level of abstraction implies some loss of control. Alternative is to write everything in assembler.
@e-oz consider your own history and take your own advice.
@GSasu the main driver of Cypress is usability. I can agree that the paywall is a problem, but gains in productivity are significant. It all depends on the problem you are trying to solve. We use Cypress for all our e2e needs (including live monitoring) and it works fantastic. It's not a perfect fit for everyone, but then again neither is WebdriverIO (which is also great).
I don't agree that Cypress is a regression in terms of architecture though. It's not trying to be Selenium or even fit all the same use cases.
@csvan I'm not saying to people what they "should" or "should not" do. As I understand, Ng team is asking for your opinion here, not for the instructions.
Ng team is asking for your opinion here, not for the instructions.
It is my opinion. You reading it as an instruction is not my problem, especially when the context makes it obvious that was not the case. If you are not sure what people intended, you can always ask.
@GSasu the main driver of Cypress is usability. I can agree that the paywall is a problem, but gains in productivity are significant. It all depends on the problem you are trying to solve. We use Cypress for all our e2e needs (including live monitoring) and it works fantastic. It's not a perfect fit for everyone, but then again neither is WebdriverIO (which is also great).
I don't agree that Cypress is a regression in terms of architecture though. It's not trying to be Selenium or even fit all the same use cases.
The main driver of a test framework should be stability, reliability, reusability extendability, feature maturity, documentation and community, not usability.... that's something only FE engineers care about in a test tool. SDETs other QA engineers are usually capable of making any test tool "usable". I am happy that you are having a great time with Cypress, there are exceptions to all rules.
You don't have to believe me or take my word on the architecture regression of Cypress (I'm not the holder of absolute truth nor the father of all knowledge), but take the steps I took and then come back with the feedback:
If Cypress is so great, we might as well go back to Selenium IDE and accept being called "monkey testers" because all we do is "clicky clicky things" and we are not real engineers right?
If we are so sure of Cypress and we as a community start accepting paying for features that other tools offer for free than I'ld rather pay for a real tool like let's say Ranorex than Cypress. Just a thought 🤔
@LanderBeeuwsaert
Maybe. Any level of abstraction implies some loss of control. Alternative is to write everything in assembler.
Genius! Go on, the whole thread is listening...
@GSasu
Calm down, nobody (me included) is saying that Cypress is absolutely fantastic and the de-facto way to do e2e testing. It isn't. It is, however, extremely good at the niche of e2e testing that it is focused on, and it provides a lot of value for those needing that niche. Other tools do the same things in their respective areas, and everybody should pick whatever helps them deliver value and build sustainable solutions.
I haven't done the digging you have done regarding Cypress vs. Selenium architecture so I can't comment. Even if that is the case, it doesn't change the fact that Cypress is great at what it does, and that is what we need it for.
If Cypress is so great, we might as well go back to Selenium IDE and accept being called "monkey testers" because all we do is "clicky clicky things" and we are not real engineers right?
I don't understand how you are seeing it this way. Our testers are engineers and they love Cypress because it has increased their productivity tremendously. That's value delivered, and that's what we are after. I am really happy for anyone having the same results with another framework, and encourage them to stick with it.
@csvan I was and still am very calm, don't know what made you think I wasn't 🤔 . The example with the "monkey testers" is because historically speaking that's how we were described as in the past with Selenium 1. SO to me, anything that wants to reproduce this is just plain old bad. In terms of the "de-facto" you haven't said it, but I don't think you know how large is the amount of engineers that come in and tell SDETs that they want Cypress because that's the industry standard.
I don't and probably never will consider it a standard, it's far from it, but maybe we should all bold out the fact that it's not a standard maybe that way other engineers stop calling it that way.
I wish other tools receive the same marketing that Cypress receives, I'm sure at that point Cypress would drown in it's own pot.
@GSasu I absolutely don't consider Cypress a standard (or think it should be) so no disagreement there :)
I think from my perspective (Admittedly a weak one), of attempting to create a new JS framework from all the ones mentioned was that I came from a Ruby / scripting area (This was in a previous job, and I've since gone back to ruby).
What I found when using Cypress and what others may consider the "leading standard", was it was very easy to setup as a novice JS user. But it also didn't let me do the things I wanted to as an Engineer with several years experience. Perhaps this was misconfiguration on my part, but Cypress felt very good and easy if you did things the way they told you. For example just some 1liners that I found almost impossible in Cypress were things like
Compare that to my brief experience with wd.io, which whilst it wasn't as nice to setup, and I couldn't get working as well, was permitting for me to to code things and store things the way I wanted. I think Angular sat somewhere inbetween the two when I trialled it.
So for me, given everyone is proclaiming pros and cons, I think the best thing angular can do is just link to every migration guide it gets, but steers 100% clear of saying this is the migration you should do. A few people have said this is good and a few have said this is bad. But when you trade on your name, when you leave your area, the name you pass on will be associated to you.
Go with cypress
Why not consider CodeceptJS? https://codecept.io/. It's a nice API with "I" syntax which supports any engines underneath:
Supported Engines out of the box (most popular):
If anything changes in the future, just switch the engine underneath without rewriting any tests ❤️
More advantages: Exclusive support for Shadow DOM elements for both WebdriverIO and Playwright. Just write a simple CSS locator without worrying about Shadow/WebComponent implementations
I.click('any-shadow-element') // run against Webdriver.io and Playwright
^^ We have implemented at Salesforce. For more info on ShadowDOM support, take a look at SaleforceEngineeringBlog/WebdriverIO-ShadowDOM
Hey everyone! Thank you so much for all your feedback. We will review the comments thoroughly, and post an update here once we decide the next steps. We truly appreciate all the inputs!
Hey everyone, thanks for all the great feedback, the Angular team really appreciates your comments.
To summarize, a few high level takeaways from the RFC are:
For point (2), the stance of Angular is that we want to be opinionated when it comes to the core experience. This means sane defaults in new applications, fast serve/build, and advanced optimizations and best practices enabled out of the box. We strongly believe there are opportunities for the community / open source partners to chime in when it comes to peripheral infrastructure like e2e testing. Leveraging the power of the community will enable our team to scale better, and help us focus on the core features that make Angular unique. In light of this, we have decided not to include Protractor in new projects starting from version 12, and instead work with partners to integrate their solution with Angular CLI via schematic/builder to make sure the ng e2e
command continues to work seamlessly. So far we have reached out and heard back from Cypress, TestCafe, and WebdriverIO. We will continue to expand our collaboration with other partners going forward (please reach out to us!).
For point (3), we are exploring the possibility of a shared ownership of the project with other enterprise partners. This effort will keep Protractor going in the form of version 6 (no Control Flow, no legacy deps, no jasminewd). As many users have pointed out, Protractor without Control Flow is still very much useful on its own. Projects that have migrated away from Control Flow will benefit the most from such effort. If your team / company is interested in such collaboration, please get in touch with us at devrel@angular.io. If we garner enough interest, we will work out a proposal with interested parties.
Thank you everyone for taking the time to respond to this RFC!
TLDR
The Angular team plans to end development of Protractor at the end of 2022 (in conjunction with Angular v15).
Why?
Protractor was created in 2013 when WebDriver APIs were not yet a standard and end-to-end (e2e) tests were hard to write due to lack of support for
async
/await
. To solve this problem, Protractor wraps selenium-webdriver and abstracted asynchronous operations from developers with the use of Control Flow.Since then, the JavaScript standard and ecosystem advanced considerably, providing modern syntax and much better development tools. Nonetheless, Protractor is not able to leverage such technology without forcing users to rewrite their tests. Meanwhile, robust alternatives have emerged in the web testing space. Developers will see more benefits from adopting a more modern testing tool than from updating to a breaking version of Protractor which does not provide additional functionality or developer ergonomic improvements.
We would like to hear from the community on
This RFC will close on Friday April 16, 2021.
State of Protractor
The Angular team created Protractor in 2013 when WebDriver APIs were not yet a standard and end-to-end (e2e) tests were challenging to set up. The proliferation of callback patterns in JavaScript back then made asynchronous operations difficult to write and manage.
Protractor, echoing the approach taken by the underlying
selenium-webdriver
, solved this problem by managing promises via Control Flow. Control Flow makes asynchronous calls appear synchronous, thereby avoids the use of Promises entirely.This strategy worked well for some time, but as the JavaScript standards and toolings evolve, Protractor regressed. Starting from ES2017,
async
/await
makes handling Promises significantly easier because they no longer have to be chained. However,async
/await
is fundamentally incompatible with Control Flow, and support for Control Flow is dropped fromselenium-webdriver
v4.0 completely. Protractor, being dependent onselenium-webdriver
, is not able to upgrade to the new version without introducing a huge breaking change and forcing users to do a migration for all their tests.Besides dropping support for Control Flow, making Protractor compatible with ES2017 involves a significant amount of work to overhaul its implementation. Legacy dependencies such as jasminewd2 and Q promise library ought to be removed, which will bring about further breaking changes.
Since Protractor was initially designed to support AngularJS, many of its features like locators and mock modules are specific to AngularJS. These features only work in AngularJS and not Angular. They will no longer be relevant once development on AngularJS ceases by December 31, 2021.
Removing Control Flow and dropping AngularJS-specific features would make Protractor essentially just a wrapper around selenium-webdriver that provides no additional functionality.
Current Landscape for Web Testing
Many testing solutions have appeared since the creation of Protractor, and they offer better stability and much improved support for features like cross-browsers testing. Some even eschew the WebDriver standards completely in favor of using the DevTools protocols directly.
In order to understand the adoption trend of these alternatives in the Angular community, we conducted a survey on e2e testing in January 2021. We received close to 1000 responses and got a lot of feedback from the community. A clear signal that we received is that there is no one-size-fits-all solution for all Angular projects out there. Fewer than 20% of the respondents use Protractor in their project. Some users prioritize the cross-browsers support that the WebDriver standards guarantee, whereas other users prefer the stability and flexibility that DevTools protocol offers. A similar survey focused entirely on enterprise customers yielded comparable results.
Moving Forward
Deprecating Protractor is not a decision that is taken lightly by the team. Inside Google, we support thousands of projects and hundreds of thousands of test targets that depend on Protractor. After evaluating the costs and benefits of updating Protractor vs migration effort imposed on users, we conclude that the best strategy to set users up for success in the long term is to encourage migration to a more modern and framework-agnostic testing platform. The reasoning is as follows:
Removing Control Flow from user code is a gigantic effort because it forces users to update all their tests. Google went through this effort, and it took a major infrastructure team half of their time in 2018. Even then, they could only migrate tests written in TypeScript due to the availability of more robust tooling for source code analysis and transformation.
After going through such a massive endeavour, we realize the “new” Protractor is no better than other existing solutions. The only feature that differentiates Protractor from
selenium-webdriver
at this point is the ability to automatically wait for the application under test to become stable (waitForAngular
feature in Protractor).Although
waitForAngular
is useful, it strongly couples the testing platform to the Angular framework. Some teams at Google have found that solutions that do not require knowledge of Angular can perform the tests equally well by using a robust retry strategy. We believe this is how e2e testing should be done going forward, and projects in Google are already converging towards a testing platform that is WebDriver compliant and framework agnostic. This allows our web test team to maintain a single solution for all web applications.Externally, there are many excellent alternatives available to the open source community, such as:
(The list is sorted in alphabetical order and is non-exhaustive.)
Deprecation
The most important focus for the Angular team is to ensure a smooth transition for Protractor users. We have been engaging with different vendors to ensure
So far, the Angular team has reached out to the teams at Cypress and WebdriverIO to achieve the goals above. We will provide regular updates as more resources become available. If there is a particular platform that you’d like us to reach out to, please let us know in the comments below.
Our proposed deprecation timeline is as follows:
@angular-devkit/build-angular:protractor
is usedNo decision is final until we assess all the feedback from this RFC.
Extended LTS
For users who require extended long-term support after the end of life of Protractor, we suggest reaching out to our external partners:
If your team / company provides similar services, please let us know!
FAQs
ProtractorHarnessEnvironment
for using harnesses in Protractor. This environment will be deprecated and replaced with a new environment backed byselenium-webdriver
. The harness APIs themselves will be unchanged.ng e2e
for new projects starting from version 12? The CLI will not include a default e2e builder for new projects, and users will decide which third-party builder to install. This is because picking an e2e platform really depends on the requirements of the project and there is no clear best fit for all Angular projects.@angular-devkit/build-angular:protractor
), which meansng e2e
will no longer run Protractor. However, we are open to revising the timeline based on user feedback. Please let us know in the comments below.Testability
provided by@angular/core
to check if the app is stable. This feature can be integrated into other platforms in a straightforward way. See the example forselenium-webdriver
.selenium-webdriver
is most similar to Protractor because it is used under the hood. If your tests still use Control Flow, you’ll have to remove it first by addingasync
/await
to the Protractor calls. After that, you can replace the Protractor methods with those inselenium-webdriver
. Although they are not 1:1 replacements, they are very close.selenium-webdriver
to provide APIs that are compatible with Protractor's. This is an idea that needs further exploration. Please let us know in the comments below if such tool is useful to you.Open questions
Links
This section is a work in progress. We will continue to consolidate more links and migration guides here.