search

angular / protractor

E2E test framework for Angular apps
http://www.protractortest.org
MIT License
8.75k stars 2.31k forks source link

Future of Angular E2E & Plans for Protractor #5502

Closed kyliau closed 3 years ago

kyliau commented 3 years ago

TLDR

The Angular team plans to end development of Protractor at the end of 2022 (in conjunction with Angular v15).

Why?

Protractor was created in 2013 when WebDriver APIs were not yet a standard and end-to-end (e2e) tests were hard to write due to lack of support for async / await. To solve this problem, Protractor wraps selenium-webdriver and abstracted asynchronous operations from developers with the use of Control Flow.

Since then, the JavaScript standard and ecosystem advanced considerably, providing modern syntax and much better development tools. Nonetheless, Protractor is not able to leverage such technology without forcing users to rewrite their tests. Meanwhile, robust alternatives have emerged in the web testing space. Developers will see more benefits from adopting a more modern testing tool than from updating to a breaking version of Protractor which does not provide additional functionality or developer ergonomic improvements.

We would like to hear from the community on

This RFC will close on Friday April 16, 2021.

State of Protractor

The Angular team created Protractor in 2013 when WebDriver APIs were not yet a standard and end-to-end (e2e) tests were challenging to set up. The proliferation of callback patterns in JavaScript back then made asynchronous operations difficult to write and manage.

Protractor, echoing the approach taken by the underlying selenium-webdriver, solved this problem by managing promises via Control Flow. Control Flow makes asynchronous calls appear synchronous, thereby avoids the use of Promises entirely.

This strategy worked well for some time, but as the JavaScript standards and toolings evolve, Protractor regressed. Starting from ES2017, async / await makes handling Promises significantly easier because they no longer have to be chained. However, async / await is fundamentally incompatible with Control Flow, and support for Control Flow is dropped from selenium-webdriver v4.0 completely. Protractor, being dependent on selenium-webdriver, is not able to upgrade to the new version without introducing a huge breaking change and forcing users to do a migration for all their tests.

Besides dropping support for Control Flow, making Protractor compatible with ES2017 involves a significant amount of work to overhaul its implementation. Legacy dependencies such as jasminewd2 and Q promise library ought to be removed, which will bring about further breaking changes.

Since Protractor was initially designed to support AngularJS, many of its features like locators and mock modules are specific to AngularJS. These features only work in AngularJS and not Angular. They will no longer be relevant once development on AngularJS ceases by December 31, 2021.

Removing Control Flow and dropping AngularJS-specific features would make Protractor essentially just a wrapper around selenium-webdriver that provides no additional functionality.

Current Landscape for Web Testing

Many testing solutions have appeared since the creation of Protractor, and they offer better stability and much improved support for features like cross-browsers testing. Some even eschew the WebDriver standards completely in favor of using the DevTools protocols directly.

In order to understand the adoption trend of these alternatives in the Angular community, we conducted a survey on e2e testing in January 2021. We received close to 1000 responses and got a lot of feedback from the community. A clear signal that we received is that there is no one-size-fits-all solution for all Angular projects out there. Fewer than 20% of the respondents use Protractor in their project. Some users prioritize the cross-browsers support that the WebDriver standards guarantee, whereas other users prefer the stability and flexibility that DevTools protocol offers. A similar survey focused entirely on enterprise customers yielded comparable results.

Screen Shot 2021-03-24 at 6 31 33 PM

Moving Forward

Deprecating Protractor is not a decision that is taken lightly by the team. Inside Google, we support thousands of projects and hundreds of thousands of test targets that depend on Protractor. After evaluating the costs and benefits of updating Protractor vs migration effort imposed on users, we conclude that the best strategy to set users up for success in the long term is to encourage migration to a more modern and framework-agnostic testing platform. The reasoning is as follows:

Removing Control Flow from user code is a gigantic effort because it forces users to update all their tests. Google went through this effort, and it took a major infrastructure team half of their time in 2018. Even then, they could only migrate tests written in TypeScript due to the availability of more robust tooling for source code analysis and transformation.

After going through such a massive endeavour, we realize the “new” Protractor is no better than other existing solutions. The only feature that differentiates Protractor from selenium-webdriver at this point is the ability to automatically wait for the application under test to become stable (waitForAngular feature in Protractor).

Although waitForAngular is useful, it strongly couples the testing platform to the Angular framework. Some teams at Google have found that solutions that do not require knowledge of Angular can perform the tests equally well by using a robust retry strategy. We believe this is how e2e testing should be done going forward, and projects in Google are already converging towards a testing platform that is WebDriver compliant and framework agnostic. This allows our web test team to maintain a single solution for all web applications.

Externally, there are many excellent alternatives available to the open source community, such as:

  1. Cypress
  2. PlayWright
  3. Puppeteer
  4. Selenium-webdriver
  5. TestCafe
  6. WebdriverIO

(The list is sorted in alphabetical order and is non-exhaustive.)

Deprecation

The most important focus for the Angular team is to ensure a smooth transition for Protractor users. We have been engaging with different vendors to ensure

So far, the Angular team has reached out to the teams at Cypress and WebdriverIO to achieve the goals above. We will provide regular updates as more resources become available. If there is a particular platform that you’d like us to reach out to, please let us know in the comments below.

Our proposed deprecation timeline is as follows:

No decision is final until we assess all the feedback from this RFC.

Extended LTS

For users who require extended long-term support after the end of life of Protractor, we suggest reaching out to our external partners:

If your team / company provides similar services, please let us know!

FAQs

  1. What about Component Harnesses? Angular CDK's component harnesses include a ProtractorHarnessEnvironment for using harnesses in Protractor. This environment will be deprecated and replaced with a new environment backed by selenium-webdriver. The harness APIs themselves will be unchanged.
  2. How will Angular CLI handle ng e2e for new projects starting from version 12? The CLI will not include a default e2e builder for new projects, and users will decide which third-party builder to install. This is because picking an e2e platform really depends on the requirements of the project and there is no clear best fit for all Angular projects.
  3. In version 15, would all Protractor tests break? The Protractor builder will continue to work until version 15. After that, we plan to remove the Protractor builder (@angular-devkit/build-angular:protractor), which means ng e2e will no longer run Protractor. However, we are open to revising the timeline based on user feedback. Please let us know in the comments below.
  4. Protractor is useful because it automatically waits for the application to become stable. How can I use that in other platforms? Under the hood, Protractor uses Testability provided by @angular/core to check if the app is stable. This feature can be integrated into other platforms in a straightforward way. See the example for selenium-webdriver.
  5. Which e2e framework is the lowest effort and cost to migrate to from Protractor? In terms of APIs, selenium-webdriver is most similar to Protractor because it is used under the hood. If your tests still use Control Flow, you’ll have to remove it first by adding async / await to the Protractor calls. After that, you can replace the Protractor methods with those in selenium-webdriver. Although they are not 1:1 replacements, they are very close.
  6. Where will e2e migration strategies be documented? We are consolidating a list of migration strategies doc written by vendors and community members. See Links section below.
  7. What other packages will be deprecated along with Protractor at the end of 2022?
  8. My team just did a migration to disable Control Flow. What are our options? It is possible to create a thin wrapper around selenium-webdriver to provide APIs that are compatible with Protractor's. This is an idea that needs further exploration. Please let us know in the comments below if such tool is useful to you.

Open questions

  1. Should the Angular team provide a different deprecation timeline?
  2. In addition to migration docs and CLI integration, how can the Angular team support Protractor users in the transition to other platforms?
  3. What are the alternatives to deprecating Protractor?

Links

This section is a work in progress. We will continue to consolidate more links and migration guides here.

  1. Migrating from Protractor to Cypress https://nx.dev/latest/angular/modern-angular/protractor-to-cypress
  2. Cypress official doc for Protractor users (placeholder link for now, will be live in the future) http://on.cypress.io/protractor-to-cypress
pavankjadda commented 3 years ago

Cypress is great alternative. I have been using it since few months, easy to configure and write tests. May be provide CLI option to add it to the new or existing project?

mgechev commented 3 years ago

@pavankjadda we're working with the Cypress team to ensure smooth integration with the Angular CLI.

batbrain9392 commented 3 years ago

We have huge protractor e2e code @ Blueface Comcast. A good documentation for common migration use cases would be highly beneficial.

yjaaidi commented 3 years ago

If you are looking for CDK Harness support on Cypress, here it is: https://github.com/jscutlery/test-utils/tree/main/packages/cypress-harness

amirrustam commented 3 years ago

Hi @batbrain9392. I'm Amir from the Cypress team. We're working on helpful migration guidance. To make this material better for you and rest of the community, I'd love to learn more about your project.

If you're up for it, feel free to email, or schedule a chat.

chihab commented 3 years ago

I am quite convinced Cypress offers the best DX out there but no more opinion on ng e2e is definitely the best opinion.

jessesanders commented 3 years ago

I'm glad to see protractor going away and that there is a good plan on how to deprecate it. Selenium/Webdriver is based on technology almost 20 years old and is far too fragile for today's dynamic pages to fulfill the need for a reliable testing platform. Cypress offers a much more consistent and reliable testing platform where testing edge cases becomes almost trivial.

LanderBeeuwsaert commented 3 years ago

We are using both Cypress and Playwright and have found Playwright to be both easier to use and more robust. Added benefits are amongst others ability to test Safari/webkit, less flaky tests and possibilities for conditional testing. Can somebody else that is using both frameworks comment on the advantages they see in Cypress vs Playwright? Are we missing something?

vsravuri commented 3 years ago

@amirrustam

Thanks for your assistance. I too have lot of code written in Protractor to migrate to Cypress. I will schedule a chat with you help you understand my requirement.

zbagley commented 3 years ago

Looking forward to the e2e upgrade, and have my fingers crossed the transition from Protractor will be fairly seamless. Are the any plans for a CLI update to provide a similar tool to ng update for e2e side of the transition? I know how rough updates were at the start, and really hoping the e2e update will be less painful that the early days and closer to how updates are now.

amirrustam commented 3 years ago

Hi @LanderBeeuwsaert. Amir from Cypress team here. Playwright is also a fantastic tool (and team)!

With regards to Safari, we've added WebKit support to the Cypress roadmap.

Can you elaborate more on your mentioning of "more robust"? I can address any potential concerns for you and the community, and your feedback is always helpful and appreciated for improving the tool.

If you're interested in deeper dive conversation on the topic, feel free to email, or schedule a chat.

amirrustam commented 3 years ago

Sure thing @vsravuri, and I extend the same offer to everyone else here as well. The Cypress team is here to help as much as we possibly can.

cnishina commented 3 years ago

As for # 8 on the FAQs: "It is possible to create a thin wrapper around selenium-webdriver to provide APIs that are compatible with Protractor's."

I am working on this library. It still needs to go through internal open source approvals. 🤞

alvipeo commented 3 years ago

I tried Cypress and then switched to Playwright. More features and easier to use.

LanderBeeuwsaert commented 3 years ago

Hey @amirrustam , let me first say that Cypress for us has helped us out amazingly as well. For us however there have been some things that have made it difficult (or even not possible) to use. No framework is perfect, and it's cool to see the Cypress team working on a lot of these improvements, so I'm sure it's gonna get even better when time goes on.

For reference, for us the dealbreakers have been:

nartc commented 3 years ago

Thanks so much for putting this (and the whole thought process) together for such a complex problem, @kyliau, and the team!

With AngularCLI dropping scaffolding E2E tests out of the box, it will cause more problems for beginners wanting to start with Angular. We pride ourselves so much in being a complete framework with all this built-in tools and seeing the AngularCLI starting to drop each toolings out of its box (like Linting) hurts a bit.

That said, I do understand the concern about being opinionated about which E2E solution to bake in the CLI when there are gazillion options out in the ecosystem. It would be nice to settle with the most-voted option (looks like Cypress won by a landslide here) and provide the option to opt-out of Cypress.

Another option is the documentations need to be very deliberated about the importance of E2E testing. With AngularCLI not scaffolding an E2E solution, the chance of getting into E2E testing is getting even worse IMO

adisreyaj commented 3 years ago

Thanks for taking this decision which was long overdue. Cypress seems to be the obvious way to go given its support and community. And if there aren't any huge turn downs of using it with Angular, I'd say its a win win.

I guess this would give the Angular team more time to invest in the framework core.

oscarvaldes commented 3 years ago

I've had an amazing experience using Cypress as my go-to e2e framework for the past 3 years for both enterprise and small scale projects. The community and its' integration with https://nx.dev/latest/angular/cypress/overview is just an added bonus. Also with the increased popularity of web components today, something that is framework agnostic is almost a must have. Great job and details describing the decision making behind this! It's very clear a lot of careful thought and analysis was behind this decision.

LeJeanbono commented 3 years ago

If you manage to do some Cypress component testing in browser with angular, fell free to use https://github.com/bahmutov/cypress-angular-unit-test

MikaStark commented 3 years ago

Knowning that Protractor will be deprecated. Is there a way to remove it from angular 11 projects properly in order to get ehead of v12 deprecation ?

Is uninstalling protractor, removing e2e target in angular.json and removing e2e folder enough ?

christian-bromann commented 3 years ago

Hey folks 👋 , this is Chris maintaining the WebdriverIO project. First of, big thank you to everyone involved developing Protractor, it has been a great framework and WebdriverIO has adopted a couple of feature ideas from it. We are glad that the Angular team has reached out to us to include WebdriverIO as an transition option. The team is happy to offer a free, open governed and flexible framework, hosted under the OpenJS Foundation alongside projects like Node.js and webpack.

WebdriverIO allows you to leverage both the ability to run true cross browser testing based on WebDriver as well as a lot of debugging and introspection capabilities through the DevTools protocol. With the help of various plugins the framework provides running performance and PWA audits through Google Lighthouse as well as mocking and stubing any kind of browser requests. It comes with a Jest style built in assertion library and tons of reporters, some maintained by the project some by the community.

We are currently actively working on the following items to make the transition for Protractor users as smooth as possible:

Feel free to raise an issue in the WebdriverIO project if you see anything else from the Protractor ecosystem you want to see adopted. WebdriverIO, as it is right now, can already being used for any kind of Angular projects but with these additional initiatives we hope to help with the transition even further. Cheers!

Update (16/04/2021): we implemented and battle tested a codemod that transforms ~90% of Protractors API and we continue to extend it as we support the migration process of various organisations. A migration guide was drafted and will be published soon.

Update (19/04/2021): we released a migration guide that explains how a Protractor migration can be run using our codemod. The guide will be extended as we get more feedback from folks running into issues.

Update (20/04/2021): a community member has build an expected condition library that is a drop-in replacement for the Protractor one. We will update the codemod within the next days to simplify the migration. Thanks @elaichenkov!

StanislavKharchenko commented 3 years ago

Thanks @kyliau for news and plans share. I would like to share my opinion and several points which could be critical and need to be revised.

  1. First of all - yes, Control Flow should be removed and this is not thing that should live in 2021 and of course in future.
  2. Million users around the world are using Protractor and rely on it. Why? Different reasons:
    • simple of usage within modern async/await
    • selenium-webdriver wrapper (selenium is w3c standard now)
    • own config and runner
    • waitForAngular hook
    • Protractor is own @Google and @Angular project, which adds more confidence (company with the great name) Unfortunately not all users know about deprecation plan and still thinking that Protractor project maintained by Google (see above)
  3. Migration will be simple for teams using E2E as part of its own web projects based on Angular (using under angular-cli). But, this will be a very huge pain for companies and teams, which have independent E2E infrastructure with Protractor as dependency.

By this concerns, users should be concentrated on migration to unknown 3d-party framework without any guarantee what will with it in future and not on test cases development and E2E coverage.

My propositions and options:

  1. Deprecate legacy Protractor which have Control-Flow and notify users (according to current plan till the end of 2022);
  2. Introduce some light version of selenium-webdriver wrapper which will have: Protractor API, runner, configuration to use Jasmine 3.X, default hook for Angular testing (waitForAngular) and, the main, modern selenium 4.0 which is fully w3c. Here could be sub options: 2.1 Implement next gen light version of Protractor or pick-up Protractor 6 where control-flow was removed and it on selenium Most of work was done here, just refresh it and it will help users instead of deep investigation of external 3d frameworks, its documentation, transitions and more-more pains. IMO, this will require less time-investments. 2.2 Another new lib, like teased by Craig @cnishina, with the further support. Actually, such lib will have very-very simple support since it will be fully rely on selenium-webdriver, its API and functionality. From your side need just support config+runner, waitForAngular and dependency upgrades from time to time.

We asking to re-thinking about initial plan and have compromise option. Thank you in advance!

aboudard commented 3 years ago

I think giving the choice on install is the best option. It's great that Cypress has this success, but it might not be the best default choice for every team and project. I can name a situation where it can be tricky : behind a corporate proxy ...

@MikaStark I think you can do exactly that, that's what we do when we enable Cypress.

DavertMik commented 3 years ago

Consider using CodeceptJS https://codecept.io as an alternative. With it, you can easily switch between engines like WebDriver or Playwright. So one engine gets deprecated and a new one emerges you will need to write 0 code to transition.

santoshyadavdev commented 3 years ago

Great to see the Cypress team is extending support for smooth integration with CLI. Looking forward to having Cypress as default for E2E.

AlexSkorkin commented 3 years ago

Hi,

This is Alex from the DevExpress TestCafe team. TestCafe supports many of the mentioned features out of the box:

[V] Safari support, including Safari mobile [V] Cross-browser support (Chrome, FF, IE, Edge, etc.) [V] Latest JavaScript/TypeScript Syntax Support [V] Run parallel browser instances [V] Robust and stable test cases [V] Automatic awaiting mechanisms and much more.

We'd love to chat and answer your questions if you have any. Feel free to contact us at testcafeteam@devexpress.com.

alvipeo commented 3 years ago

Playwright is owned by Microsoft, and they also own TypeScript. Consider this fact too please. They also own VS Code.

csvan commented 3 years ago

We migrated our e2e codebase (thousands of tests) from Protractor to Cypress over a year ago due to the uncertain status of Protractor, and have seen great improvements in productivity and reliability as a result. I can strongly recommend it for those seeking an alternative.

Thanks to the Protractor devs for all your hard work over the years, and thank you for making the state of the project clear so that the community can adapt.

uLucasFraga commented 3 years ago

It was good while it lasted! #goodbye 😎

kylecordes commented 3 years ago

Although a long video almost certainly isn’t the comment information density the Angular team is looking for, others might enjoy this hour+ discussion today among me and a couple other folks at Oasis Digital about Protractor, Cypress etc.

https://www.youtube.com/watch?v=tEdvyaTYPPY

mdmintz commented 3 years ago

Hey folks, 🖐️✋, This is Michael Mintz of the SeleniumBase project. I'd like to start by thanking the entire Protractor/Angular community for all the hard work that they've put into creating Angular and making Protractor an amazing tool for web automation. It's never easy to sunset a project that one has spent countless time and effort on, but sometimes the benefits of moving on to bigger projects outweighs the costs of continual maintenance on a project in an uphill climb.

I'd like to offer an existing test automation alternative that offers an easy migration path from Protractor: SeleniumBase, a Python framework. SeleniumBase has had active funding and support from companies such as HubSpot, Veracode, and iboss.

Some of you may have seen me before, as I've spoken at multiple meetups for companies such as Google (https://www.meetup.com/GDGCloudBoston/events/230839686/?showDescription=true), Microsoft (https://events.powercommunity.com/sessions/testing-web-and-mobile-apps-on-microsoft-edge-with-python-and-selenium/), The Ministry of Testing (https://www.meetup.com/ministry-of-testing-boston/events/275574548/), Boston Code Camp (https://www.bostoncodecamp.com/CC25/sessions/details/16352), Boston Python, Granite State Code Camp, the "Rapid Software Testing" community, and others. I also cohosted HubSpot's old marketing tech podcast 4 times while filling in for Mike Volpe. And of course you may already be familiar with my SeleniumBase work.

I took the time over lunch break today to demonstrate how easy it is to switch from Protractor to SeleniumBase. The SeleniumBase/examples/migration/protractor folder contains tests that have been migrated over from https://github.com/angular/protractor/tree/master/example , where the original file ends in .spec.js and the updated version ends in _test.py. Here's an example of a Protractor test, followed by the SeleniumBase version:

Original Protractor test (https://github.com/angular/protractor/blob/master/example/angular_material/mat_paginator_spec.js)

describe('angular-material paginator component page', () => {
  const EC = protractor.ExpectedConditions;

  beforeAll(async() => {
    await browser.get('https://material.angular.io/components/paginator/examples');
    await browser.wait(EC.elementToBeClickable($('.mat-button-wrapper>.mat-icon')), 5000);
  });
  it('Should navigate to next page', async() => {
    await $('button[aria-label=\'Next page\']').click();
    await expect($('.mat-paginator-range-label').getAttribute('innerText')).toEqual('11 - 20 of 100');
  });
  it('Should navigate to previous page', async() => {
    await $('button[aria-label=\'Previous page\']').click();
    await expect($('.mat-paginator-range-label').getAttribute('innerText')).toEqual('1 - 10 of 100');
  });
  it('Should change list length to 5 items per page', async() => {
    await $('mat-select>div').click();
    const fiveItemsOption = $$('mat-option>.mat-option-text').first();
    await fiveItemsOption.click();
    await expect($('.mat-paginator-range-label').getAttribute('innerText')).toEqual('1 - 5 of 100');
  });
});

Here's the above test converted to SeleniumBase (SeleniumBase/examples/migration/protractor/mat_paginator_test.py)

from seleniumbase import BaseCase

class AngularMaterialPaginatorTests(BaseCase):

    def test_pagination(self):
        self.open("https://material.angular.io/components/paginator/examples")
        self.assert_element(".mat-button-wrapper > .mat-icon")
        # Verify navigation to the next page
        self.click('button[aria-label="Next page"]')
        self.assert_text("11 – 20 of 100", ".mat-paginator-range-label")
        # Verify navigation to the previous page
        self.click('button[aria-label="Previous page"]')
        self.assert_text("1 – 10 of 100", ".mat-paginator-range-label")
        # Verify changed list length to 5 items per page
        self.click("mat-select > div")
        self.click("mat-option > .mat-option-text")
        self.assert_text("1 – 5 of 100", ".mat-paginator-range-label")

The end result is a clean, simple test that you can run in all web browsers: Chrome, Edge, Safari, Firefox, IE, and Opera. (SeleniumBase adds pytest command-line options for setting the browser and much more.)

If you have any questions, you know where to find me.

Phonesis commented 3 years ago

Just to add to the conversation. I work for a large company and we use Protractor to drive thousands of end to end tests. Of course, we have considered alternatives (Cypress, Test Cafe, and most promisingly Playwright).

Playwright is the only alternative that offers what we need. A free model (no hidden pay walls), ability to emulate different browsers (not just chromium), and easy to mix it in with Cucumber.js.

The main draw to Protractor for us though is it works great with tabs/frames (we have 'legacy' pains to deal with) as it is driven by webdriver protocol. Also, it has a very nice parallel execution configuration for sharding capabilities etc using the MultiCapability config model.

I really see no obvious alternative offering free model, parallel execution, webkit support, and also cucumber support.

The closest seems to be Playwright though and am surprised so many are opting to go the Cypress route when Playwright is most likely the superior alternative at this point.

mkmurali commented 3 years ago

As many teams will be evaluating different options to move away from protractor and migrate all those test cases, please check this option described below as well.

Typically e2e tests are built by developers and many frameworks are oriented towards developers (and automation test engineers) with programming background. While there are tools to extract and provide documentation that can be reviewed by anybody in team, contributing to test cases is still limited to those with programming background.

We attempted to make e2e tests (user journey tests / acceptance) accessible to anybody in the team, review as well as contribute, by writing test cases without programming background, using Gauge and Taiko. We have implemented most common steps/actions a typical user will use to interact with an application in a browser at https://github.com/softrams/gauge-taiko-steps. So with this combination, any team member can write e2e user journey tests. Those developers and automation engineers with programming background could help with building additional steps as needed, but everybody in team can create executable test scripts that are also much easier to read/review without any programming background. Another side benefit is, with little effort we could switch the driver (currently Taiko) seamlessly if needed, without having to update any test cases.

We use Angular a lot in our projects and have been using this Gauge/Taiko framework with better results for about an year.

deleonio commented 3 years ago

Why does it have to be such a big toy as @cypress-io to write tests for the top of the test pyramid?

Ever thought of @nightwatchjs?

csvan commented 3 years ago

@Phonesis the only thing that isn't free about Cypress is the Dashboard and parallel execution, and both of those have open source alternatives (see e.g. https://github.com/sorry-cypress/sorry-cypress). Multi-browser and iframe support are both built in, and there are plugins for using Cucumber etc.

Cypress can feel monolithic and heavy at first, but it's surprisingly flexible and easy to work with. The community is very active and you will find a ton of plugins meeting different needs you may bump into.

StanislavKharchenko commented 3 years ago

Reading comments and understanding something another. No constructive pros/cons regarding tools, nor Protractor, nor alternatives (Cypress, etc). Just who chose what, who loves what and which tool is using now with. I've already described pros above my post and could continue: API usability (extended from selenium), configuration, remote running, parallel execution, and more-more. Could thousands of e2e run on different remote machines using cypress out of the box? What about iframes, parallel runs, different browsers (not only chromium based)? Can't see this, only feature requests and plans in roadmap. So, community should rely on external tools and waiting for bicycles which are working now with Protractor. Can I also rely to young Playwright which is from 2020? Maybe, but also a lot of uncertainty.

Well, unfortunately, RFC looks like an advertising of external tools, not more. And I'm very not sure about percentage in use. Just if looks downloads from npm:

protractor cypress

Actually, Protractor has been deprecated a year ago when version 6.0 rolled back by uknown reasons. From this time we can't see any development on this project We are all people and we understanding that in big opensource world it happens that there are not enough resources for development and support of tools which as result lead to deprecation. We understand this as main reason why Protractor is being deprecated and I'm not understanding people who refer to this project as garbage in twitter. There are a lot of time and resources spent for development, it working now and people using it. Very hard to realize that this work and time thrown away. Special thanks to @Google and Julie (@juliemr), Craig (@cnishina), Mike (@heathkit) and all contributors for hard work and investment to Protractor.

pauleustice commented 3 years ago

Unfortunately this RFC is on course to become a soapbox for everyone to proclaim their favourite alternative to Protractor 😄

Aside from thanking the Protractor team for years of hard work which has directly led to thousands of more stable, reliable websites, my primary point would be to request that the CLI doesn't support or favour any one E2E library out of the box. That tight coupling to an unaffiliated project could lead to problems in future if the chosen project becomes stagnant. Clearly, there are many great tools out there that already work well with Angular, and as such I believe the Angular team should leave it to development teams to assess what best fits their need.

Phonesis commented 3 years ago

@Phonesis the only thing that isn't free about Cypress is the Dashboard and parallel execution, and both of those have open source alternatives (see e.g. https://github.com/sorry-cypress/sorry-cypress). Multi-browser and iframe support are both built in, and there are plugins for using Cucumber etc.

Cypress can feel monolithic and heavy at first, but it's surprisingly flexible and easy to work with. The community is very active and you will find a ton of plugins meeting different needs you may bump into.

Thanks for info but the fact you need an open source alternative to run in parallel is not really ideal solution for us - lots of considerations around security / nexus scans etc. There is only chromium support so it isn't really multi-browser either (although looks like Webkit on roadmap).

csvan commented 3 years ago

There is only chromium support so it isn't really multi-browser either

Firefox is supported as well, with more on the way. It's true that the range of supported browsers is not as broad as Selenium though. https://docs.cypress.io/guides/guides/cross-browser-testing

FDIM commented 3 years ago

While I'm happy to see a decision, it's kind of disappointing that angular is moving away from all batteries included approach. First linting, now e2e tests and I guess time will tell when same thing will happen for unit tests?

I understand that people have different needs, but I also believe that having certain defaults brings a lot of value. Could you consider having cli builders for cypress, playwright(others?) and provide an option for initial scaffolding for users to choose from? You are the trusted experts and could continue to steer the community towards best practices.

aschlei commented 3 years ago

I'm approaching this from the perspective of Quality Control/Testing. I'm not a developer, but I do write the automated tests. We have thousands of Protractor tests that have already been converted to async/await. The best migration path that I see here is the option from FAQ#8, a thin wrapper with a compatible API. Ideally, we would be able to keep all of our tests and conf files mostly as-is, with minimal changes needed. The next best option is migration to straight selenium-webdriver; detailed guides for doing this conversion would be greatly appreciated if the then wrapper doesn't work out.

Providing resources for people who want to switch over to Cypress or some other framework is fine, but that seems to be more of a "complete rewrite" than a "migration".

Personally, I think the migration path to a thin wrapper or selenium-webdriver should be top priority (since these would be the easiest paths and actually rely on the same underlying technology). I think these things should be nailed down first, and then we can start the deprecation timer (a year or two after that work is complete), rather than starting the timer now and hoping that stuff will be completed in time for people to start planning and implementing their migrations.

kayvanbree commented 3 years ago

Cypress is awesome. Cool to see that I wasn't wrong when I adopted it 3 years ago. 👍

I would like to see something included by default in Angular though, so I don't think deprecating Protractor without a replacement is the best plan. Maybe include Cypress by default then?

jan-molak commented 3 years ago

Hey folks, huge thanks to Julie (@juliemr), Craig (@cnishina), Mike (@heathkit), and all the people who have contributed to Protractor over the years. I really value your hard work ❤️

@kyliau - From what I've seen in the industry and over the last 6-7 years working with Protractor, teams opting for Angular tend to do so because it makes building complex applications easier. However, complex applications typically mean complex workflows, and complex workflows mean complex tests (i.e fill in a 10-page long insurance claim form that changes depending on your answers, enter patient data into a healthcare system that only works in IE, etc.).

Additionally, software systems tested with Protractor, particularly in insurance, healthcare, finance, or banking industries, tend to be assembled from other, often legacy systems; so it's not just the nice Angular app our test needs to interact with, it's all the other "stuff" around it. This means that you'll often see iframes, multi-tab/window navigation, and an odd JavaScript alert every now and then. Not to mention that many of those auxiliary systems only work in IE (sadly, that still happens), and don't provide interfaces other than the UI to make them easier to test. While Protractor is not as shiny as some of the other alternatives, it's still perfectly capable of handling those scenarios.

The ability to run tests in parallel, integrate with test grids, and use the same framework for both web and mobile testing was another benefit of sticking with Protractor, and probably another reason why it's still downloaded so often.

It feels like the above points, as well as the fact that a complete re-write of a test suite is typically not given the highest priority for most delivery teams, FAQ#8 and a thin abstraction layer proposed in the comment by @cnishina look like the least disruptive way to migrate from Protractor to another, conceptually similar test framework (i.e. WebdriverIO, see comment).

A note to Serenity/JS users reading this thread - Serenity/JS will continue to support Protractor until at least the end of 2022, with support for other test runners like WebdriverIO and Playwright (and maybe even Cypress) coming soon.

We're also planning to make it easy for people to switch from one test runner to another if your tests follow the Screenplay Pattern, so watch this space and follow us on Twitter for updates 😃

Update 2021-04-10 - 🥁 In our recent survey, WebdriverIO has received the most votes, so expect @serenity-js/webdriverio integration coming soon. You can watch progress at serenity-js/serenity-js#805

sbley commented 3 years ago

@kyliau Would you mind to share the full results of the survey you conducted in January 2021?

StanislavKharchenko commented 3 years ago

In addition, we would like to say that regarding this and any similar topic need to dedicate at least 2 groups within any IT company who actually involved in E2E: directly developers who using e2e inside its angular projects (with angular-cli) and parallel teams (Automation QA) who building and supporting E2E infrastructure for testing angular and not only. The project provided by @jan-molak (Serenity-js) is a great example - automation QA engineers create and maintain e2e solutions which using Protractor as dependency.

Keen @kyliau, please, don't give up the Protractor in favor of Cypress or any other product. All these solutions do their job well and you understand it perfectly. If people have chosen Protractor and new people continuing to choose it - then it solves business needs.

Please consider this and provide ability to support and improve Protractor as independent E2E solution. We will wonderful to see Protractor with new selenium 4.0 and without control flow. It will be very good choise to have it as next gen, as more light ver (developing by Craig).

Thank you in advance!

Phonesis commented 3 years ago

In addition, we would like to say that regarding this and any similar topic need to dedicate at least 2 groups within any IT company who actually involved in E2E: directly developers who using e2e inside its angular projects (with angular-cli) and parallel teams (Automation QA) who building and supporting E2E infrastructure for testing angular and not only. The project provided by @jan-molak (Serenity-js) is a great example - automation QA engineers create and maintain e2e solutions which using Protractor as dependency.

Keen @kyliau, please, don't give up the Protractor in favor of Cypress or any other product. All these solutions do their job well and you understand it perfectly. If people have chosen Protractor and new people continuing to choose it - then it solves business needs.

Please consider this and provide ability to support and improve Protractor as independent E2E solution. We will wonderful to see Protractor with new selenium 4.0 and without control flow. It will be very good choise to have it as next gen, as more light ver (developing by Craig).

Thank you in advance!

Fully agreed. A new version of Protractor with just async/await support and Selenium 4 would be the ideal move.

andredesousa commented 3 years ago

In my opnion, the best migration path is the option from FAQ#8.

I am working for BMW and we have hundreds of developers working in internal projects using Angular and Protractor. Of course, we have some projects with other frameworks and tools (React, Jest, Cypress). Cypress is not an alternative, it works, but it is not the same. We are writing tests with Cucumber and we need to run these tests in different environments; multiple browser tabs open; in parallel; using sellenium grids; exporting the results to Jira, etc., etc. So, how can we use Cypress and migrate our tests? I can't find an alternative without losing resources.

Control flow is very old and used in AngularJs applications. I migrated old apps to async/await. Too easy. For large projects,I recommed creating two suites, because, it will take a long time. Deprecating Protractor, in my opinion, is too much. A new version with breaking changes will make everyone happy.

alvipeo commented 3 years ago

Angular team, would it be better if you get together with Microsoft (as you did for TypeScript) and come up with smooth migration from Protractor to Playwright? This way 1) it may be (I hope) a smooth migration and not too much trouble for you guys; 2) MS will still own Playwright and you communicate with each other as you do for TypeScript; 3) Angular devs and QA teams will get the best experience and long term support from two giants; 3) we as people who work with Angular will have confidence in both.

How about that?

praveendvd commented 3 years ago

@kyliau Thanks for the update , this is a really sad news as protractor when used as it is supposed to by disabling control flow and using await instead, proved to be an amazing easy tool.

So we have some projects that is in maintenance mode and no other future test scripts are required , could we keep using protractor, or we need to migrate legacy projects also ?

e-oz commented 3 years ago

Thank you for all the fish, Protractor team.

Please do not leave ng-cli without a built-in testing tool (when I use ng generate component I would love to still have some test files auto-generated).