Support E2E tests in Neuron's repo by github action

[Keith-CY]

This feature requires introducing E2E test code into Neuron's Repo and synchronizing it with Github Action runs.

Neuron currently has an automated test service at https://github.com/nervosnetwork/neuron-automation, but it is a timed service that is not triggered by every commit. It is also not able to keep up with the updates to https://github.com/nervosnetwork/neuron, which puts a burden on maintenance.

If we can maintain E2E tests in Neuron's Repo, we can improve efficiency and reduce maintenance costs.

Following are the test frameworks suggested in community:

• cypress: https://www.cypress.io/
• cypress + electron: https://www.cypress.io/blog/2019/09/26/testing-electron-js-applications-using-cypress-alpha-release/
• playwright: https://playwright.dev/
• playwright + electron: https://playwright.dev/docs/api/class-electron

[FrederLu]

Cypress and Playwright have some commonalities and differences in the e2e test scenario:

common ground:

1. Both support various user operations and custom scripts, which can be used to test various aspects of web applications, such as UI, functionality, performance, etc.
2. Both provide easy-to-use API and documentation.
3. Both support headless browser mode and visual mode to run tests.
4. Both provide rich test reports and logs.

difference:

1. The supported browsers are different: Cypress only supports Chrome, while Playwright supports Chrome, Firefox and WebKit.
2. Different restrictions on test scripts: Cypress can only run test scripts in the same browser window, and Playwright can run test scripts in multiple browsers and pages.
3. Assertions and test reports are different: Cypress has a built-in assertion library and a visual test runner, and the test report is more intuitive and easy to understand. Playwright requires the use of third-party libraries to implement assertions and test reports.
4. Cypress is free and open source, while Playwright is a paid tool.

In general, Cypress is more suitable for use in single-browser e2e test scenarios, and can provide more intuitive and easy-to-understand test reports and a visual test runner. Playwright is more suitable for use in e2e test scenarios of multiple browsers and pages, and can provide more flexible test script writing and execution.

[Keith-CY]

Cypress and Playwright have some commonalities and differences in the e2e test scenario:

common ground:

1. Both support various user operations and custom scripts, which can be used to test various aspects of web applications, such as UI, functionality, performance, etc.
2. Both provide easy-to-use API and documentation.
3. Both support headless browser mode and visual mode to run tests.
4. Both provide rich test reports and logs.

difference:

1. The supported browsers are different: Cypress only supports Chrome, while Playwright supports Chrome, Firefox and WebKit.
2. Different restrictions on test scripts: Cypress can only run test scripts in the same browser window, and Playwright can run test scripts in multiple browsers and pages.
3. Assertions and test reports are different: Cypress has a built-in assertion library and a visual test runner, and the test report is more intuitive and easy to understand. Playwright requires the use of third-party libraries to implement assertions and test reports.
4. Cypress is free and open source, while Playwright is a paid tool.

In general, Cypress is more suitable for use in single-browser e2e test scenarios, and can provide more intuitive and easy-to-understand test reports and a visual test runner. Playwright is more suitable for use in e2e test scenarios of multiple browsers and pages, and can provide more flexible test script writing and execution.

Only chrome/chromium is used in Neuron because it's the built-in webview of electron so the difference in browsers can be ignored.

Ref:

• webview in electron: https://www.electronjs.org/docs/latest/api/webview-tag

[silySuper]

1.In consideration of a great advantage between Cypress and PlayWright is PlayWright can test multiple browsers in parallel，but Cypress can not.If the difference in browsers can be ignored,Cypress may be the better choice. 2.I have tried to run some easy UI examples successfully through Cypress and PlayWright on my local computer，Cypress is easier for me to understand and use,beacause its documents is more detailed 3.I have found some good videos talking about Cypress Vs PlayWright ,introduce differences in detail https://www.youtube.com/watch?v=fncL63KRA-0 https://www.youtube.com/watch?v=cixkQHNkoZ4

[Keith-CY]

Let's make a decision. From the consideration above and the that we only care about chromium in practice, cypress should be a better choice.

[Keith-CY]

Now we can schedule how and when to move e2e tests from https://github.com/nervosnetwork/neuron-automation to https://github.com/nervosnetwork/neuron/

[FrederLu]

Now we can schedule how and when to move e2e tests from https://github.com/nervosnetwork/neuron-automation to https://github.com/nervosnetwork/neuron/

Later, Neuron's E2E will be developed by using cypress. First of all, familiarize yourself with and sort out the use cases in the https://github.com/nervosnetwork/neuron-automation warehouse. In theory, these use cases will be extended as long as the original functions of Neuron are not adjusted. After sorting out the cases above, combine the characteristics of Cypress to sort out and output the automation cases. https://github.com/nervosnetwork/neuron/ is Neuron's existing warehouse. It is more reasonable to open a new warehouse for the automation project later.

[littleLip520]

After reading the project, I think we need to consider the following points when writing a new project with cypress:

1. The original project was written in JAVA and needs to be converted to JS, which requires some learning costs.
2. Confirm whether the original page elements are available. --The latest submission code of the original project is more than half a year old, which is relatively long.
3. Whether to start scripting at this stage. --neuron wallet will have a new UI soon, and the elements can be expected to change a lot. At this stage, starting automatic writing can predict that there will be a large amount of debugging work in the future (re-acquire elements)
4. It is better to upgrade the structure of the project, such as doing a layering (page object layer, case layer, tool layer, etc).

[silySuper]

My suggestion is that we can split the task into several parts similar to test plan firstly ，then assign in QA team.

[Keith-CY]

I prefer to add e2e inside https://github.com/nervosnetwork/neuron/ so each commit of developers will be tested by e2e.

test each commit can be achieved by github action between two repositories too, so it's OK to keep them separate.

The difference is that e2e inside neuron repo prevents merging if tests fail, while e2e triggered by neuron repo will not prevent it, only an alert will be reported.