Support automated tests for full-node mode and light-client mode

[Keith-CY]

Recently we found some unexpected behaviors of Neuron when it was in the light client mode. Then I realized that Neuron is a very special case in CKB ecosystem because it's the only product that supports full-node and light-client simultaneously. That means, tests should be doubled for the same function.

Due to the increment in workload, we should accelerate the automation of tests.

It would be challenging because we haven't figured out how to run the e2e tests with full-node in the github runner but now there comes another ckb client.

I'll connect to acceptance team to see if they have handled such situations.

Ref:

• Support E2E tests in Neuron's repo by github action: https://github.com/Magickbase/neuron-public-issues/issues/132

[Keith-CY]

IMO, Neuron's E2E testing can be divided into two categories:

1. UI interaction testing, involving link navigation, form input, and data display.
2. Testing the synchronization of on-chain data, ensuring that data remains consistent and complete for a given wallet when synchronized to a specific height.

The first category of testing is similar to standard software testing processes and can be supported by automating verification by importing test data into Neuron's database. The specific workflow is as follows:

1. Trigger automated testing upon code submission.
2. Before running the tests, repackage Neuron.
3. Before running Neuron, replace its built-in database with a test database, or use a script to write data into Neuron. Using a script is recommended, as it allows for adapting to data format changes, such as adding or removing fields.
4. Run Neuron and use testing tools to verify the interaction of the pages.

The second category of testing is different from standard software testing and is more similar to end-to-end testing for backend services. One challenging aspect is the need to cover both full node and light client modes. However, after isolating the full node and light client databases (https://github.com/Magickbase/neuron-public-issues/issues/294), it is possible to complete this testing by comparing database snapshots. The specific steps are as follows:

1. Select a wallet with complete test data and export databases for both the full node and the light client, synchronized to the same height, as the testing baseline.
2. Build a persistent testing environment to provide external nodes and light clients.
3. Before running the tests, repackage Neuron.
4. Run Neuron and wait for it to synchronize to the specified height.
5. Use a script to compare Neuron's database with the baseline database.

Since the data volume for light clients is smaller, testing the synchronization of light clients can also be performed by importing them into the testing environment each time, rather than maintaining a persistent environment.

[silySuper]

In order to ensure the continuity of dynamic operations，we should consider how to make data usable.For example,if we want to test send transaction firstly ,then send lock time transaction,sufficient balance in wallet is needed to advoid throw abnormal tips and automated situation are interrupted.

[Keith-CY]

Now that E2E tests are grouped into 3 parts, we can make schedules for each one.

I'm a bit familiar with test frameworks for UI/interactions, so I will lead this part.

There are 2 frameworks for an electron app

• cypress: https://docs.cypress.io/guides/guides/launching-browsers#Electron-Browser
• playwright: https://playwright.dev/docs/api/class-electron

They are almost the same for our project except

1. cypress is somehow heavy because it relies on a local app(CI would fail because it cannot download the app, really bothering);
2. playwright is so fresh that its support for electron is experimental.

Personally prefer playwright for our test framework of UI because failure to download cypress in CI did annoy me, any idea from @yanguoyu @homura @WhiteMinds @devchenyan @zhangyouxin @Daryl-L @PainterPuppets

[Keith-CY]

Now that E2E tests are grouped into 3 parts, we can make schedules for each one.

I'm a bit familiar with test frameworks for UI/interactions, so I will lead this part.

There are 2 frameworks for an electron app

• cypress: https://docs.cypress.io/guides/guides/launching-browsers#Electron-Browser

• playwright: https://playwright.dev/docs/api/class-electron

They are almost the same for our project except

1. cypress is somehow heavy because it relies on a local app(CI would fail because it cannot download the app, really bothering);

2. playwright is so fresh that its support for electron is experimental.

Personally prefer playwright for our test framework of UI because failure to download cypress in CI did annoy me, any idea from @yanguoyu @homura @WhiteMinds @devchenyan @zhangyouxin @Daryl-L @PainterPuppets

Playwright will be used if there's no more feedback

[Danie0918]

@Keith-CY This issue will be taken by @devchenyan, please hand over the relevant work content.

[Keith-CY]

@Keith-CY This issue will be taken by @devchenyan, please hand over the relevant work content.

I'll DM @devchenyan for this task

[silySuper]

Which github code link is related to this ?

[Keith-CY]

Which github code link is related to this ?

The PR is https://github.com/nervosnetwork/neuron/pull/3004 but I found the CI failed, please have a check @devchenyan

[Keith-CY]

Which github code link is related to this ?

The PR is nervosnetwork/neuron#3004 but I found the CI failed, please have a check @devchenyan

Any update on this PR @devchenyan

[silySuper]

Now neuron can launch ,but can not locate element,what is the way to locate electron element ?@devchenyan

[silySuper]

If launch by manual operation，we can write send transaction as first example ,does send page element can be located?can you upload a video for writen process if it is convenient for you ? @devchenyan

[Danie0918]

Any update on this PR @devchenyan @silySuper

[silySuper]

Already know how to location element, main examples is writing.

[silySuper]

network can not connected by playwritetest,always show is connecting either innernal node or light client node

main.log

@devchenyan

[silySuper]

Now is solving in full node yarn start can sync normally，but test：e2e can not sync.

[silySuper]

1.test environment sync is OK now 2.main test case(full node and light client node)： (page）

a.send transaction--done b.check history transation --done c.nervos dao deposit --done d.one cell consume--done e.create account in asset accounts--done f.amend transaction--done g.claim in customized page-done (menu)

h.sign message--to do i.multisig message --to do j.offline message--to do k.broadcast message--to do (hard wallet) l:receive m:send

@Danie0918 progress is updated here.

[Keith-CY]

1.test environment sync is OK now 2.main test case： a.send transaction--done b.check history transation --done c.nervos dao deposit -- is doing d.two cells consume--to do e.create account in asset accounts--to do f.amend transaction--to do(full node) g.claim in customized page--to do (window) h.sign message--to do i.multisig message --to do j.offline message--to do k.broadcast message--to do

3.I have no permission to push code to enable-playwrite branch,please open it. @Keith-CY

Permission updated

[silySuper]

Performance test situations to do: 1.A single transaction has a large number of cells. 2.Extra large transactions. 3.A single wallet has a large number of addresses. 4.A large number of wallets.

[silySuper]

1.svg element is hard to locate.It will be more convenient to write locator if it is img.I look other some websites svg locate,they does not use svg,but use like this: await page.locator('footer').filter({ hasText: '长沙红胖子Qt 2024-06-14 10:09 0 0' }).getByRole('link').nth(1).click();maybe it need to write detailed hierarchical relationships 2.some elements can only be located by xpath,it need to be modified if the page has little change,and may increase maintain cost，can you add id to each elemment if it is convenient for you? @devchenyan @yanguoyu

[yanguoyu]

1.svg element is hard to locate.It will be more convenient to write locator if it is img.I look other some websites svg locate,they does not use svg,but use like this: await page.locator('footer').filter({ hasText: '长沙红胖子Qt 2024-06-14 10:09 0 0' }).getByRole('link').nth(1).click();maybe it need to write detailed hierarchical relationships

Why does the SVG is hard to locate? I don't know the difference between locating svg and img.

2.some elements can only be located by xpath,it need to be modified if the page has little change,and may increase maintain cost，can you add id to each elemment if it is convenient for you?

Adding an ID for each element is hard to maintain because the ID should not be the same on the global page. How about adding an ID for the page root or component root? Anyway, e2e tests do need to be modified when the page changes.

[silySuper]

1.I have not seen anything about svg in playwrite official website yet,so I search other websites and only find https://www.cnblogs.com/yoyoketang/p/17354636.html,which does not written integrally.But I have solve how to locate svg by random attempt. 2.increase change is needed,but other change less and less.

[yanguoyu]

1.I have not seen anything about svg in playwrite official website yet,so I search other websites and only find https://www.cnblogs.com/yoyoketang/p/17354636.html,which does not written integrally.But I have solve how to locate svg by random attempt.

I guess use selector to locate the SVG is also ok? Or use https://playwright.dev/docs/api/class-page#page-query-selector? eg: await page.locator('svg').click();

2.increase change is needed,but other change less and less.

Add ID for each element is a bit wired even if it's just for testing. What do you think? @Keith-CY

[silySuper]

1.I have not seen anything about svg in playwrite official website yet,so I search other websites and only find https://www.cnblogs.com/yoyoketang/p/17354636.html,which does not written integrally.But I have solve how to locate svg by random attempt.

I guess use selector to locate the SVG is also ok? Or use https://playwright.dev/docs/api/class-page#page-query-selector? eg: await page.locator('svg').click();

2.increase change is needed,but other change less and less.

Add ID for each element is a bit wired even if it's just for testing. What do you think? @Keith-CY

1.await page.locator('svg').click();not use ,svg need to locate detailed svg.because several svg is in one page,I have solved by name() and xpath

[yanguoyu]

1.I have not seen anything about svg in playwrite official website yet,so I search other websites and only find https://www.cnblogs.com/yoyoketang/p/17354636.html,which does not written integrally.But I have solve how to locate svg by random attempt.

I guess use selector to locate the SVG is also ok? Or use https://playwright.dev/docs/api/class-page#page-query-selector? eg: await page.locator('svg').click();

2.increase change is needed,but other change less and less.

Add ID for each element is a bit wired even if it's just for testing. What do you think? @Keith-CY

1.await page.locator('svg').click();not use ,svg need to locate detailed svg.because several svg is in one page,I have solved by name() and xpath

The locator function can be called by the locator result. How about use page.locator('path').locator(svg)?

[silySuper]

1.test environment sync is OK now 2.main test case(full node and light client node)： (page）

a.send transaction--done b.check history transation --done c.nervos dao deposit --done d.one cell consume--done e.create account in asset accounts--done f.amend transaction--done g.claim in customized page-done (menu)

h.sign message--to do i.multisig message --to do j.offline message--to do k.broadcast message--to do (hard wallet) l:receive m:send

@Danie0918 progress is updated here.

For menu function automation,palywrite can not operated,but can use AppleScript.Now this script can run in my local neuron:

but can not run in app launched by playwrite becauseof this:

can this appname Electron changed to Neuron in playwrite? @devchenyan @yanguoyu

[silySuper]

updated

(menu-tool) click menu by AppleScript because playwrite does not support Operating system automation h.sign message--done i.multisig message --is doing(Key point：address must be different and effective every time) j.offline message--to do(Key point：json document in local must be different and effective every time) k.broadcast message--to do (menu-hard wallet) l:create --done m:receive--done n:send--done （enhancement） o:optimize code layering--to do

[silySuper]

updated

(menu-tool) i.multisig message--done j.offline message--done k.broadcast message--done (enhancement） o:optimize code layering and run all--is doing（plan to finish before date 2024.7.5 ） (add new case) p.export transaction--done q.sign and export transaction--done @Danie0918 @Keith-CY

[silySuper]

Applescript can not find process Neuron now(which can run last week)，I have searched running process，does not have process name Neuron.But When I replace Electron ,it shows menubar which is not belong to Neuron.

[silySuper]

Testnet network can not sync ,and shows is connecting after refresh several times.

main.log

bundled-ckb.log

bundled-ckb.log

please have a look~

@devchenyan @yanguoyu

[yanguoyu]

Testnet network can not sync ,and shows is connecting after refresh several times.

main.log

bundled-ckb.log

bundled-ckb.log

please have a look~

@devchenyan @yanguoyu

From the main.log you connect to the local ckb node. So may be the bundled-ckb.log should be your local ckb node log but not export from the Neuron.

CKB:    fail to start bundled CKB with error:
[2024-07-03T08:59:42.080Z] [error] Error: ENOENT: no such file or directory, copyfile 'neuron/packages/neuron-wallet/light/ckb_light_testnet.toml' -> '/Users/chllp/Library/Application Support/Electron/chains/light/testnet/ckb_light.toml'

And from the log, neuron start light client failed, it show that you start with development. So you'd better run ./scripts/download-ckb.sh to download the light client binary file.

[silySuper]

Change to light client is not my main point.I want to use testnet to run cases,but it can not. bundled-ckb.log has exported above.

[silySuper]

I have found that a Electron helper can not exit automatically when neuron exit.

https://github.com/Magickbase/neuron-public-issues/assets/25971273/7e9c3861-1c35-4485-87b7-fbed586ac2d9

when force out this helper ,and open neuron again ,tne sync become normal,maybe my problem is related to https://github.com/orgs/Magickbase/projects/6/views/2?pane=issue&itemId=68796884

[yanguoyu]

I have found that a Electron helper can not exit automatically when neuron exit.

2024-07-04.10.22.05.mov when force out this helper ,and open neuron again ,tne sync become normal,maybe my problem is related to https://github.com/orgs/Magickbase/projects/6/views/2?pane=issue&itemId=68796884

I will check this, but it's an exception branch, so it may not break the main process.

[yanguoyu]

Change to light client is not my main point.I want to use testnet to run cases,but it can not. bundled-ckb.log has exported above.

Does this log exported from the ckb that you started locally?

[silySuper]

Change to light client is not my main point.I want to use testnet to run cases,but it can not. bundled-ckb.log has exported above.

Does this log exported from the ckb that you started locally?

Yes,only have a little content in this log when network is abnormal.

[yanguoyu]

Change to light client is not my main point.I want to use testnet to run cases,but it can not. bundled-ckb.log has exported above.

Does this log exported from the ckb that you started locally?

Yes,only have a little content in this log when network is abnormal.

So, it may has a problem when you start the local ckb node. It is not a problem with Neuron. Because the locally started ckb is not controlled by Neuron.

[silySuper]

Change to light client is not my main point.I want to use testnet to run cases,but it can not. bundled-ckb.log has exported above.

Does this log exported from the ckb that you started locally?

Yes,only have a little content in this log when network is abnormal.

So, it may has a problem when you start the local ckb node. It is not a problem with Neuron. Because the locally started ckb is not controlled by Neuron.

Ckb node is running normally in background,it happened when neruon exit.I have operated stop ckb node to reproduct sync problem before ,but sync is normal.

[yanguoyu]

sync is normal.

So currently sync is normal?

[silySuper]

sync is normal.

So currently sync is normal?

Now is normal.

[silySuper]

/neuron/packages/e2e/README.md has updated.

[Danie0918]

All test cases have been completed, but they need to be run manually on a regular basis and cannot be fully automated, pending subsequent optimization.

[silySuper]

Maybe CI failed is not related to my testcase.I found that CI never success in history. I will try to check configuration firstly.

[silySuper]

CI environment shows an database error.

[yanguoyu]

CI environment shows an database error.

Which action throws this error or it can be reproduced locally?

[silySuper]

CI environment shows an database error.

Which action throws this error or it can be reproduced locally?

After UI automic create wallet it shows this today，test cases are same，you can rerun test cases to see error.

[yanguoyu]

CI environment shows an database error.

Which action throws this error or it can be reproduced locally?

After UI automic create wallet it shows this today，test cases are same，you can rerun test cases to see error.

I ran with the branch enable-playright, and I had no problem. Please describe how to reproduce the issue.

[silySuper]

I do nothing,just run CI...OK, if it is right now ,I will go on.

[silySuper]

I have balance ,and have no confirming record in history,also sync 100%.Why always show insufficient balance when I send transaction?

[yanguoyu]

I have balance ,and have no confirming record in history,also sync 100%.Why always show insufficient balance when I send transaction?

When you input the amount, the transactions have not synced yet. After the transactions are synced, the amount doesn't change, so the page will not refresh in the screenshot. After I change the amount, it will show correct.

https://github.com/user-attachments/assets/5edae3a0-33e5-42f9-8881-be49853f52a6

Besides, after this commit, And change this line https://github.com/nervosnetwork/neuron/blob/develop/packages/neuron-wallet/src/services/settings.ts#L123 locally

// do not push this change to the remote
- locale: app.getLocale(),
+ locale: 'en',

Running locally e2e test case will be similar to the action running.

[silySuper]